Analog simulation method, device and equipment for surgical operation and storage medium

Abstract

The invention discloses an analog simulation method, device and equipment for a surgical operation and a storage medium. The method comprises the following steps: performing image processing on a plurality of obtained DICOM sequence images to generate a three-dimensional virtual human body model; the plurality of DICOM sequence images are medical image data of the same patient; projecting the three-dimensional virtual human body model on an interactive screen by adopting a polarization projection mode, so that the three-dimensional virtual human body model is stereoscopically displayed in the interactive screen; and in response to the identified user interaction operation, executing a simulation operation on the three-dimensional virtual human body model. According to the method, through preoperative simulation drilling of the surgical operation, the accuracy of planning the surgical operation process by a doctor is improved, and the surgical risk is reduced.

Description

Technical Field

[0001] The present application relates to the field of virtual simulation technology, and particularly to a method, device, equipment and storage medium for simulating and emulating a surgical operation. Background Art

[0002] In the medical field, surgical operations are high-risk and high-precision medical activities. Any minor mistake may have a serious impact on the life and health of patients. Therefore, doctors must have rich experience and excellent skills when performing surgical operations, and make full preoperative preparations before the surgical operation to cope with various possible situations.

[0003] Currently, for the disease diagnosis and surgical procedure planning before a surgical operation, it is mainly up to doctors to rely on the two-dimensional medical images of patients, make judgments based on professional knowledge and experience, and make treatment plans accordingly.

[0004] However, due to the limitations of doctors' clinical experience, during the operation, doctors can only determine the next operation by continuously dissecting and exposing the lesion tissue, resulting in difficulty for doctors to make accurate judgments and plans for surgical operations, and further increasing the surgical risk. Summary of the Invention

[0005] Based on the above problems, the present application provides a method, device, equipment and storage medium for simulating and emulating a surgical operation, aiming to improve the accuracy of doctors' surgical operation process planning through preoperative simulation drills of surgical operations and reduce the surgical risk.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] In a first aspect, the present application provides a method for simulating and emulating a surgical operation, including:

[0008] Performing image processing on the obtained multiple DICOM sequence images to generate a three-dimensional virtual human body model; the multiple DICOM sequence images are medical image data of the same patient;

[0009] Projecting the three-dimensional virtual human body model on an interactive screen by means of polarized projection, so that the three-dimensional virtual human body model is stereoscopically displayed on the interactive screen;

[0010] Responding to the recognized user interaction operation, and performing a simulation operation on the three-dimensional virtual human body model.

[0011] Optionally, in the method as described above, the performing image processing on the obtained multiple DICOM sequence images to generate a three-dimensional virtual human body model includes:

[0012] For each of the multiple DICOM sequence images, perform the following operations:

[0013] Perform semantic segmentation processing on the sequence image through a preset semantic segmentation algorithm to obtain a semantic segmentation map of the sequence image;

[0014] Call Slicer software to analyze the semantic segmentation map of the sequence image to obtain a target surgical path;

[0015] Based on each semantic segmentation map and the target surgical path corresponding to each semantic segmentation map, construct a three-dimensional virtual human model through a multi-plane three-dimensional reconstruction algorithm; the three-dimensional virtual human model contains lesion tissue information and a surgical planning path; the surgical planning path is determined according to the target surgical path corresponding to each semantic segmentation map.

[0016] Optionally, in the method as described above, the calling Slicer software to analyze the semantic segmentation map of the sequence image to obtain a target surgical path includes:

[0017] Locate the lesion tissue in the semantic segmentation map through Slicer software to generate the neurovascular structure of the lesion tissue;

[0018] Based on the neurovascular structure, use the ant colony optimization algorithm to optimize a preset surgical path to generate a target surgical path; the target surgical path does not intersect with the neurovascular structure.

[0019] Optionally, in the method as described above, the adopting a polarized projection method to project the three-dimensional virtual human model on an interactive screen so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen includes:

[0020] Based on the three-dimensional virtual human model, generate two polarized images of the three-dimensional virtual human model by using a polarized projection method; the polarization directions of the two polarized images are different;

[0021] Display the two polarized images through the interactive screen so that the user synthesizes the two polarized images received by the left and right eyes into a stereoscopically displayed three-dimensional virtual human model in the brain.

[0022] Optionally, in the method as described above, the user interaction operation includes user gesture operations and the user's operation on a force feedback device, and the responding to the recognized user interaction operation to perform a simulation operation on the three-dimensional virtual human model includes:

[0023] Use a gesture recognition algorithm to recognize the user gesture operation to obtain a target gesture operation;

[0024] Map the target gesture operation to the corresponding simulation operation on the three-dimensional virtual human model and perform physical simulation on the three-dimensional virtual human model;

[0025] In response to the user's operation on the force feedback device, perform a corresponding simulation operation on the three-dimensional virtual human model; the force feedback device is used to control the movement trajectory of the virtual surgical instrument.

[0026] Optionally, in the method as described above, the step of, in response to the user's operation on the force feedback device, performing a corresponding simulation operation on the three-dimensional virtual human model includes:

[0027] In response to the user's operation on the force feedback device, obtain the motion parameters of the force feedback device;

[0028] When it is determined that the force feedback device comes into contact with the three-dimensional virtual human model, perform corresponding physical simulation on the three-dimensional virtual human model according to the motion parameters.

[0029] Optionally, in the method as described above, the method further includes:

[0030] Generate a mechanical feedback signal according to the result of the physical simulation, and feed the mechanical feedback signal back to the force feedback device, so that the user can sense the feedback force corresponding to the mechanical feedback signal through the force feedback device.

[0031] In a second aspect, the present application provides a surgical simulation device, including:

[0032] An image processing module, configured to perform image processing on multiple acquired DICOM sequence images to generate a three-dimensional virtual human model; the multiple DICOM sequence images are medical image data of the same patient;

[0033] An image projection module, configured to project the three-dimensional virtual human model on an interactive screen in a polarized projection manner, so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen;

[0034] A user interaction module, configured to perform a simulation operation on the three-dimensional virtual human model in response to an identified user interaction operation.

[0035] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0036] The memory stores computer-executable instructions;

[0037] The processor executes the computer-executable instructions stored in the memory to implement the surgical simulation method according to any one of the above embodiments.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the surgical simulation method described in any one of the above embodiments when executed by a processor.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] The present application processes the acquired multiple DICOM sequence images to generate a three-dimensional virtual human model of the patient; and projects the three-dimensional virtual human model onto an interactive screen by means of polarized projection, so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen, and further enables a doctor to perform corresponding surgical simulations on the three-dimensional virtual human model on the interactive screen, thereby improving the accuracy of the doctor's surgical process planning, reducing surgical risks, and facilitating the achievement of an ideal surgical result that is scientific, precise, minimally invasive, and safe. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0042] Figure 1A Flowchart of the inventive concept of a surgical simulation method provided by the present application;

[0043] Figure 1B Schematic flowchart of an embodiment of a surgical simulation method provided by the present application;

[0044] Figure 2 Schematic flowchart of another embodiment of a surgical simulation method provided by the present application;

[0045] Figure 3 Schematic flowchart of still another embodiment of a surgical simulation method provided by the present application;

[0046] Figure 4 Schematic flowchart of yet another embodiment of a surgical simulation method provided by the present application;

[0047] Figure 5 Schematic structural diagram of an embodiment of a surgical simulation device provided by the present application;

[0048] Figure 6A schematic structural diagram of an embodiment of an electronic device provided by the present application. Detailed implementation manners

[0049] As described above, for the current disease diagnosis and surgical procedure planning before surgery, doctors mainly rely on two-dimensional medical images of patients and make judgments based on professional knowledge and experience, and accordingly make treatment plans. During the surgery, doctors can only continuously dissect and expose the lesion tissue to determine the next operation, which highly depends on the clinical experience of doctors.

[0050] After research, the inventors proposed a simulation method, device, equipment and storage medium for surgical operations to solve the technical problem in the prior art that the surgical operation process is uncertain and can only rely on the clinical experience of doctors, resulting in a relatively high surgical risk.

[0051] To enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0052] See Figure 1A , this figure is a flowchart of the inventive concept of a simulation method for surgical operations provided by the present application. As Figure 1A shown, semantic segmentation, surgical path planning and model reconstruction are performed on the preoperative medical images of the patient to obtain a three-dimensional virtual human model of the patient. In the Unity 3D platform, the three-dimensional virtual human model is projected onto an interactive screen through holographic projection. At the same time, using gesture recognition interaction and force feedback interaction through a force feedback device, the operations of the user are mapped onto the operations on the three-dimensional virtual human model through the Unity 3D platform.

[0053] See Figure 1B , this figure is a schematic flowchart of an embodiment of a simulation method for surgical operations provided by the present application. The method includes:

[0054] S101: Perform image processing on multiple obtained DICOM sequence images to generate a three-dimensional virtual human model.

[0055] Among them, the multiple DICOM sequence images are medical image data of the same patient.

[0056] In this embodiment, first, a self-designed semantic segmentation algorithm is used to perform semantic segmentation processing on the medical image data of the same patient to obtain a semantic segmentation map corresponding to the medical image data of the patient; then, by calling the Slicer software, the lesion tissue in the semantic segmentation map is located, and the neurovascular structure of the lesion tissue is generated. Based on the neurovascular structure, avoiding key tissue parts such as blood vessels and nerves, a shortest direct surgical path with safety constraints as the premise is planned, and this path is the surgical instrument approach when the surgeon performs the operation. Then, the ant colony optimization algorithm is used to optimize the preset surgical path to generate the final target surgical path. Based on each generated semantic segmentation map and the three-dimensional representation of the target surgical path corresponding to each semantic segmentation map in the three-dimensional coordinate system, a three-dimensional virtual human model is constructed through a multi-plane three-dimensional reconstruction algorithm.

[0057] S102: In a polarized projection manner, project the three-dimensional virtual human model onto an interactive screen so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen.

[0058] In this embodiment, the three-dimensional virtual human model is pushed to an interactive display screen and, in a polarized projection manner, a stereoscopic display effect on the interactive screen is achieved by adjusting the polarization angle and the viewing distance. Specifically, the three-dimensional virtual human model is sent to a projector, and through the polarizing plate on the projector, two polarized images with different polarization directions are generated, and the two polarized images are projected onto the interactive screen for display, so that the user combines the two polarized images received by the left and right eyes in the brain to form a stereoscopically displayed three-dimensional virtual human model.

[0059] It can be understood that the interactive screen can be a photon suspension screen, and the user can achieve viewing by using the method of naked-eye 3D or wearing 3D glasses. At the same time, the interactive screen can be arranged on the operating table, and the projection device is suspended above to project and display the three-dimensional virtual human model, and the surgeon can perform interactive operations in front of the operating table.

[0060] S103: In response to the recognized user interaction operation, perform a simulation operation on the three-dimensional virtual human model.

[0061] In this embodiment, the user interaction operations may include the user's gesture operations and the operations of the user simulating the manipulation of surgical instruments through a force feedback device. For the user's gesture operations, a gesture recognition algorithm is used to recognize the user's gesture operations collected by the somatosensory camera to obtain the target gesture operations, so as to map the target gesture operations to the corresponding simulation operations on the three-dimensional virtual human model on the Unity 3D platform to perform physical simulation on the three-dimensional virtual human model, and realize interactive operations such as stretching, dragging, rotating, and resetting on the three-dimensional virtual human model. For the operations of the user simulating the manipulation of surgical instruments through a force feedback device, according to the manipulation of the motion trajectory of the virtual surgical instrument by the user through the force feedback device, when it is determined that the virtual surgical instrument simulated by the user controlling the force feedback device contacts the three-dimensional virtual human model, the corresponding virtual instrument moves in real-time synchronization, and corresponding physical simulation is performed on the three-dimensional virtual human model in the Unity 3D platform.

[0062] In this embodiment, by performing image processing on the acquired multiple DICOM sequence images, a three-dimensional virtual human model of the patient is generated; and through the method of polarized projection, the three-dimensional virtual human model is projected on an interactive screen, so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen, and further enables a doctor to perform corresponding surgical simulations on the three-dimensional virtual human model on the interactive screen, thereby improving the accuracy of the doctor's surgical process planning, reducing surgical risks, and facilitating the achievement of ideal surgical results that are scientific, precise, minimally invasive, and safe.

[0063] See Figure 2 , which is a schematic flowchart of another embodiment of a surgical simulation method provided by this application. As Figure 2 shown, based on the above embodiment, a specific implementation manner of "performing image processing on the acquired multiple DICOM sequence images to generate a three-dimensional virtual human model" in S101 includes:

[0064] For each sequence image in the multiple DICOM sequence images, the following operations are performed:

[0065] S1011: Perform semantic segmentation processing on the sequence image through a preset semantic segmentation algorithm to obtain the semantic segmentation map of the sequence image.

[0066] Among them, the self-designed semantic segmentation algorithm is named the ResPVT algorithm. The network architecture of the ResPVT algorithm uses a dual-backbone form combining a core backbone and an auxiliary backbone as the encoder of the network. Among them, the core backbone uses a pyramid-shaped Transformer (PVT), and the auxiliary backbone uses a CNN structure ResNet. The ResPVT algorithm iteratively takes the output features of the auxiliary backbone as part of the input features and inputs them into the core backbone stage by stage. Finally, the core backbone outputs the final features, and then generates the prediction results. At the same time, in the ResPVT algorithm, the segmentation performance is improved by combining the attention mechanism and the feature fusion module. The publicly available brain tumor dataset is used as the training set, and the magnetic resonance sequence images of the subjects are used as the test set to train the ResPVT algorithm to obtain the semantic segmentation map of the lesion sites of the subjects.

[0067] S1012: Call the Slicer software to analyze the semantic segmentation map of the sequence images to obtain the target surgical path.

[0068] In this embodiment, since the surgical path is simplified to a straight line, the surgical path can be represented by six parameters. Since the target surgical path always passes through the surgical target tissue, in fact, the surgical path can be represented by the slopes on three axes, namely the X-axis slope, the Y-axis slope, and the Z-axis slope. The distance between the target surgical path and the lesion tissue site segmented by semantics and the length of the target surgical path are used as optimization variables, and the ant colony optimization algorithm is used to optimize the optimization variables to obtain the target surgical path.

[0069] Among them, the ant colony optimization algorithm is an optimization algorithm based on the foraging behavior of ants in nature and is suitable for solving path optimization problems. The specific steps include initializing the ant colony and parameters; calculating the path length of each ant and the distance from the key tissue sites; updating the pheromone; selecting the path according to the pheromone concentration; repeating the above steps until the convergence condition is reached.

[0070] 1013: Based on each semantic segmentation map and the target surgical path corresponding to each semantic segmentation map, construct a three-dimensional virtual human model through a multi-plane three-dimensional reconstruction algorithm.

[0071] Among them, the three-dimensional virtual human model contains lesion tissue information and surgical planning paths; the surgical planning paths are determined according to the target surgical paths corresponding to each semantic segmentation map.

[0072] It can be understood that the target surgical path obtained from these semantic segmentation maps is the cross-section of the final surgical planning path in the semantic segmentation map. Then, the surgical path cross-sections of all semantic segmentation maps are combined together to form a straight line, that is, the final surgical planning path.

[0073] In this embodiment, for example, if there are n semantic segmentation maps, there will be correspondingly n target surgical paths. Add a y coordinate in the vertical direction in the xOz plane, represent the n target surgical paths corresponding to the n semantic segmentation maps in a three-dimensional coordinate system, and form a three-dimensional virtual knowledge-based human body model containing lesion tissue information and surgical planning paths through a multi-plane three-dimensional reconstruction algorithm.

[0074] In this embodiment, through a preset semantic segmentation algorithm, precise semantic segmentation processing is performed on the sequence images, accurately distinguishing different tissues or structures in the images, obtaining the semantic segmentation maps of the sequence images, and providing an accurate basis for subsequent surgical path planning. By calling the Slicer software to deeply analyze the semantic segmentation maps of the sequence images, relatively accurate target surgical paths are generated, improving the accuracy and efficiency of surgical path planning. Based on each semantic segmentation map and the target surgical path corresponding to each semantic segmentation map, a three-dimensional virtual human body model is constructed through a multi-plane three-dimensional reconstruction algorithm, which can truly restore the anatomical structure and lesion conditions of the human body. This enables doctors to perform surgical simulations in a virtual environment and more intuitively understand the surgical process.

[0075] Further, on the basis of the above embodiment, a specific implementation manner of "calling the Slicer software to analyze the semantic segmentation maps of the sequence images to obtain the target surgical paths" in S1012 includes:

[0076] Locate the lesion tissue in the semantic segmentation map through the Slicer software to generate the neurovascular structure of the lesion tissue.

[0077] In this embodiment, after loading the semantic segmentation map of the patient's medical image data in the Slicer software, use the existing blood vessel segmentation algorithm in the software to segment the neurovascular structure and generate the neurovascular structure of the lesion tissue.

[0078] Based on the neurovascular structure, use the ant colony optimization algorithm to optimize the preset surgical path and generate the target surgical path.

[0079] Among them, the target surgical path does not intersect with the neurovascular structure.

[0080] In this embodiment, the ant colony optimization algorithm is initialized, and relevant parameters of the ant colony optimization algorithm are set, such as the number of ants, the pheromone evaporation coefficient, the pheromone enhancement coefficient, etc. On the basis of avoiding key tissue sites such as neurovascular structures, when randomly placing ants at the starting point or near the starting point of the preset surgical path, the ants select the next node to reach according to the pheromone concentration and neurovascular structures at the current position with a certain probability. During the search process, the ants will leave pheromones to mark the paths they have passed, and other ants will be attracted by the paths with pheromones left when choosing paths, thus tending to choose these paths. When an ant completes the search of a path, the pheromones on the path are updated according to factors such as the length and safety of the path, and shorter and safer paths will obtain higher pheromone concentrations, thereby attracting more ants to choose these paths.

[0081] Repeat the above process of path search and pheromone update until the preset number of iterations is reached or a surgical path that meets the requirements is found. Among all the searched paths, select an optimal path for further optimization, such as smoothing processing, avoiding sharp corners, etc. At this time, the optimized path obtained is used as the target surgical path.

[0082] In this embodiment, the lesion tissue in the semantic segmentation map is located through Slicer software to generate the neurovascular structure of the lesion tissue; based on the neurovascular structure, important neurovascular structures are avoided to reduce injuries and complications during the operation, and then the ant colony optimization algorithm is used to optimize the preset surgical path to generate a more reasonable and safe target surgical path, improving the safety and effectiveness of the operation.

[0083] See Figure 3 , which is a schematic flowchart of still another embodiment of a surgical simulation method provided by this application. As Figure 3 shown, on the basis of the above embodiment, a specific implementation manner of "projecting the three-dimensional virtual human model on the interactive screen in a polarized projection manner so that the three-dimensional virtual human model is stereoscopically displayed on the interactive screen" in S102 includes:

[0084] S1021: Based on the three-dimensional virtual human model, generate two polarized images of the three-dimensional virtual human model by using the polarized projection method.

[0085] Among them, the polarization directions of the two polarized images are different.

[0086] S1022: Display the two polarized images through the interactive screen so that the user synthesizes the three-dimensional virtual human model that is stereoscopically displayed in the brain from the two polarized images received by the left and right eyes.

[0087] In this embodiment, the three-dimensional virtual human model generates two polarized images of the three-dimensional virtual human model through a projector and a polarizing plate placed on the projector, and projects the two polarized images onto an interactive screen. Among them, the polarizing plate placed on the projector can transmit two polarized images of the model with different polarization directions to the user, and each lens of the polarized glasses can only receive a picture with one polarization direction. At this time, the user can wear polarized glasses so that the left and right eyes receive two groups of pictures, and finally synthesize a stereoscopic image through the brain.

[0088] It can be understood that the interactive screen can be a photon suspension screen, which can be viewed by using the method of naked-eye 3D or wearing 3D glasses. This screen can be arranged on the operating table, and the projection device is suspended above to project and display the three-dimensional virtual human model, and the user can perform interactive operations in front of the operating table.

[0089] In this embodiment, based on the three-dimensional virtual human model, two polarized images with different polarization directions of the three-dimensional virtual human model are generated through polarized projection. The two polarized images are displayed through the interactive screen. When these images are synthesized in the brain, a strong sense of three-dimensionality can be generated, making the three-dimensional virtual human model look more real. This sense of three-dimensionality helps the user better understand and perceive the structural form of the three-dimensional virtual human model. And it helps the subsequent user to interact with the three-dimensional virtual human model through gestures, touches or other interaction methods.

[0090] See Figure 4 , which is a schematic flowchart of another embodiment of a surgical simulation method provided by this application; as Figure 4 shown, on the basis of the above embodiment, the user interaction operation includes the user's gesture operation and the operation on the force feedback device. Then, a specific implementation manner of "responding to the recognized user interaction operation and performing a simulation operation on the three-dimensional virtual human model" in S103 includes:

[0091] S1031: Use a gesture recognition algorithm to recognize the user's gesture operation to obtain a target gesture operation.

[0092] In this embodiment, the gesture data of the user is collected through a somatosensory camera, and the collected gesture data is input into a trained gesture recognition algorithm model for real-time detection and recognition to obtain the corresponding target gesture action.

[0093] S1032: Map the target gesture operation to the corresponding simulation operation on the three-dimensional virtual human model to perform physical simulation on the three-dimensional virtual human model.

[0094] In this embodiment, according to the obtained target gesture action, based on the predefined mapping rules from gesture to action, the recognized gesture operation is converted into an instruction that can be understood by the three-dimensional virtual human model and input into the three-dimensional virtual human model, so that the model executes corresponding actions according to the instruction, such as joint bending, limb movement, etc., and physical simulation of the three-dimensional virtual human model is performed through a physics engine (such as PhysX, Havok, etc.).

[0095] It can be understood that the three-dimensional virtual human model and the somatosensory camera device SDK can be integrated in Unity 3D. A rigid body property and an elastic collision mechanism Mesh Collider are set on the surface of the three-dimensional virtual human model, and an Interaction Behaviour script is attached to it. At the same time, the Interaction Manager plug-in is imported, so that this system can recognize postures such as hands outstretched, fists clenched, and fingers flexed, so as to achieve corresponding interaction effects when a virtual hand touches the three-dimensional virtual human model for operation within the Unity 3D interface.

[0096] S1033: In response to the user's operation on the force feedback device, perform corresponding simulation operations on the three-dimensional virtual human model.

[0097] Among them, the force feedback device is used to control the movement trajectory of the virtual surgical instrument.

[0098] In this embodiment, the force feedback device is used to control the movement trajectory of the virtual surgical instrument and the contact force feedback when the virtual surgical instrument touches the three-dimensional virtual human model. By importing the Open Haptics SDK in Unity 3D and combining it with the Haptics with Grabber component, the force feedback device takes over the corresponding virtual surgical instrument within the Unity3D interface, and sets the touchable property for the three-dimensional virtual human model, so that when the operator moves the link of the force feedback device, the corresponding virtual instrument moves in real time synchronously.

[0099] In this embodiment, using a gesture recognition algorithm to recognize the user's gesture operation to obtain the target gesture operation, mapping the target gesture operation to the corresponding simulation operation on the three-dimensional virtual human model for physical simulation of the three-dimensional virtual human model can achieve precise control of the three-dimensional virtual human model. At the same time, in response to the user's operation on the force feedback device, performing corresponding simulation operations on the three-dimensional virtual human model can instantly feel the feedback force of objects or models in the virtual environment, thereby improving the operation accuracy.

[0100] Further, on the basis of the above embodiment, a specific implementation manner of "in response to the user's operation on the force feedback device, perform corresponding simulation operations on the three-dimensional virtual human model" in S1033 includes:

[0101] In response to the user's operation on the force feedback device, motion parameters of the force feedback device are obtained.

[0102] In this embodiment, various operations of the user on the force feedback device are tracked and recorded in real time, including but not limited to position movement, rotation angle, applied force, etc. From the captured user operations, key motion parameters are accurately extracted, and these parameters will be used for subsequent physical simulation.

[0103] When it is determined that the force feedback device is in contact with the three-dimensional virtual human body model, a corresponding physical simulation is performed on the three-dimensional virtual human body model according to the motion parameters.

[0104] In this embodiment, when it is detected that the force feedback device is in contact with the three-dimensional virtual human body model, the motion parameters (such as position, rotation, force, etc.) obtained from the force feedback device are input into the physical engine, and the state of the three-dimensional virtual human body model is adjusted in real time. Then, based on the results of the contact detection and the simulation of the physical engine, a real contact response is generated, such as deformation, rebound or movement of the model.

[0105] In this embodiment, the force feedback device can accurately capture the user's operation actions, and by simulating the physical process of objects receiving force and generating reaction force in the real environment, the user can feel the force feedback when interacting with the virtual object during the operation. Then it is converted into motion parameters. When the force feedback device comes into contact with the three-dimensional virtual human model, the three-dimensional virtual human model is physically simulated according to these motion parameters, which can simulate real collision, deformation and other effects, thereby more accurately reflecting the user's operation intention, thereby improving the accuracy and precision of the interaction.

[0106] Furthermore, based on the above embodiment, the method may further include:

[0107] According to the result of the physical simulation, a mechanical feedback signal is generated, and the mechanical feedback signal is fed back to the force feedback device, so that the user can perceive the feedback force corresponding to the mechanical feedback signal through the force feedback device.

[0108] In this embodiment, the mechanical information such as reaction force and torque generated by the three-dimensional virtual human model during the force-bearing process is calculated based on the results of the physical simulation. These mechanical information are then converted into electrical signals or other transmittable signal forms, so that the converted mechanical feedback signals are transmitted to the force feedback device, so that the user can perceive the existence of the feedback force corresponding to the mechanical feedback signal through the force feedback device.

[0109] In this embodiment, the mechanical feedback signal generated through physical simulation can simulate the mutual forces between objects in the real environment, such as resistance, friction, etc., and accurately reflect the interaction situation between objects in the virtual environment. This simulation enables users to feel a more real touch when operating the force feedback device, thereby enhancing the user's interaction experience.

[0110] See Figure 5 , which is a schematic structural diagram of an embodiment of a surgical simulation device provided by this application. As Figure 5 shown, the device 50 includes an image processing module 51, an image projection module 52, and a user interaction module 53.

[0111] Among them, the image processing module 51 is used to perform image processing on multiple DICOM sequence images obtained to generate a three-dimensional virtual human body model; the multiple DICOM sequence images are medical image data of the same patient;

[0112] The image projection module 52 is used to project the three-dimensional virtual human body model onto an interactive screen by means of polarized projection, so that the three-dimensional virtual human body model is stereoscopically displayed on the interactive screen;

[0113] The user interaction module 53 is used to perform a simulation operation on the three-dimensional virtual human body model in response to the recognized user interaction operation.

[0114] The surgical simulation device provided by the embodiment of this application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so details will not be described here.

[0115] Further, on the basis of the above embodiment, the image processing module 51 is specifically used to perform the following operations on each sequence image in the multiple DICOM sequence images: perform semantic segmentation processing on the sequence image through a preset semantic segmentation algorithm to obtain the semantic segmentation map of the sequence image; call Slicer software to analyze the semantic segmentation map of the sequence image to obtain the target surgical path; based on each semantic segmentation map and the target surgical path corresponding to each semantic segmentation map, construct a three-dimensional virtual human body model through a multi-plane three-dimensional reconstruction algorithm; the three-dimensional virtual human body model contains lesion tissue information and a surgical planning path; the surgical planning path is determined according to the target surgical path corresponding to each semantic segmentation map.

[0116] The surgical simulation device provided by the embodiment of this application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, so details will not be described here.

[0117] Further, on the basis of the above embodiments, when the image processing module 51 analyzes the semantic segmentation map of the sequence images by invoking the Slicer software to obtain the target surgical path, the image processing module 51 is specifically configured to locate the lesion tissue in the semantic segmentation map through the Slicer software to generate the neurovascular structure of the lesion tissue; based on the neurovascular structure, use the ant colony optimization algorithm to optimize the preset surgical path to generate the target surgical path; the target surgical path does not intersect with the neurovascular structure.

[0118] The surgical simulation device provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, which will not be elaborated here.

[0119] Further, on the basis of the above embodiments, the image projection module 52 is specifically configured to generate two polarized images of the three-dimensional virtual human body model based on the three-dimensional virtual human body model by using the polarized projection method; the polarization directions of the two polarized images are different; the two polarized images are displayed through an interactive screen so that the user can synthesize the three-dimensional virtual human body model that is stereoscopically displayed in the brain with the two polarized images received by the left and right eyes.

[0120] The surgical simulation device provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, which will not be elaborated here.

[0121] Further, on the basis of the above embodiments, the user interaction module 53 is specifically configured to use a gesture recognition algorithm to recognize the user's gesture operations to obtain the target gesture operations; map the target gesture operations to the corresponding simulation operations on the three-dimensional virtual human body model to perform physical simulation on the three-dimensional virtual human body model; in response to the user's operation on the force feedback device, perform the corresponding simulation operation on the three-dimensional virtual human body model; the force feedback device is used to control the movement trajectory of the virtual surgical instrument.

[0122] The surgical simulation device provided by the embodiments of the present application can execute the technical solutions shown in the above method embodiments, and its implementation principle and beneficial effects are similar, which will not be elaborated here.

[0123] Further, on the basis of the above embodiments, when the user interaction module 53 is used to perform the corresponding simulation operation on the three-dimensional virtual human body model in response to the user's operation on the force feedback device, the user interaction module 53 is specifically configured to obtain the movement parameters of the force feedback device in response to the user's operation on the force feedback device; when it is determined that the force feedback device comes into contact with the three-dimensional virtual human body model, perform the corresponding physical simulation on the three-dimensional virtual human body model according to the movement parameters.

[0124] The surgical simulation device provided by the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, so details are not described herein again.

[0125] Further, on the basis of the above embodiment, the user interaction module 53 is further specifically configured to generate a mechanical feedback signal according to the result of the physical simulation, and feed the mechanical feedback signal to the force feedback device, so that the user can perceive the feedback force corresponding to the mechanical feedback signal through the force feedback device.

[0126] The surgical simulation device provided by the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, so details are not described herein again.

[0127] See Figure 6 , which is a schematic structural diagram of an embodiment of an electronic device provided by the embodiment of the present application. The electronic device 60 may include: a processor 61 and a memory 62.

[0128] Wherein, the processor 61 is communicatively connected to the memory 62, and the memory 62 is used to store computer execution instructions; the processor 61 is configured to execute the technical solution in any of the foregoing method embodiments by executing the computer execution instructions stored in the memory 62.

[0129] Optionally, the memory 62 may be either independent or integrated with the processor 61. Optionally, when the memory 62 is a device independent of the processor 61, the electronic device 60 may further include: a bus for connecting the above devices.

[0130] This electronic device is used to execute the technical solution in any of the foregoing method embodiments, and its implementation principle and technical effects are similar, so details are not described herein again.

[0131] The embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the above method, and its implementation principle and technical effects are similar, so details are not described herein again.

[0132] It should be noted that the various embodiments in this specification are described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description of the method embodiments. The apparatus embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components referred to as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0133] As described above, this is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A surgical simulation method, characterized in that: include: Performing image processing on the acquired multiple DICOM sequence images to generate a three-dimensional virtual human body model; the multiple DICOM sequence images are medical imaging data of the same patient; The three-dimensional virtual human body model is projected onto an interactive screen by means of polarized projection, so that the three-dimensional virtual human body model is displayed in stereo on the interactive screen; In response to the recognized user interaction operation, a simulation operation is performed on the three-dimensional virtual human body model.

2. The method according to claim 1, characterized in that The method of performing image processing on the acquired multiple DICOM sequence images to generate a three-dimensional virtual human body model includes: For each of the plurality of DICOM sequence images, the following operations are performed: Performing semantic segmentation processing on the sequence of images by using a preset semantic segmentation algorithm to obtain a semantic segmentation map of the sequence of images; Calling Slicer software to analyze the semantic segmentation map of the sequence images to obtain the target surgical path; Based on each of the semantic segmentation maps and the target surgical path corresponding to each of the semantic segmentation maps, a three-dimensional virtual human body model is constructed through a multi-plane three-dimensional reconstruction algorithm; the three-dimensional virtual human body model contains lesion tissue information and a surgical planning path; the surgical planning path is determined according to the target surgical path corresponding to each semantic segmentation map.

3. The method according to claim 2, characterized in that The calling of Slicer software to analyze the semantic segmentation map of the sequence image to obtain the target surgical path includes: Using Slicer software to locate the lesion tissue in the semantic segmentation map, and generate the neurovascular structure of the lesion tissue; Based on the neurovascular structure, the preset surgical path is optimized using an ant colony optimization algorithm to generate a target surgical path; the target surgical path does not intersect with the neurovascular structure.

4. The method according to claim 1, characterized in that: The method of projecting the three-dimensional virtual human body model onto an interactive screen by means of polarized projection so that the three-dimensional virtual human body model is stereoscopically displayed on the interactive screen includes: Based on the three-dimensional virtual human body model, two polarization images of the three-dimensional virtual human body model are generated by polarization projection; the two polarization images correspond to different polarization directions; The two polarized images are displayed through an interactive screen, so that the user can synthesize the two polarized images received by the left and right eyes into a three-dimensional virtual human body model for stereoscopic display in the brain.

5. The method according to claim 1, characterized in that The user interaction operation includes a user gesture operation and an operation of the user on a force feedback device, and in response to the identified user interaction operation, performing a simulation operation on the three-dimensional virtual human body model includes: Performing gesture recognition on the user's gesture operation using a gesture recognition algorithm to obtain a target gesture operation; Mapping the target gesture operation to a corresponding simulation operation on the three-dimensional virtual human body model to perform physical simulation on the three-dimensional virtual human body model; In response to the user's operation on the force feedback device, a corresponding simulation operation is performed on the three-dimensional virtual human body model; the force feedback device is used to control the motion trajectory of the virtual surgical instrument.

6. The method according to claim 5, characterized in that The step of performing a corresponding simulation operation on the three-dimensional virtual human body model in response to the user's operation on the force feedback device comprises: In response to a user's operation on the force feedback device, acquiring a motion parameter of the force feedback device; When it is determined that the force feedback device is in contact with the three-dimensional virtual human body model, a corresponding physical simulation is performed on the three-dimensional virtual human body model according to the motion parameters.

7. The method according to claim 6, characterized in that The method further comprises: A mechanical feedback signal is generated according to the result of the physical simulation, and the mechanical feedback signal is fed back to the force feedback device, so that the user can perceive the feedback force corresponding to the mechanical feedback signal through the force feedback device.

8. A surgical simulation device, characterized in that: include: An image processing module is used to process the acquired multiple DICOM sequence images to generate a three-dimensional virtual human body model; The multiple DICOM sequence images are medical imaging data of the same patient; An image projection module, used to project the three-dimensional virtual human model onto an interactive screen by means of polarized projection, so that the three-dimensional virtual human model is displayed in stereo on the interactive screen; The user interaction module is used to perform simulation operations on the three-dimensional virtual human body model in response to the recognized user interaction operations.

9. An electronic device, characterized in that: The device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images