Surgical training simulator based on immersive virtual reality

Abstract

The invention provides a system for virtual reality simulation of various procedures for training of practitioners in the field of health, characterized in that the system comprises: a mechanical interface arranged in a box, the mechanical interface comprising two control handles simulating surgical forceps, and an electronic system, the electronic system is operably connected to the display screen to generate a visual interface, and the visual interface comprises images synchronized with sports skill exercises; and a camera and a sensor providing real-time information about the location and direction of the user's action. According to the invention, the camera is arranged in the virtual reality helmet system and the sensor is arranged on the control handle, the camera and the sensor are operably connected to software, the software allows stereoscopic images to be generated from the images taken by the camera, and the visualization interface comprises a 3D object having a similar geometry to the biological or surgical object.

Description

Technical Field

[0001] The invention relates to the field of medicine or veterinary medicine and health, in particular to the field of computer-assisted surgery, or manipulators or robots specifically adapted for surgical operations, or simulators for the training of medical professionals. In particular, the invention provides an immersive and portable virtual reality simulator for the training of minimally invasive surgery, as well as for the preparation of students or professionals in open surgical techniques. Background Art

[0002] In the field of health education, an important part is the training of health professionals and students of health-related professions through virtual reality simulations. These simulations help to teach trainees how to interact in real surgeries, thus improving their skills, space and time related to real surgeries, without exposing patients to learning errors. In addition, students participating in such simulations are able to face different conditions, which helps improve their ability to clinically diagnose patients and make diagnostic conclusions, as researchers from the University del Desarrollo point out in a scientific paper (Remote simulation and remote debriefing for clinical reasoning improvement of medical undergraduates; Marco Ortiz-Arévalo, Trinidad Campussano-Schialer, Alvaro Tolosa-Villarreal, Adriel Marco, Soledad Armijo-Rivera, Joaquín Díaz-Schmidt et al.; Educación Médica; Volume 22, Issue 5, 2021.9-2021.10, Pages 283-286).

[0003] Several devices exist to help surgeons train for surgical interventions, to help them master certain operations that must be performed during real surgeries and, therefore, to improve their skills in these events. These devices have different mechanical and electronic configurations and, to some extent, help to visualize the objects and components used in the intervention or used during or during the procedure, but these existing devices focus on specific types of surgeries, such as simulators for teaching childbirth and neonatal care, as described by Giselle Riquelme in her work entitled "Integrating simulation technology into nursing teaching for childbirth and neonatal care" (Educación Médica Superior; Vol. 31, No. 4, March 10, 2017). However, many times, these simulations involve systems that are difficult to adjust, making the system less practical as a simulator and multidisciplinary for performing different types of surgeries in a single device.

[0004] Patent document with publication number CN106448403A describes an example of such a system for simulating surgical operations under virtual reality technology. The patent relates to a surgical training system for simulating thoracoscopic surgery, which allows surgeons to simulate various types of surgical operations. The technical solution adopted and described in the document is a training system for simulating thoracoscopic surgery, which includes a main body, a monitor, a simulated endoscope, simulated surgical instruments and a control system; the monitor is fixed to the main body through an arm, and the simulated endoscope and simulated surgical instruments are connected to the main body; the simulated endoscope and simulated surgical instruments are provided with rotation sensors and motion sensors, and the control system is electrically connected. The control system includes a data receiving and processing device, a 3D model processing server and a 3D platform system server.

[0005] On the other hand, patent document with publication number ES2346025A1 relates to a simulation system for surgical training, which has the technical characteristics of being designed to allow the training of trainee surgeons in situations that are as realistic as possible, without any simulation of material aging, and thus allowing unlimited repetitions in the same procedure. Trainee surgeons can not only observe a simulated situation of a real operation, but can also study any other aspect of this situation through different visualization modes. In addition, the system allows customized exercises by using patient data from diagnostic tests such as scanning, mapping and 3D CT, so that trainees can check the surgical status, decide the location of the incision, and practice surgical operations before performing actual surgery on the patient. The invention provides a solution for learning this technology, which is a crucial procedure in minimally invasive surgery.

[0006] However, existing technologies focus on improving various aspects of a specific surgery without considering the mobility of the system generating the simulation, ignoring the feasibility of training trainees in different locations and possible remote evaluation of trainees' performance, and do not have different configurations for training in different situations that may arise in actual surgery.

[0007] Therefore, there is a need to build a portable system that helps surgeons and students perfect the operation of surgical instruments and master the necessary operating skills to avoid mistakes in real operations, and this system can simulate various types of operations. Summary of the invention

[0008] The present invention provides an immersive virtual reality simulator system for minimally invasive surgery and other surgical operations, characterized in that the system comprises:

[0009] A mechanical interface enclosed in a carrying case for portability, the mechanical interface connected to a display screen to simulate a surgical procedure; wherein the mechanical interface includes two handles for simulating surgical forceps to immerse a user's senses; wherein the mechanical interface includes an electronic system operably connected to the display screen to generate a visual interface; wherein the visual interface includes images synchronized with the motor exercise;

[0010] A camera and a sensor that can provide position information and orientation information about a user's movements in real time, wherein the camera is disposed in a virtual reality helmet system, such as Oculus Quest or HTC Vive, and the sensor is mounted on a handle; wherein the camera and the sensor are operably connected to software to allow a stereoscopic image to be generated from pictures taken by the camera; and wherein the visualization interface has three-dimensional objects with geometric structures similar to biological objects and surgical objects that may be used in surgery.

[0011] In a preferred embodiment of the invention the virtual reality system is characterized in that said mechanical interface is contained in a portable case.

[0012] In a further preferred embodiment, the virtual reality system is characterized in that it comprises a system for inputting different possible surgical models.

[0013] In a further preferred embodiment, the virtual reality system is characterized in that the helmet comprises a supplement for acquiring a stereoscopic image of the camera.

[0014] In another preferred embodiment, the virtual reality system is characterized in that it comprises a web (WEB) architecture data storage system.

[0015] In another preferred embodiment, the virtual reality system is characterized in that the software transmits physical parameters such as position, direction, distance and speed of the simulated surgical forceps based on the user's manipulation of the handle.

[0016] In a more preferred embodiment, the surgical forceps system used in the present invention is characterized in that the rotation applied by the surgical forceps is based on a ball joint.

[0017] In another preferred embodiment, the virtual reality system is characterized in that it uses three-dimensional games and simulation environments to reproduce training scenarios detailed in a simulation engine such as Unity 3D or Unreal.

[0018] In another preferred embodiment, the virtual reality system is characterized in that a three-dimensional model is generated for the three-dimensional game and simulation environment in a three-dimensional graphics editor (such as 3DMax, Blender, Maya, and others).

[0019] In another different preferred embodiment, the virtual reality system is characterized by having a box containing a three-dimensional model simulated in a three-dimensional game and simulation environment.

[0020] In another preferred embodiment, the virtual reality system is characterized in that the size of the containing box will be related to the surgical procedure being simulated, such as, for example, the size of the containing box is 20cm*20cm*25cm for simulation of grasping and carrying objects expected to be carried out using surgical forceps.

[0021] In another preferred embodiment, the virtual reality system is characterized in that it comprises a system for presenting the results of the user's actions in a simulation, wherein the results are presented visually on a display screen or are sent via a network (WEB).

[0022] The present invention also provides a device for obtaining virtual reality simulation of various surgeries, characterized in that the device includes: a mechanical interface, which is connected to a display screen to visualize the simulation of the surgical process; wherein the mechanical interface includes two handles for simulating surgical forceps to immerse the user's senses; and wherein the mechanical interface stores information of the simulation performed in an electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A photograph of a simulator of the object of the present invention in use by surgical personnel in a training center.

[0024] Figure 2 An exploded view of a cutaway plane showing the construction of the component including the rotation of a surgical forceps type handle.

[0025] Figure 3 A longitudinal cross-sectional profile view showing the mechanical and geometrical configurations of a simulated mechanism of surgical forceps.

[0026] Figure 4 A schematic diagram showing the degrees of freedom of rotation of surgical forceps in a rectangular coordinate system.

[0027] Figure 5 The internal components contained in the glove box are shown, illustrating the protective foam layer and the couplings for the simulator controller.

[0028] Figure 6 Two photographs showing different angles of the portable simulator illustrate its components including: a carrying case, a virtual reality headset, and a mechanical interface for simulating the operation of surgical forceps attached to the headset controller.

[0029] Figure 7Shown are simulated visualizations of some elements used during the simulation, such as: surgical forceps, biological objects or surgical objects. DETAILED DESCRIPTION

[0030] The present invention describes in detail a system and a device, which provide technical tools including a system for virtual reality simulation of multiple surgical training. The system and device are mainly used by students of medical or other health-related professions to practice and evaluate their performance in surgery, and by health practitioners to train before each new operation, thereby reducing preparation time and improving the operation skills and movement characteristics of different surgical treatment processes. The system and device provide the possibility of shortening the adaptation time of surgeons for each operation, and due to its portability, training can be carried out in different locations, which is convenient for training at home, and does not necessarily have to be carried out in a hospital center. Figure 5 As shown, the technology is a box-like model (3d) containing an electronic system (1d) that can be connected to a display screen when in use, the box-like model being mounted on a protective foam (2d).

[0031] The advantage of the system provided in the present invention is its portability, since the system comprises a virtual reality device and an electromechanical interface coupled to a workbench attached thereto, the system can be transported from one place to another. Since the system comprises a WIFI network system sufficient to perform the training, the system does not comprise cables and the trainee can be mobile, and if there is no network connection, the system can work in autonomous mode, which allows to send the training data in a delayed manner.

[0032] The system also allows for curriculum incorporation for these surgical skills. Since the instructor is not directly involved in the training process, face-to-face instruction is not required. The system allows the trainee to complete each stage of training using a simulator, which provides feedback to the trainee and the course instructor. The use of real surgical instruments is also no longer required, as they will be virtually presented.

[0033] In particular, the immersive virtual reality simulator-type system for minimally invasive surgery of the present invention is characterized in that it includes a mechanical interface connected to a display screen for simulating surgical operations; wherein the mechanical interface includes two handles for simulating surgical forceps to immerse the user's senses therein; wherein the mechanical structure includes an electronic system, which is operably connected to the display screen to generate a visualization interface; wherein the visualization interface includes images synchronized with motion exercises; cameras and sensors, which are used to provide information about the position and direction of the user's movements in real time, wherein the camera is installed in a virtual reality helmet system (such as Oculus Quest) and the sensor is installed on the handle; wherein the camera and the sensor are operably connected to software to allow stereoscopic images to be generated based on images taken by the camera; and wherein the visualization interface has three-dimensional objects with geometric structures similar to biological objects and surgical objects used in surgery.

[0034] In the context of the present invention and by way of general description throughout the specification, a handle will be understood as a system of mechanical parts in which a user of the present invention interacts to simulate surgical forceps.

[0035] In a preferred embodiment the virtual reality system is characterized in that the mechanical interface is comprised in a portable case.

[0036] In the context of the present invention and by way of general description throughout the specification, a carrier box will be understood as a box containing all the elements of the system of the present invention, which can be moved and used wherever the user sees fit.

[0037] In the context of the present invention and by way of general description throughout the specification, a simulator assembly will be understood as a structure comprising a base that supports the overall mechanical structure of the forceps and the electronics connected to a display screen.

[0038] In the context of the present invention and by way of general description throughout the specification, protective foam will be understood as a foam with a variable geometry that is located inside the cargo box and protects the overall structure of the simulator assembly.

[0039] In a preferred embodiment of the invention, the virtual reality system is characterized in that it comprises a system for inputting models of different possible surgeries.

[0040] In another preferred embodiment, the virtual reality system is characterized in that it comprises a web (WEB) architecture data storage system.

[0041] In another preferred embodiment, the virtual reality system is characterized in that the software transmits physical parameters such as position, direction, distance and speed of the user.

[0042] In a more preferred embodiment, the surgical forceps system used in the present invention is characterized in that the rotation applied by the surgical forceps is based on Figure 2 The ball joint (5a) shown, wherein the shaft (2a) is connected via a ring (3a)(6a) to a supporting base plate (4a), which base plate comprises a fastening system with bolts (1a) and nuts (7a). Figure 3 A cross-section of the coupling at the lower part of the overall structure is shown, while at the upper part it is shown how the shaft (2a) is connected to the linear bearing (8a) for interaction with the user (controller), wherein the shaft at the upper part includes a cover (10a) and a fastening system (9a).

[0043] In another preferred embodiment of the present invention, the virtual reality system uses Unity 3D games and simulation environments (Ronghai Wang, April 6, 2017; A surgical training system for four medicalpunctures based on virtual reality and haptic feedback; 2017 IEEE Symposium on 3D User Interfaces (3DUI)) to fully utilize the convenience of combining graphic features and physical behaviors.

[0044] In another preferred embodiment, the virtual reality system has three-dimensional models of objects to use in the scene, all of which are made in 3D Max (Li Yaqin, June 22-24, 2010; The applying research for 3D mesh models watermarking based on 3D MAX; 2010 Second International Conference on Educational Technology and Computers). These models are set to the origin of the coordinate system, and their vertices are updated according to the proportions of the objects from the Xform modifier (Mikko Honkala, July 2006; Multimodal interaction with xforms; Proceedings of the Sixth International Conference on Network Engineering).

[0045] Check the input parameters of all models that will be loaded into the scene and configure the main scale factors where a model is assigned to avoid scale changes and animation calls.

[0046] In various preferred embodiments, the virtual reality system is characterized by having a box containing a three-dimensional model that is simulated in a three-dimensional gaming and simulation environment.

[0047] In a preferred embodiment, the box containing the virtual reality system of the present invention may have different sizes depending on the surgical procedure to be simulated, and in another preferred embodiment, Figure 7 As shown, the size of the box in the three-dimensional game and simulation environment is, for example, 20cm*20cm*25cm.

[0048] In a preferred embodiment, the virtual reality system is characterized in that it comprises a system for visually presenting the result or transmitting the result via a network (WEB).

[0049] The present invention also provides a device for obtaining virtual reality simulation of various surgeries, characterized in that the device includes: a mechanical interface connected to a display screen for visualization of simulated surgical operations; wherein the mechanical interface includes two handles for simulating surgical forceps to immerse the user's senses; wherein the mechanical interface is contained in a portable box; and wherein the mechanical interface stores the simulated information in an electronic device.

[0050] The product’s design will allow for the problem being solved through surgical training, which will help generate tools that are adaptable in use and transportation, helping to improve the motor and manipulation skills of health practitioners and students, thereby improving surgical outcomes.

[0051] Example of implementation

[0052] Example 1: Scene for presenting simulation

[0053] The simulator has a constant scene responsible for maintaining the objects throughout the simulation, two practice scenes, and one scene for presenting the results of the training episodes.

[0054] The constant scene contains the immersive camera system of the virtual reality helmet as a basic element, and connected to it are the left and right surgical forceps added in the same object hierarchy. This ensures that the virtual surgical forceps are always available when another practice scene is selected. The following will describe these basic elements of the simulator (Unity game object) one by one:

[0055] -OVRCameraRig: The tracking space of the VR headset, which has a VR camera that replaces the traditional camera of the Unity environment. It provides access to the interface with the hardware. You can see how in this same element a "sub-element" is set up from the correct controller to the pre-response controller containing the model of the correct surgical forceps.

[0056] -ClampsLogic: The object contains the necessary scripts for the operation of the clamps in relation to the operation of the scene's objects. It includes code developed in C# for: grabbing and dropping elements; and switching elements between clamps.

[0057] -GlobalSceneManager: Contains the controller code (scripts) for loading and downloading scenes. It manages the control of the exercise data storage in the server through the API developed for this purpose.

[0058] -NetworkManagerLogin: Responsible for using the API developed to perform the user registration and authentication process.

[0059] A coordination scene is generated to perform a first exercise, allowing the user to start getting familiar with the synchronization work between the directions of both hands and vision. It uses the environment of a containing box and incorporates five spheres of a first color and five spheres of a second color. The spheres of the first color will be "touched" by the left forceps and the spheres of the second color will be "touched" by the right forceps. There is also a sphere of a third color, and if either forceps touches it, the exercise ends, which indicates that a serious error has occurred. The behavior of the spheres inside the box is to move randomly in the horizontal plane. The scene also has a system for detecting errors, which stores how many times the spheres were touched with the wrong forceps, or if a serious error has occurred. In addition, the time of the exercise is stored.

[0060] The scenario of object grasping and transfer simulates one of the exercises most often used by surgical students and tested in the mechanical simulator. The scenario consists in placing a container in a cassette with five elements that must be taken out and placed in another target container located at the end of the cassette. For these procedures, algorithms were developed that allow grasping and transporting physical elements and exchanging elements between surgical forceps. The simulator is able to measure the following parameters:

[0061] - Distance travelled by the right hand (meters).

[0062] - Distance traveled by the left hand (meters).

[0063] - Economy of movement of the right hand (optimal distance, data provided by the teacher / distance travelled).

[0064] - Economy of movement of the left hand (optimal distance, data provided by the teacher / distance travelled).

[0065] - Average speed of the right hand (m / s).

[0066] - Average speed of the left hand (m / s).

[0067] - The position and orientation of the tip of the surgical forceps in each simulation frame.

[0068] The scenes showing the results are designed to allow the participants of the simulation to observe a summary of their actions during training. This information is collected from the network and displayed on the device through an arc interface to allow observation of the data. These development objects include:

[0069] -ResultsManager: Controls the display and filling of tables based on information obtained from the server. It includes the ResultsManager.cs component (script) developed to include these functions.

[0070] -CanvasTotalResults: Contains the tables that will be displayed. These tables include the Total Data table, the Coordination Results table, and the Results table for the Object Grasping and Transfer exercises. The last object that was expanded disabled the Results tables for Coordination and Grasping so that when displayed in this scene, only the Total Results will be shown. During the simulation, when the user chooses to visualize the activities corresponding to these exercises, these tables will be activated.

[0071] Example 2: Design of a mechanical interface for simulation of the rotational behavior of surgical forceps.

[0072] During the simulation, the movements of the surgical forceps must be mimicked to immerse the trainee's senses in the training process and bring them closer to how they will perform this activity during an actual surgical intervention. In this example, it is analyzed how many degrees of freedom exist in the process.

[0073] like Figure 4 As shown, at the entrance of the cannula needle (7c) in the abdominal cavity (4c), there is a moving fulcrum for the surgical forceps. During the braking process of the surgical forceps, three directions of rotation are maintained in space, the first rotation is around the longitudinal axis "Z" (roll) (1c), which allows the rotation of the surgical forceps, and the other two rotations are rotations (pitch and yaw) on the axis "X" (2c) and the axis "Y" (3c), which allow the surgical forceps to be supported and tilted at the joint. In addition, there are two other movements: one is the longitudinal movement, which represents the movement of the instrument, and the other is the opening of the surgical forceps, which allows grasping the elements in the cavity. In order to solve this problem, a ball connector (5a) is generated, which can be realized as follows Figure 4 The pitch and yaw rotations described in . It can be observed that the ball joint is contained in the support base and is fastened to the base by two rings (one located above and one located below). These rings are attached to the support base plate (4a) by screws (1a) and nuts (7a).

[0074] The same device controller is used to simulate opening and closing to perform the operation through software used to capture the motion, so the design focuses on solving the issues mentioned above for longitudinal movement and rotation.

[0075] Example 3: Design of a mechanical interface for simulation of the translational behavior of surgical forceps.

[0076] like Figure 4 As shown, the longitudinal movement is performed by moving the shaft of the forceps over the trocar (7c) to form a cylindrical connection. Many simulators simulate this process in a similar manner; however, since space must be left for the free movement of the forceps, the simulator is larger.

[0077] Since portability is a design requirement, in order to keep the size limited and avoid large parts such as forceps, the design concept is to move the hand instead of the forceps, i.e. the reverse process. In the present invention, no shaft or rod is introduced, but the simulated forceps are moved by the same shaft that supports the simulated forceps. In this way, a similar effect is obtained in the simulation, but the space used in depth is reduced relative to the original object.

[0078] from Figure 3 It can be observed how the axis (2a) moves upwards from the ball joint (5a) and how, in this case, the controller (2) simulating the support of the surgical forceps moves in longitudinal motion through this element to allow the surgeon to access any position in the virtual training space. Figure 3 In the figure, it can be seen how the linear bearing (8a) is connected between the shaft and the user's hand. This connection ensures the movement of the instrument in depth and the rotation around its axis (roll axis).

[0079] The forceps were modeled so that the simulated depth coincided with the maximum dimension of the shaft, and the area corresponding to the object grasping segment was constructed separately and integrated into the hierarchy as an additional component. The latter facilitated the subsequent rotation of the element relative to the fulcrum. These forceps also had collision zones added to allow the simulation of contacts with other elements, as well as the grasping and transport of these elements when practice required.

[0080] Example 4: Materials and construction systems within simulator components.

[0081] The handles that allow the movement and simulation of the surgical forceps used in surgical procedures are made of 8mm solid stainless steel rods, which ensure that they can withstand mechanical tensile stress or possible falls when mounted on the ball joint. The rod and ball joint are connected by threading, ensuring that the connection is neat and always remains in the same position. As the final element, the linear bearing should be considered to produce smooth and unobstructed sliding, thus simulating a normal working environment.

[0082] The fastening system of the VR headset controls is designed and built with three anchor points to ensure correct positioning. These anchor points are made of non-slip material to provide a better grip during the turns and translations that occur in daily use of the simulator.

[0083] The grip is designed to keep the controller's coordinate system aligned with the axes of the handle used for movement, and to ensure that no axis moves freely where the user applies force.

Claims

1. An immersive virtual reality simulator system for minimally invasive surgery and other surgical procedures, characterized in that: include: - a mechanical interface, which is connected to a display screen and is used to simulate a surgical procedure; - The mechanical interface includes two handles that simulate surgical forceps and immerse the user's senses; - the mechanical interface includes an electronic system operatively connected to a display screen to generate a visual interface; - wherein said visualization interface comprises images synchronized with the motor exercise; - a camera and a sensor, for providing position information and direction information of user movement in real time, wherein the camera is installed in the virtual reality helmet system, and the sensor is installed on the handle; - wherein the camera and the sensor are operatively connected to software to generate a stereoscopic image from images captured by the camera; as well as - wherein the visualization interface has three-dimensional objects with geometric structures similar to biological objects and surgical objects.

2. The system according to claim 1, characterized in that The mechanical interface is included in a portable case.

3. The system according to claim 1, characterized in that The system includes a system for inputting models of different types of surgeries.

4. The system according to claim 1, characterized in that The virtual reality helmet contains internal electronic supplement for stereoscopic images of the camera.

5. The system according to claim 1, wherein: The system includes a web (WEB) architecture data storage system.

6. The system according to claim 1, wherein: The software delivers information on physical parameters related to the position, direction, distance and speed of the simulated surgical forceps based on the user's operation on the handle.

7. The system according to claim 1, characterized in that The system uses three-dimensional games and simulated environments created by means of a simulation engine.

8. The system according to claim 7, characterized in that The system includes a three-dimensional model in the three-dimensional gaming and simulation environment.

9. The system according to claim 7, characterized in that The system also includes a box containing the three-dimensional game and the three-dimensional model in the simulated environment.

10. The system according to claim 9, characterized in that The receiving box has a size corresponding to the surgical procedure being simulated.

11. The system according to claim 1, characterized in that The system includes a system for presenting the results of the user's movements during the simulation, wherein the results are visually displayed on a display screen or transmitted via a network (WEB).

12. A device for simulating minimally invasive surgery through immersive virtual reality technology, characterized in that: The device includes: a mechanical interface connected to a display screen to visualize a simulation of a surgical procedure; wherein the mechanical interface includes two handles for simulating surgical forceps to immerse the user's senses; and wherein the mechanical interface stores information of the simulation performed in an electronic device.

13. The device according to claim 12, characterized in that The mechanical interface is included in a portable case.

Images