Medical training system and method for medical training

Abstract

A medical training system comprises a simulator assembly configured to perform at least one medical procedure by using an instrument in a simulated medical procedure setup, the simulator assembly being manually operated by a user. The system further comprises a control unit providing the simulated medical procedure setup, monitoring at least a seeking path of the instrument moved by the user during the simulated medical procedure. The control unit is capable of differentiating between the seeking path and a retracting path and it uses only the seeking path of the instrument in order to assess a proficiency of the user. The inventive medical training system enables to assess the proficiency of a user with a high level of accuracy.

Description

[0001] TITLE

[0002] MEDICAL TRAINING SYSTEM AND METHOD FOR MEDICAL TRAINING

[0003] TECHNICAL FIELD

[0004] The present invention relates to a medical training system for medical procedures, especially for surgical operations, and a method for medical training.

[0005] PRIOR ART

[0006] Medical procedures, especially surgical operations, such as endoscopy, arthroscopy, laparoscopy, and other minimally invasive surgery applications, can be trained in medical training systems, also called medical training simulators, which simulate the medical procedure setup. The users, namely trainees, physicians, and surgeons, learn to master the indirect hand-eye coordination required by the manipulation of medical instrumentation, such as endoscope or an ultrasound probe, in addition to the conventional medical instruments and procedures. Computerized medical training simulators enable the users to develop and improve their practice in a virtual reality environment before practicing in the real world-operation room.

[0007] Box Trainers typically consist of a box with at least one insert designed to mimic specific anatomical features or surgical scenarios. The inserts are tangible and represents, for example, cavities, incisions, or pathways for tangible instruments to navigate. The Box Trainers are used for training specific skills, such as performing specific movements. WO 2018 / 209274 A1 and US 10 902 745 B2 describe such Box Trainers.

[0008] In some embodiments, medical training systems provide a mixed reality scenario where the user jointly interacts with real objects in a physical environment and in a related virtual environment. These simulators include a screen for displaying the virtual reality and a human anatomy model in real size, such as a joint model or an organ model, as well as a tangible medical procedure instrument. The model is adapted with sensors and mobile members for guiding, tracking, and controlling the medical instrument operation within the anatomy model. Such medical training systems are for example disclosed in WO 2014 / 041491 A1 , US 2015 / 0325151 A1 , and WO 2020 / 164829 A1.

[0009] The historical type of surgery, making a relatively large incision in a patient to access the surgical site, is nowadays often replaced by minimally invasive (including robotic) surgery allowing a surgeon to perform procedures through relatively small incisions. In telesurgery, the surgeon uses some form of remote control to manipulate surgical instrument movements rather than directly holding and moving the instruments by hand. The surgeon is provided with an image of the surgical site at a remote location. While viewing typically a three-dimensional image of the surgical site on a suitable display, the surgeon performs the surgical procedure on the patient by manipulating master control input devices, which in turn control the motion of robotic instruments. This enables small, minimal invasive surgical apertures to treat tissues at surgical sites within the patient. An example of such a teleoperated surgery system is well-known under the name da Vinci® Surgical System, and it is for example described in EP 2 884 935 B1 .

[0010] Some medical training systems provide training in the use of such teleoperated medical treating systems. For example, some medical training systems can be coupled to a surgeon console instead of the actual other system components, to provide a surgeon with a simulation performing the procedure. With such a system, the surgeon can learn how simulated instruments respond to manipulation of the console controls, i.e. , of the master control input devices. Instead of real-world pictures, the user sees a virtual operation site and a simulated medical procedure setup. Such a medical training system is for example described in EP 3 084 747 B1.

[0011] Other training systems providing training in the use of teleoperated medical treating systems comprise a surgeon control with master input devices as a stand-alone-solution, i.e., without being coupled to a real teleoperated medical treating system. The RoboS of the applicant is such a robotic surgery simulator, allowing independent learning by mirroring surgery robotic consoles.

[0012] Magdalena K. Chmarra et al. (2008), Retracting and seeking movements during laparoscopic goal-oriented movements. Is the shortest path length optimal? Surg Endosc 22: 943-949, state that minimally invasive surgery (MIS) requires a high degree of eye-hand coordination from the surgeon. They examined goal-oriented movement in a Box Trainer, where they divided the goal-oriented movement into a seeking phase and a retracting phase. They came to the conclusion that taking the shortest path length is not always the optimal movement. They also found that novices are less efficient in the seeking phase than experts. They emphasized that the retracting phase is very important in the MIS, since it improves safety by avoiding intermediate tissue contact. They therefore recommended analyzing motions in both phases, the seeking path and the retracting phase.

[0013] SUMMARY OF THE INVENTION

[0014] It is an object of the present invention to provide an improved medical training system for medical procedures, especially for surgical operations, and an improved method for medical training.

[0015] This object is achieved by a medical training system with the features of claim 1 , a method with the features of claim 22 and a non-transitory machine readable medium according to claim 25.

[0016] The inventive medical training system comprises a simulator assembly configured to perform at least one medical procedure by using an instrument, preferably a tangible and / or a virtual instrument, in a simulated medical procedure setup, the simulator assembly being manually operated by a user. The system further comprises a control unit providing the simulated medical procedure setup, monitoring at least the seeking path of the instrument, preferably the tangible and / or the virtual instrument, moved by the user during the simulated medical procedure. The control unit is capable of differentiating between the seeking path and a retracting path of the instrument, preferably the tangible and / or the virtual instrument, and the control unit is capable of using only the seeking path of the instrument, preferably the tangible and / or the virtual instrument, in order to assess a proficiency of the user.

[0017] Preferably, the simulator assembly is a tangible simulator assembly. Preferably, the instrument is a tangible or virtual instrument.

[0018] In preferred embodiments, the system monitors more than the seeking path, preferably the seeking and the retracting path.

[0019] The seeking path is the way taken by an instrument until it reaches a place or region of interest. The retracting path is the way taken by an instrument when leaving the place or region of interest. This retracting path can also be a path which has not the purpose to go back to a starting point so that it can be a remaining path as well. The retracting path can also be called retracting or remaining path

[0020] The place or region of interest is herein called also point of interest. It can be a point, a region, a sphere or any other defined place in the three-dimensional space. The place or region of interest is preferably the place where a specific task has to be fulfilled or another important place.

[0021] The seeking phase and the retracting phase is the time needed for the respective path. The retracting phase can also be terminated when a new seeking path is entered and therefore, when a new seeking phase is initiated. One seeking phase and one retracting phase define a seeking-retracting phase. The seeking path and / or the retracting path may lay within a human body or within a facsimile surgical volume or they may be external to the body or volume.

[0022] The seeking path can be the entire way the instrument, preferably of the tangible instrument and / or the virtual instrument, takes until it has fulfilled the task. It can also be only a part of this way. The seeking path is not the way the instrument takes after fulfilment of a specific task. However, this way can be a new seeking path for a new assessment, in case a next task has to be fulfilled or a next important place must be reached.

[0023] By assessing the seeking path only, the assessment is improved. By removing or by not including or by not recording the data of the retracting path, which is similar for an expert and a novice user, the result is therefore not diluted. The quality and fidelity of the assessment is therefore improved.

[0024] The seeking path to be assessed may be displayed on a screen allowing the user to receive real-time feedback.

[0025] In preferred embodiment, the training system defines an initial point and a final point and therein the seeking path comprises a first end and a second end, wherein the initial point defines the first end and the final point defines the second end.

[0026] Preferably, the initial point is a starting point of a movement of the instrument, preferably of the tangible instrument and / or of the virtual instrument, and / or the final point is a point where a given task is fulfilled by the instrument.

[0027] Preferably, the medical training system defines a volume within the final point is positioned and a point of origin at the instrument, preferably at the tangible instrument and / or at the virtual instrument, wherein the initial point is defined as point where the point of origin enters the volume.

[0028] In some embodiments, the final point, the initial point, the volume and the point of origin are defined in the real environment, i.e. in the real world, by using sensors of the training system. In addition or alternatively, the final point, the initial, the volume and the point of origin are defined in the virtual environment, i.e. in the virtual world.

[0029] Preferably, the volume is a sphere and / or the point of origin is a sphere, especially in the virtual world.

[0030] In some embodiments, the volume and / or the point of origin is hard-coded. In other embodiments, the volume and / or the point of origin is variable by the training system based on previous assessments and / or by user setting.

[0031] In some embodiments, the control unit can determine the length of the seeking path. This can be done with well known-means, for example using the semi-implicit Euler method, using velocity and then position, starting with initial conditions and calculating the state at each step, building the trajectory point by point calculating the forces and accelerations at the point. Alternatively or additionally, the control unit is capable of determining the time to cover the seeking path, i.e. the duration of the seeking phase, this means the time needed for the instrument, preferably for the tangible and / or for the virtual instrument, to reach the final point and to fulfil a given task. Alternatively or additionally, the control unit is capable of monitoring the alignment and orientation of the instrument, preferably of the tangible and / or of the virtual instrument, on the seeking path. In preferred embodiments, the control unit is capable of detecting and recording the seeking path in the three-dimensional space.

[0032] In some embodiments, only virtual instruments are present, wherein at least one finger of the user acts as the tangible instrument. This is achieved by the trainee interacting directly with a touchscreen of the training system, thereby moving a virtual instrument displayed on the screen with his finger or fingers in a virtual surgical setting. Preferably, the tangible instrument comprises at least one sensor, herein called input sensor. Preferably, the control unit is capable of acquiring position data from the at least one input sensor of the tangible instrument.

[0033] In preferred embodiments, at least one sensor detects the position of the tangible instrument, wherein a virtual instrument is present moving and acting according to the movement and action of the tangible instrument detected by the at least one sensor. In preferred embodiments, an initial point and a final point are detected with regard to the virtual instrument, i.e. in the virtual world, wherein the initial point and the final point define the first end and a second end of the seeking path. In other embodiments, at least one of the initial and final points are detected with regard to the tangible instrument, i.e. in the real world, for example by determining a point passed by the tangible instrument and when the task is also fulfilled in the real world. The task can for example be a placement of a real object into a special area. In other embodiments, the initial and final points detected can be within the way to the final task, where the remaining way is not considered as seeking path for the assessment made but it is actually the retracting path for this procedure or scenario. Part of the remaining path however can be used as seeking part in another assessment.

[0034] In some embodiments, the retracting path is monitored as well but analysed in a separate step and assessed separately from the seeking path. In preferred embodiments, the control unit does not assess the retracting path. In some embodiments, the control unit does not record the retracting path.

[0035] The medical training system may be any type of system for training medical procedures, especially surgical operations. In preferred embodiments, the medical training system is a simulator for minimal invasive surgery, preferably for robotic surgery, and / or for laparoscopy.

[0036] Preferably, the control unit comprises a memory, preferably a database, with data concerning at least one optimal seeking path. More preferably, the control unit comprises a memory, preferably a database, with a set of optimal seeking paths. The control unit is capable of comparing data of the monitored seeking path with the data of the at least one optimal seeking path or the set of optimal seeking paths in order to assess the proficiency of the user. Preferably, the control unit interprets input data of at least one of a plurality of input sensors, for example of an electromagnetic sensor or of an optical sensor, such as a camera. The control unit preferably applies at least one seeking-retracting phase determination algorithm to at least one of the data subsets, which differentiates between the seeking phase and the retracting phase. In one embodiment, this seeking-retracting phase determination algorithm generates at least one modified data subset, which contains the data of the seeking phase. In a further embodiment, the data subset contains only the seeking path. In another embodiment, the data is modified by applying a label indicating that the data is from the seeking phase. In this embodiment, the data set can include other data from the input sensors. In an additional embodiment, the seeking path trajectory is calculated within a defined volume of a sphere. The path length of the seeking trajectory is determined numerically, enabling precise measurement of the instrument's movement within the sphere. As an option, the control unit may then compare the at least one modified data set or subset to a set of seeking paths data stored in a database and / or data storage and / or code, i.e. the stored set of optimal seeking paths. As an option, at least one proficiency assessment algorithm is applied between the modified data set or subset and the stored set of optimal seeking paths, in order to assess the proficiency of the user.

[0037] In preferred embodiments, the data of the optimal seeking path is an optimal path length and / or an optimal time to cover the optimal seeking path and / or for an optimal orientation of the instrument, preferably of the tangible and / or of the virtual instrument, on the optimal seeking path.

[0038] In some embodiments, a scoring unit is employed wherein a maximum score is assigned to a predefined optimal path length. A minimum score is assigned to a path length that exceeds a designated threshold. The maximum score can be determined through various proficiency scoring methods. For example, the maximum score may correspond to the median of path lengths achieved by expert users, the upper quartile of expert scores, or a value derived through a contrasting groups' standard-setting method, allowing for consequences analysis to ensure fairness and accuracy in proficiency assessment. This approach ensures that the scoring unit reflects both the precision and efficiency of the user’s performance, offering a robust framework for evaluating skill levels in the simulated medical procedure.

[0039] In some embodiments, the control unit uses video recording as the input sensor. In some embodiments, the control unit uses input from human interaction as part of the differentiation algorithm, comparison method, or the proficiency assessment algorithm. In some embodiments, the assessment algorithm is trained data from an artificial neural network.

[0040] In some embodiments, the medical training system is simulating a teleoperated medical treating system, for example one of the systems mentioned in the introductory part of this text, especially the da Vinci® System. The medical training system in these embodiments preferably comprises a surgeon control with master input devices which manipulate simulated virtual instruments of the simulated teleoperated medical treating system.

[0041] In other embodiments, the medical training system is a mixed reality scenario medical training system enabling a user to interact with real objects in a physical environment and in a virtual environment, especially as described in the introductory part of this text. For example, it is one of the training systems built and sold by the applicant since years.

[0042] In some embodiment, the control unit comprises a web-based application, wherein the training system includes web-based data and software used for performing training scenarios and for assessing the seeking path.

[0043] The inventive method assesses a proficiency of a user of a medical training system as mentioned above. The control unit monitors a seeking path of the instrument, preferably of the tangible and / or the virtual instrument, moved by the user during the simulated medical procedure and the control unit compares the monitored seeking path with a stored optimal seeking path in order to assess the proficiency of the user. The control unit interprets the data of the at least one input sensor relaying parameters of the tangible or the virtual instrument moved by the user during the simulated medical procedure. The control unit applies at least one seeking-retracting phase determination algorithm, and at least one proficiency assessment algorithm to a set of stored seeking paths in order to assess and generate a proficiency metric of the user.

[0044] Preferably, the control unit determines the length of the seeking path and compares the determined length with at least one stored length of the optimal seeking path. Alternatively or additionally, the control unit determines the time to cover the seeking path and compares the determined time with at least one stored time of the optimal seeking path. Preferably, a robotic end-effector handling is simulated. In other variants, a minimal invasive surgery, especially a laparoscopy, is simulated. The inventive non-transitory machine-readable medium comprises a multiple of machinereading instructions which when executed by one or more processors associated with the medical training system and to perform the method described in this text.

[0045] The medical training system and method are preferably designed for deployment in hospitals, medical training centers, and research facilities. Preferably, they are adaptable for application in diverse medical disciplines, including general surgery, gynecology, urology, and beyond. In some embodiments, the medical training system and method serve as tools for medical device simulation, thereby facilitating the testing and refinement of surgical instruments and robotics in controlled virtual environments.

[0046] Further embodiments and variants of the invention are laid down in the dependent claims.

[0047] BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Preferred embodiments of the invention are described in the following with reference to the drawings, which are for the purpose of illustrating the present preferred embodiments of the invention and not for the purpose of limiting the same. In the drawings,

[0049] Figure 1 shows a mixed reality scenario medical training system according to a first embodiment of the invention,

[0050] Figure 2 shows a measurement of an instrument path according to the state of the art;

[0051] Figure 3 shows a measurement of an instrument path according to the invention;

[0052] Figure 4 shows a medical training system simulation a teleoperated medical treating system, also called robotic surgery simulator, in a schematic representation;

[0053] Figure 5a shows a box trainer comprising a portable smart device in a schematic representation with a tangible instrument in use;

[0054] Figur 5b shows the box trainer of figure 5a with a finger of the user being used as tangible instrument; Figure 6 shows a schematic representation of an image displayed on a schematic screen of a medical training system during a performance of a task with an instrument;

[0055] Figure 7 shows a schematic representation of an image displayed on the screen of the medical training system when performing the task according to figure 6, wherein the seeking path used is visualized;

[0056] Figure 8 shows a part of a real image displayed on a real screen when performing the task of figure 6 in an early stage of performance;

[0057] Figure 9 shows a part of a real image displayed on the real screen when performing the task of figure 6 at a later stage of performance;

[0058] Figure 10 shows a part of a real image displayed on the real screen at an even later stage of performance;

[0059] Figure 11 shows a part of a real image displayed on the real screen at an even later stage of performance;

[0060] Figure 12 shows a part of a real image displayed on the real screen shortly before completing the task of figure 6;

[0061] Figure 13 shows a part of a real image displayed on the real screen shortly after having completed the task of figure 6;

[0062] Figure 14 shows a part of a real image displayed on the real screen after having retracted an instrument after having completed the task of figure 6;

[0063] Figure 15 shows a part of a real image displayed the real screen after having completed the task of figure 6 in an enlarged and rotated view;

[0064] Figure 16 shows a part of an image displayed on a screen of a medical simulator displaying an anatomical scenario in a first stage of medical treatment and

[0065] Figure 17 shows the part of figure 16 in a second stage of medical treatment. DESCRIPTION OF PREFERRED EMBODIMENTS

[0066] The training system shown in figure 1 is a mixed reality scenario medical training system. The basic elements are known in the state of the art. The mixed reality scenario medical training system comprises a main body with a control unit 1 , preferably arranged within a main body, a display 2, a human anatomy model 3 in real size and tangible instruments 40, 41.

[0067] The human anatomy model 3 is usually representing only a specific part of a human body. In figure 1 , an insufflated abdomen is shown for laparoscopy training.

[0068] In the example shown, the tangible instruments to be held in the hands 40 correspond to real instruments used in laparoscopy. The pedals 41 are used for activation of electrocauterization elements and other device mechanism controls. In other mixed reality scenario medical training systems, the tangible instruments 40, 41 are other types of medical instruments.

[0069] The tangible instruments 40, 41 have input sensors 44, 45 implemented within, for the acquisition of position, orientation, other degrees of freedom, and relative position to the other instruments and the anatomical body 3. The anatomical body 3 also contains input sensors 44, 45 within for acquisition of parameters. The input sensors connect to the control unit 1 for data acquisition. In figure 1 , some exemplary placements of sensors 44, 45 are shown.

[0070] The control unit 1 controls the virtual views displayed on the screen or display 2 based on the activities of the user, i.e. , his use of the tangible instruments 40, 41. The virtual views are generated by the values of the input sensors in tangible instruments 40, 41 , and anatomical body 3 that are controlled and manipulated by the user. On the screen or display 2, a real or a virtual picture of the human anatomical part is shown as well as the position and movements of the instrument activated by the trainee or user. Reference 43 in figure 1 refers to a virtual instrument visualizing the movement of the tangible instrument 40 on the screen 2.

[0071] The control unit 1 has also input means for user input, such as his information about his skills and experiences. The input means are not shown in figure 1.

[0072] The control unit 1 has means of acquiring input data. The control unit 1 has a memory for storing data, for example means of storing data sets for an optimal path and for data sets obtained from the sensors. In some embodiments, the control unit 1 is connectable to a cloud. In these embodiments, the control unit 1 may not comprise an own memory.

[0073] Instead of a mixed reality scenario medical training system, a training system simulating a teleoperated medical treating system can also be used, especially a robotic surgery system, such as a system simulating for example the well-known da Vinci® system and / or the well- known MMI Symani® Surgical System.

[0074] Figure 4 shows such a training system simulating a teleoperated medical treating system in a schematic representation. This system comprises a control unit 1 located in a main body, a screen or display 2 and tangible instruments, such as handheld instruments 40 and pedals 41 , as well. Usually, no tangible human anatomy model 3 is present. Contrary to the mixed reality scenario medical training system, the tangible instruments have not the same shape or size as the instruments used on the patient during robotic surgery. The tangible instruments of the training system correspond to the master control input devices of the teleoperated medical treating systems or they are at least similar to them providing the same or at least some of the functionalities of the master control input devices.

[0075] Figures 5a and 5b show a Box Trainer. The Box Trainer comprises a main body 10 with at least one through-opening 42 leading into an interior of the main body 10. At least one tangible task place 46 is arranged within this interior.

[0076] In the embodiment shown, a tablet 20 is arranged on top of the main body 10 of the Box Trainer. At least one tangible instrument 40 is connected with a wire 400 with the tablet 20. In addition or alternatively, wireless connection is possible as well.

[0077] At least one sensor 44 is located in or at the at least one through-opening 42 and / or at the at least one tangible task place 46. At least one camera 45 of the tablet 20 is capable of viewing into the interior of the main body 10. Additional cameras or other optical sensors may be present at other places, detecting the interior or the outside of the Box Trainer as well. The tablet 20 with its touchscreen allows input means for user input and as control unit of the Box Trainer. The tablet 2 also displays the virtual instrument 43 visualizing the movements of the tangible instrument 40 as well as virtual task places 47 referring to the tangible task places 46. The tablet 2 also provides the control unit of the training system.

[0078] In other embodiments, the Box Trainer comprises a screen, a control unit and user input means instead of the tablet.

[0079] As shown in figure 5b, a program can be run on the tablet 20 which allows to replace the tangible instrument 40 shown in figure 5a by the user's finger or fingers. The user can move his finger or his fingers on the touchscreen in order to activate, for example to move, the virtual instrument in the virtual medical scenario displayed on the touchscreen. The finger itself is in this case the tangible instrument and the touchscreen the at least one sensor.

[0080] The tablet can also be used in the above-mentioned mixed reality scenario medical training system instead or in addition to the screen.

[0081] The tablet can also be used as a stand-alone device for medical training.

[0082] The combination of the tablet with the above mentioned physical medical training systems, especially with the Box Trainer, provides a flexible and portable solution for surgical training, combining tangible surgical trainer inserts with digital interaction. This versatile design provides a hybrid surgical training platform. It utilizes traditional tangible surgical instruments and allows for direct finger interaction on the touch screen, therefore adapting to various training needs and providing a portable, efficient training solution for both novice and advanced users.

[0083] The different types of training systems mentioned above are well known in the state of the art. They use tangible instruments such as handheld surgical tools, pedals or master control input devices, which are equipped with sensors 44, 45 to monitor position, orientation, and movement when manipulated by a user. When anatomy models 3 are used, they are preferably equipped with sensors 44, 45 as well. In some embodiments, additional sensors are present. The combination of the sensors of the entire system enable to detect interactions of the tangible instruments with the models.

[0084] These medical training systems foster skill acquisition through visual feedback, often giving realistic tactile feedback and often providing users with training scenarios that mimic real world surgical environments.

[0085] The control unit 1 of the different training systems mentioned above preferably provides different exercises to train the skills of the user. The user can be a trainee or an experienced person who tries out new treating methods or who wants to define the best way to perform a known method in an upcoming surgery or treatment. The user identifies himself when starting the training system. During use of the system, the control unit 1 will preferably assess the performance of the user in the completed exercises or at least track and / or document the paths taken during the performance.

[0086] During performance of the exercises, the training system act in the well-known way. This means that the movements of the real or virtual instruments activated by the user are detected, that time is determined, and that other criteria are monitored by the control unit. In the training systems, at least the time used for taking at least a part of the path and / or the length of at least a part of the path taken is monitored.

[0087] In the state of the art, when assessing the skill of a user when moving an instrument within a human body, the seeking path and the retracting path is determined and assessed. This is shown in figure 2. In the inventive medical training system and with the inventive medical training method however, only the seeking path is assessed, as can be seen in figure 3. This will be described later in the text in more detail.

[0088] The tangible instruments 40, 41 used in the training system are for example laparoscopic tools, like cameras, graspers, or scissors, or robotic surgery consoles replicas comprising master control input devices, emulating as consoles of a teleoperated medical treating system, such as the da Vinci® Surgical System or the MMI Symani® Surgical System.

[0089] The software component of the training system is powered by an engine, such as Unity or Unreal. The engine delivers highly accurate physical simulations and rendering models. The engine supports detailed anatomical visualization and dynamic interactions between the virtual instruments and simulated virtual tissues, allowing for realistic deformation, collision detection, and force feedback. These features are preferably further enhanced by physics libraries such as NVIDIA PhysX, enabling lifelike replication of surgical environments.

[0090] Preferred embodiments of such inventive training systems comprise advanced sensor technologies for tracking movements of the tangible instruments 40, 41. For example, the training system comprises Aurora® electromagnetic sensors from Northern Digital Inc (NDI), renowned for their sub-millimeter spatial accuracy and rapid response times. These electromagnetic input sensors 44 provide high-resolution tracking data critical for precise instrument navigation within the simulated environment. In addition, or alternatively, some preferred embodiments comprise optical sensors 45 for tracking the movements of the tangible instruments 40, 41. For example, at least one camera comprising optical input sensors 45, such as an Intel® RealSense camera, is used. The camera with the optical sensors 45 is capable of real-time video capture and analysis. The use of at least one camera with optical sensors 45 augments the training system’s ability to evaluate orientation and spatial movement, ensuring comprehensive data collection for assessment of the proficiency of the user. The sensors 44, 45 are placed at and / or in and / or near the tangible instruments 40, 41 and / or the human anatomy body.

[0091] The electromagnetic sensors 44 and optical sensors 45 provide high-resolution tracking data to map at least the tangible instrument’s 40, 41 seeking path in three-dimensional space. The system identifies the seeking path as the trajectory from the instrument's starting point to a predefined target, such as a specific anatomical region or surgical site. As mentioned above, the seeking path is a or the critical component of the training system's proficiency assessment, i.e. the assessment of the user's proficiency.

[0092] The control unit 1 processes the seeking path using algorithms that differentiate between the seeking and retracting phases of the movement of the tangible instruments 40, 41 . These algorithms apply a seeking-retracting phase determination methodology, which isolates the data corresponding to the seeking phase while excluding or labeling data from the retracting phase. By focusing exclusively on the seeking path, the system eliminates noise introduced by the retracting path, ensuring a precise assessment of user skill and / or providing targeted training for specific upcoming surgeries.

[0093] The seeking path is further analyzed to calculate its trajectory, which is determined within a defined spherical volume. The path length is numerically calculated using established computational methods, such as the semi-implicit Euler method or equivalent trajectory modeling techniques. This calculation builds the seeking path point by point, accounting for velocity, position, and acceleration data captured from the tangible instrument.

[0094] The algorithm assigns a proficiency score to the calculated path length based on predefined benchmarks. A maximum score is assigned to an optimal path length, typically determined through methods such as the median or upper quartile of expert path lengths, or through contrasting groups' standard-setting methods that include consequences analysis. Conversely, a minimum score is assigned to path lengths exceeding a defined threshold, ensuring that the scoring unit reflects both precision and efficiency.

[0095] In one variation, the scoring can be applied to individual seeking paths, providing granular feedback on each discrete task performed during the simulated procedure. In another variation, the scoring unit can assess an accumulation of seeking paths over multiple tasks or an entire session. This cumulative scoring approach evaluates the user’s consistency and overall proficiency, enabling a more comprehensive assessment of skill development over time. The scoring algorithm of the scoring unit can further integrate additional metrics, such as the time taken to complete the seeking paths, the alignment and orientation of the instrument, and the three-dimensional spatial characteristics of the trajectory.

[0096] By leveraging these advanced algorithms and flexible scoring methods, the training system ensures both high-fidelity evaluations of individual tasks and holistic assessments of performance across broader training sessions, optimizing preparation for both general and procedure-specific surgical training. Based on this analysis, an objective feedback is given to the user, wherein the feedback highlights deviations. In some embodiments, the control unit 1 offers corrective guidance in real time. In other embodiments, the control unit 1 just provides the analysis or offers suggestions for the next try, i.e. the next training session on the same task or on a similar task.

[0097] The control unit 1 is equipped with a database of optimal seeking paths, derived from expertlevel procedural data. The database may comprise optimal times for performing a path of a specific task and / or optimal length of a path for a specific task and / or optimal two- dimensional or three-dimensional paths and / or optimal instrument orientation for performing a specific task.

[0098] This database serves as a benchmark to evaluate the proficiency of users, including new or young medical personnel, against experienced professionals. Metrics such as path length, time, and instrument orientation are used to assess skill level, with scoring algorithms providing objective feedback on performance. This functionality supports the differentiation of skill levels, enabling targeted training interventions for various user groups. As mentioned above, the control unit 1 may comprise the memory and the entire software to perform the training scenarios and the assessment.

[0099] In some embodiments, the control unit 1 of the training system is a web-based application, eliminating the need for dedicated software installations. Trainees access the application through a standard web browser, ensuring compatibility with most modern devices. The application employs libraries such as OpenCV for computer vision tasks, enabling real-time recognition and processing of the physical environment captured by the device’s camera. This technology overlays high-fidelity virtual anatomical models, procedural guides, and interactive prompts onto the live feed of a physical anatomical model or surgical dummy.

[0100] Some of these embodiments are able to track real-world interactions using the device's camera and integrated algorithms. The training system identifies and monitors trainee actions, such as cutting physical materials, suturing synthetic tissues, or manipulating physical objects like clamps or retractors. For example:

[0101] • When a user cuts into a physical surgical pad, the system detects the incision location, depth, and angle, and overlays corresponding virtual visuals, such as blood flow or tissue layers being revealed.

[0102] • When sutures are placed on a physical model, the system tracks the position and tension of the suturing action, providing real-time feedback on accuracy and consistency.

[0103] In some of these embodiments, the seeking path is tracked and analyzed using a mobile device’s camera, such as the tablet 20, and integrated algorithms powered by OpenCV. The mobile app identifies the instrument's movement as it transitions from an initial position to a target area, such as a specific organ or incision site, within the mixed reality environment. The seeking path is visually represented on the device’s screen, with virtual overlays illustrating the trajectory and providing real-time feedback on accuracy.

[0104] Some of these embodiments of the training system employ advanced computer vision techniques to detect instrument movements and classify them into seeking and retracting phases. The seeking path data is analyzed for key performance metrics, including path length, efficiency, and movement precision. These metrics are compared against optimal seeking paths stored in the training system’s centralized database. For example, during a suturing simulation, the system evaluates the instrument’s path as it approaches the needle’s entry point, providing immediate feedback on alignment and trajectory. To enhance user interaction, the mobile app visualizes the seeking path with color-coded overlays that indicate areas of deviation or inefficiency. Haptic feedback reinforces proper technique, such as applying resistance when the instrument deviates from the optimal path. This focus ensures that the seeking path remains the primary driver of skill assessment across a variety of procedures, including open surgeries and minimally invasive techniques.

[0105] Manipulation within the mixed reality environment is achieved through the device's multipoint touch interface, such as the tablet's 20 touchscreen. Users can interact directly with the screen to perform virtual tasks, such as zooming in on anatomical details, manipulating tissue layers, or highlighting key surgical landmarks. The touch interface also supports gesture-based inputs, allowing for intuitive operations such as virtual suturing, knot tying, or retracting tissue. Haptic feedback from the device enhances the training experience by simulating tactile cues like tissue tension or instrument resistance.

[0106] The training scenarios offered by the training system cover a wide range of procedures, including both minimally invasive and open surgeries. For example:

[0107] • Open Surgery Scenarios: Trainees can practice suturing techniques, including continuous and interrupted sutures, using realistic physical and virtual representations of surgical threads and needles. The system evaluates metrics such as suture placement accuracy, tension consistency, and completion time.

[0108] • Minimally Invasive Procedures: Scenarios include laparoscopic navigation and robotic- assisted surgeries, where users manipulate both real and virtual instruments, with the system tracking and analyzing their movements.

[0109] • Emergency Procedures: Simulations of trauma surgeries and life-saving interventions, with varying levels of complexity to challenge trainees.

[0110] The training system leverages the mobile device’s spatial mapping capabilities, such as depth sensors or LiDAR (if available), to ensure precise alignment of virtual overlays with real-world objects. For devices without these advanced features, the system uses adaptive algorithms within OpenCV to estimate spatial relationships and accurately track real-world instrument interactions.

[0111] The web app incorporates a centralized database for tracking trainee progress and storing performance metrics. Metrics such as incision precision, path length, instrument efficiency, and suture quality are analyzed in real time and compared against expert-level benchmarks. Trainees receive immediate feedback via visual overlays, haptic cues, and auditory alerts, fostering iterative learning and skill refinement.

[0112] In the following, the assessment of the seeking path is described.

[0113] In figure 6, a medical training task is shown, which may be performed for example by a box trainer or a robotic surgery simulator as mentioned above. A user shall place a virtual object 30 into a recess 31 having the same shape as the virtual object 30. The training session requires that the user moves the virtual object 30 by using a tangible instrument, wherein the display 2 shows the virtual 30 being moved by a virtual instrument 43. The tangible instrument is one of the tangible instruments 40, 41 described above. Activation and / or movement of the tangible instrument 40, 41 causes the virtual instrument 43 of the medical training system to move as well, wherein the virtual instrument 43 holds the virtual object 30 and is capable of releasing the virtual object 30 into the virtual recess 31 when the tangible instrument 40, 41 is activated by the user accordingly. The tangible instrument 40, 41 can have the same shape as the virtual instrument 43 and / or it can hold a tangible object as well. The tangible object can have the same size and / or shape as the virtual object 30. Accordingly, a tangible recess can be present as well, preferably being like the virtual recess 31 . In other embodiments, no tangible object and no tangible recess are present and / or the tangible instrument 40, 41 is different as the virtual instrument 43 but provides the same functionalities as the virtual instrument 43.

[0114] The seeking path until the virtual object 30 is placed within the virtual recess 31 is detected and analyzed, preferably measured.

[0115] The medical training system used for performing this task may be a robotic surgical training system as described above. The system includes a physical interface for the user to control, such as at least one tangible instrument 40, 41 , a computation device comprising the control unit 1 , the display 2, and a software implementation of a training scenario run by the control unit 1.

[0116] The physical interface includes the sensors 44, 45 that track the position of the at least one tangible instrument 40, 41 , wherein the sensors and / or the control unit 1 maps the positions into the virtual world in the training scenario.

[0117] Within the training session, the movements of the tangible instruments 40, 41 are tracked. The training scenario which is the basis of the training session comprises an initial starting point and the final goal task with optional intermediate steps. The final goal task in the example shown in figure 6 is the correct placement of the virtual object 30 into the virtual recess 31.

[0118] The tangible instrument 40, 41 is coupled to the position of the virtual instrument 43, where a movement of, for example 1 cm, would occur with the instrument 40,41 and be replayed with the virtual instrument 43 by moving it 1cm in the virtual world. The virtual environment may be rendered at a different scale than the real world. A first sphere 6 arranged at a distance around the virtual recess 31 defines the space within which the movements of the virtual instrument 43 and therefore also of the tangible instrument 40, 41 is monitored. A second sphere 60 arranged around a part of the virtual instrument 43 defines the position of the virtual instrument 43. The second sphere 60 defines an origin point. The origin point is the point which corresponds to the real instrument which would treat a patient in a surgery or in another medical treatment. In case the tangible instrument 40 is identical in shape and size with the real medical instrument, the origin point corresponds to this tangible instrument 40 as well. In case the tangible instrument 40 is simulating a master control input device of a robot surgery system, the point of origin is the point of origin used with this master control unit device as well and it corresponds to the relevant point, usually the tip, of the real robotic instrument. The point of origin can be a region, especially a volume, as shown in these embodiments. It can also be a defined area or a point.

[0119] The shape of the second sphere 60 is shown as a round ball. It can have any three- dimensional shape surrounding a volume. It can even be a two-dimensional area, such as a square, a rectangle or a circle, or a single point can be used as indicator as well. If more than one virtual instrument is used, preferably all instruments comprise a separate second sphere 60.

[0120] The first sphere 6 is also shown as a round ball. However, it can also have any three- dimensional shape surrounding a volume. It can even be a two-dimensional area, such as a square, a rectangle or a circle, or a single point can be used as well. If more than one task has to be fulfilled in sequence, preferably all tasks are surrounded by a separate first sphere 6. In some embodiments, the separate first spheres 6 may be surrounded by a major first sphere as well.

[0121] When the second sphere 60 enters the first sphere 6, i.e. the marked spot on the virtual instrument 43 enters the region of the goal task, the movements of the tangible instrument 40, 41 is tracked and the movement of the virtual instrument 43 is monitored and recorded. This is an initial point, in figures 2 and 3 being the free end 70 of the line marked as "seeking path". In case more than one virtual instrument 43 and more than one second sphere 60 are present, this initial point is reached when a pre-defined and / or a first of these second spheres 60 enters the first sphere 6.

[0122] When the task is fulfilled, a point of interest according to figures 2 and 3 is reached, marked in figure 3 with reference number 71. The way between the initial point 70 and the point of interest 71 is the seeking path 7.

[0123] In some embodiment, the control unit 1 does also detect when the second sphere 60 leaves the interior of the first sphere 6. The way between the point of interest, for example the position of the tangible instrument 40, 41 and of the virtual instrument 43 when the task is fulfilled, and the position of the second sphere 60 leaving the first sphere 6 is defining the retracting path.

[0124] The control unit 1 can therefore differentiate between the seeking path and the retracting path.

[0125] Preferably, the tracking occurs in three degrees of freedom. The position of the virtual instrument 43 is recorded until the final task is completed, in this example, until the virtual object 30 is placed within the recess 31. This defines the point of interest 71 , which is also called final point. This is shown in figure 3. The path between the initial point 70 and the final point 71 or point of interest is the seeking path 7 which has been taken in order to fulfill the given task. The initial point 70 forms the first end of the seeking path 7 to be assessed and the final point 71 forms the second end of this seeking path 7.

[0126] The first and the second spheres 6, 60 are preferably hard-coded, wherein the spheres preferably vary depending on the task to be fulfilled and the instruments to be used.

[0127] In other embodiments, the at least one of the first and second sphere or both spheres 6, 60 are dynamically set, based on parameters of a training scenario chosen or based on former training scenarios already been performed. Some embodiments allow to define at least one of the two spheres, preferably both spheres 6, 60, to be defined based on accumulated user data, preferably after having completed an appropriate statistical analysis. Figure 7 shows both spheres 6, 60.

[0128] The tracked seeking path can be used for statistical analysis for differentiation, e.g., between different population groups such as expertise levels or educational backgrounds. The length of the chosen seeking path can be determined and compared with a predefined target path length. Figure 7 shows the seeking path 7 which has been taken for placing the virtual object 30 into the virtual recess 61 of figure 6. In figure 6 the process is still in progress and the task is not yet fulfilled. The seeking path 7 is also shown in the screen of the embodiment according to figure 5a.

[0129] Some embodiments and methods do not only monitor the seeking path but do also include orientation of the tangible instruments 40, 41 and / or virtual instruments 43, adding up to an additional three degrees of freedom. Some embodiments monitor more than one tangible instrument 40, 41 , and / or virtual instruments 43 in the same training session, preferably at the same time, wherein the seeking path of each of these instruments, 40, 41 , 43 is monitored and analyzed. For example, by monitoring more than one instrument, a dominant hand seeking path can be scored differently than the seeking path performed by a nondominant hand.

[0130] The seeking path is the length of the path between the initial point and the final point. Both points depend on the training system's settings. The settings may depend on at least one of the following: the task to be fulfilled, the tangible or virtual instruments used, the user's previous performance and the user's inputs.

[0131] By choosing the shape, size and position of the first sphere 6, the initial point can be varied. The final point can be varied such that it does not have to mark the point where the task is finally fulfilled but it can also mark another point on the way to fulfill the task. This another final point is still located within the first sphere 6 chosen by the training system based on assessment results, predefined settings or by user setting.

[0132] Therefore, the point of interest or the final point does not have to be the place of the fulfilment of a final task but may be an important place on the way there. The seeking path to be assessed separately is therefore only the way until this important point, even if the instrument is moved further to fulfill the final task. The further movement of the instrument from this important point to the final task would be considered as retracting way, i.e. retracting from this important point. Therefore, the training system may treat an entire way to a final task as seeking path which has to be assessed separately or only a part of this way.

[0133] The training system may define the initial point and the final point based on the numerical data achieved from the movement of the virtual instrument in the virtual space and / or from sensors arranged in the real world detecting the movements of the tangible instruments in the real space.

[0134] Figures 10 to 15 show screenshots of a display of an already physically realized embodiment of the medical training system. The figures do only show a part of the display. It is the training system and the task as described above with regard to figures 6 and 7.

[0135] In figure 8, the virtual instrument 43 holding the virtual object 30 and especially the second sphere 60 have not yet reached the first sphere 6 surrounding the place of goal, i.e. the virtual recess 31. The tracking of the path has not been activated yet. In figure 9, the second sphere 60 has already penetrated the first sphere 6. In this example, the center position of the second sphere 60 entering the first sphere 6 mark the initial point when the tracking started.

[0136] In figures 10 and 11 the virtual object 30 is still not placed within the virtual recess 31 . The control unit 1 still tracks every activation and every movement of the tangible instruments 40, 41 thereby tracking the movement of the virtual instrument 43 shown on the display 2. When the process is not improving, as shown in figure 10, in which the virtual object 30 has an incorrect angle with regard to the virtual recess 31 , the area surrounding the virtual recess 31 and / or a background may change its color and / or a visual or an auditory alarm may be given to the user.

[0137] In figure 12, the task is almost completed. In figure 13, the task is fully completed and the virtual instrument 43 can be retracted again. Completion of the task may be indicated to the user by changing the color of the area surrounding the virtual recess 11 and / or of the background and / or by giving another visual and / or auditory signal to the user.

[0138] Depending on the embodiment, the monitored seeking path 7 is already shown on the display during the performance of the task or it is shown afterwards, when the task is completed and / or an analysis was performed. In figures 12 and 13, the seeking path 7 is already shown.

[0139] In figure 14, the second sphere 60 of the virtual instrument 43 leaves the interior of the first sphere 6. As can be seen, the path of the virtual instrument 43 shown on the display 2 is the seeking path 7 only. There is no line between the free first end 70 of the path 7 shown and the retracted virtual instrument 43 in the present position. The retracting path may be monitored and analyzed as well, but separately from the seeking path.

[0140] As can be seen in figure 15, the seeking path 7 can be analyzed by the user visually be rotating and / or enlarging the image shown on the display 2. The first end of the path 7, corresponding to the initial point, is marked with reference number 70, the second end of the path 7, corresponding to the final point, is marked with reference number 71.

[0141] Figures 16 and 17 show other screenshots of the display of the already physically realized embodiment of the medical training system described in figures 10 to 15. Instead of an abstract task, the training scenario now shows a real medical scenario, which is a salpingotomy for an ectopic pregnancy in the fallopian tubes.

[0142] In figure 16, the virtual instrument 43 just enters with its second sphere 60, marking the origin point of the real medical instrument used in this medical treatment, the first sphere 6. A second virtual instrument 430 is present, wherein its position and movement is not part of the seeking path assessment performed at this stage. In figure 17, the final point was reached and the assess seeking path 7 is shown. As mentioned above, this seeking path does not have to be the entire way the virtual instrument 43 takes until it has completed all its tasks.

[0143] The assessment of the seeking path only, instead of assessing the entire path, taken with a tangible and / or virtual instrument, improves the assessment of the user's skill, improves fulfillments of surgery tasks and reduces the computing power of the training system.

Claims

CLAIMS1. A medical training system comprising a simulator assembly configured to perform at least one medical procedure by using an instrument in a simulated medical procedure setup, the simulator assembly being manually operated by a user, and a control unit providing the simulated medical procedure setup, monitoring at least a seeking path of the instrument moved by the user during the simulated medical procedure, wherein the control unit is capable of differentiating between the seeking path and a retracting path of the instrument, and wherein the control unit is capable of using only the seeking path of the instrument in order to assess a proficiency of the user.

2. The medical training system according to claim 1 wherein the simulator assembly is a tangible simulator assembly.

3. The medical training system according to either one of claim 1 and 2 wherein the instrument is a tangible or a virtual instrument.

4. The medical training system according to any one of claim 1 to 3, wherein the training system defines an initial point and a final point and therein the seeking path comprises a first end and a second end, wherein the initial point defines the first end and the final point defines the second end.

5. The medical training system according to claim 4, wherein the initial point is a starting point of a movement of the instrument and / or the final point is a point where a given task is fulfilled by the instrument.

6. The medical training system according to either one of claims 4 and 5, wherein the training system defines a volume within the final point is positioned and a point of origin at the instrument, wherein the initial point is defined as point where the point of origin enters the volume.

7. The medical training system according to claim 6, wherein the volume is asphere and / or wherein the point of origin is a sphere.

8. The medical training system according to claim 7, wherein the volume and / or the point of origin is hard-coded.

9. The medical training system according to claim 7, wherein the volume and / or the point of origin is variable by the training system based on previous assessments and / or by user setting.

10. The medical training system according to any one of claims 1 to 9, wherein the instrument is a tangible instrument and wherein the tangible instrument comprises at least one input sensor providing input data during a movement of the tangible instrument by the user, and wherein the control unit is capable of acquiring data from the at least one input sensor.

11. The medical training system according to any one of claims 1 to 10, wherein the control unit comprises at least one seeking-retracting phase determination algorithm to differentiate between the seeking path and the retracting path, wherein the control unit is capable of generating a modified data subset comprising data from the seeking path only and wherein the control unit is capable of comparing the modified data subset to at least one stored optimal seeking path or to a stored set of optimal seeking paths, and wherein the control unit comprises at least one proficiency assessment algorithm to be applied between the modified data subset and the at least one stored optimal seeking paths or the stored set of optimal seeking paths.

12. The medical training system according to any one of claims 1 to 11 , wherein the control unit is capable of determining the length of the seeking path.

13. The medical training system according to any one of claims 1 to 12, wherein the control unit is capable of determining the time to cover the seeking path.

14. The medical training system according to any one of claims 1 to 13, wherein the control unit is capable of monitoring the alignment of the instrument on the seeking path.

15. The medical training system according to any one of claims 1 to 14, whereinthe medical training system is a simulator for minimal invasive surgery, preferably for robotic surgery, and / or for laparoscopy.

16. The medical training system according to any one of claims 1 to 15, wherein the control unit comprises a memory with data concerning at least one optimal seeking path and wherein the control unit is capable of comparing data of the monitored seeking path with the data of at least one optimal seeking path in order to assess the proficiency of the user.

17. The medical training system according to claim 16, wherein the data of at least one optimal seeking path is at least one optimal path length and / or at least one optimal time to cover the optimal seeking path and / or for an optimal orientation of the instrument on the optimal seeking path.

18. The medical training system according to any one of claims 1 to 17, wherein the monitored seeking path is assessed by automatic video analyses.

19. The medical training system according to any one of claims 1 to 18, wherein it is simulating a teleoperated medical treating system and wherein it comprises a surgeon control with master input devices which manipulate simulated virtual instruments of the simulated teleoperated medical treating system.

20. The medical training system according to any one of claims 1 to 19, wherein it is a mixed reality scenario medical training system enabling a user to interact with real objects in a physical environment and in a virtual environment.

21. The medical training system according to any one of claims 1 to 20, wherein the control unit comprises a web-based application and wherein the training system includes web-based data and software used for performing training scenarios and for assessing the seeking path.

22. A method for assessing a proficiency of a user of a medical training system according to any one of claims 1 to 21 , wherein the control unit monitors at least a seeking path of the instrument moved by the user during the simulated medical procedure and wherein the control unit differentiates between the seeking path and a retracting path, and compares the monitored seeking path with at least one stored optimal seeking path or witha stored set of optimal seeking paths in order to assess the proficiency of the user.

23. The method according to claim 22, wherein the control unit determines the length of the seeking path and compares the determined length with at least one stored length of the optimal seeking path.

24. The method according to either one claim 22 or 23, wherein the control unit determines the time to cover the seeking path and compares the determined time with at least one stored time of the optimal seeking path.

25. A non-transitory machine-readable medium comprising a multiple of machine-reading instructions which when executed by one or more processors associated with the medical training system according to any one of claims 1 to 21 performs the method of any one of claims 22 to 24.

Images