3D Tracking and Navigation Simulator for Neuroendoscopy

Abstract

A three-dimensional tracking and navigation simulator system for image-guided neuroendoscopy, the system comprising: a standards-based physical model of an anatomical region associated with neuroendoscopy, the standards-based physical model being mapped with CT / MRI slices of the anatomy; a virtual model of the anatomical region associated with neuroendoscopy (3 and / or 9); at least one surgical instrument operable to perform neuroendoscopy, the at least one surgical instrument being associated with a 3D tracking component (4); a camera unit (1) for tracking a position of the at least one surgical instrument with the 3D tracking component (4) relative to a reference unit (5) of the physical model; and one or more displays (11, 12) for displaying the position of the at least one surgical instrument in the CT / MRI slices and the endoscope output. 6, 12), and a computer workstation (11) operatively coupled to the physical model and the camera unit (1), the computer workstation comprising: a localization and tracking module configured to track tasks performed by a user with at least one surgical instrument and to estimate a position and rotation of the at least one surgical instrument in six degrees of freedom (DOF), a virtual space mapping module configured to map positions to corresponding anatomical locations in the virtual model to provide navigation of the surgical instrument in the virtual model, and an evaluation module configured to evaluate the 3D tracking of the performed tasks based on localization metrics and 3D tracking metrics and provide feedback to the user based on performance using machine learning and AI based algorithms.

Description

[Technical field]

[0001] The present invention relates to a surgical training simulator, and more particularly to a three-dimensional, i.e., 3D tracking and navigation simulator for neuroendoscopic surgical skills training with image guidance and skills assessment. [Background technology]

[0002] Minimally invasive surgical interventions are highly dependent on techniques. Neurosurgery is a more important and less controversial surgical specialty. Neuronavigation systems are support systems for image-guided neurosurgery for the localization and delineation of tumors and aid in their precise surgical resection. The basic idea is to position the tip of the pointer within any given digital image space (CT / MRI) by mapping the device space to the image space. There is a transformation matrix that maps the anatomical space / device space with the digital image space (CT / MRI) determined by the registration and calibration of the navigation device. Existing neuronavigation systems are based on stereotactic frames, optical tracking methods or electromagnetic tracking methods. Commercially available neuronavigation systems are very expensive and are not available for training under laboratory conditions. Neuronavigation is frequently used in minimally invasive neuroendoscopic procedures such as endoscopic endonasal skull base surgery.

[0003] Neuroendscopy is a type of neurosurgery where surgeons use endoscopes to navigate inside the brain. Long-axis instruments and different perspective views of the procedure make it difficult for neurosurgeons to get used to it. The learning curve is steep and difficult under the hands-on model of learning. Also, due to work time constraints and medical demands, it is not advisable to learn the surgical procedures and the corresponding skills on patients. Simulation-based learning is the solution for this, and simulators range from naturalistic simulations to highly instrumented virtual reality simulations.

[0004] Physical simulators are more suitable for developing psychomotor skills. However, physical simulators do not provide training in image-guided navigation of anatomical structures. A simulator that equips neurosurgeons with real surgical scenarios should include orientation to successfully navigate vital anatomical structures. The simulator provides a safe environment for learning and relearning. The objective evaluation involved in the simulator provides the trainee with feedback on his / her performance and specific ways to improve his / her skills.

[0005] Existing prior art does not provide a simulation for neuroendoscopy that provides training for image guidance. A skills training system that can be used as a simulator and for anatomical orientation is lacking in the current scenario of a simulator.

[0006] In this regard, reference is made to US Pat. No. 1,041,7936, which discloses a method and system for clinical procedure training via a mixed reality simulator. There are physical and virtual models of the anatomical region. It also includes a tracking component that tracks any one instrument, an anatomical feedback component that reproduces the changes in the physical model based on the interaction between the instrument and the virtual model. It can detect electrical signals, pressure and flow. The feedback to the physical model depends on the tracked position of the clinical device. The physical and virtual models are co-registered for better performance and are created from the same anatomical source such as CT, MRI. It provides a procedure simulation of nerve blocks. The physical simulator is fitted with robotic actuators. The material selection of the physical model is based on realistic appearance, texture and haptic feedback. The instruments used are purely mechanical or electronic, and as the user manipulates the instruments, the spatial tracking component utilizes a 3D tracking system and maps their position to the virtual model. The instruments have one or more sensors to monitor the 5DOF (degrees of freedom) and the data is transmitted in a wired or wireless manner. The position / tracking can be determined by an electromagnetic tracking system or an ultrasound probe.

[0007] Reference is made to US Patent Publication No. 20180042514, which discloses a system and device taught to determine the identity of the equipment involved in the procedure to aid in navigation. This includes information provided by the user to the RFID reader at a selected time. This information can be used to aid in navigation of the equipment relative to the patient. The navigation system can be used to track equipment such as catheters, needles, guidewires, and instruments, allowing the surgeon to see the relative position of the equipment to a coordinate system on the display. The coordinate system can be created for images such as in image-guided procedures and can be registered only to a specific patient. Image data can be captured and transferred to a navigation computer. The surgical navigation system includes navigating the procedure relative to the anatomical structure, tracking the equipment associated with the equipment, tracking system to track the location of the equipment, identification of the equipment based on selected information, and RFID tag-based identification and tracking of the equipment. It can also be used for multiple instruments.

[0008] Reference is made to US Patent Publication No. 20180125604, which teaches a navigation method and system for an anatomical structure that uses an instrument, identifies the position of the instrument, and navigates the instrument relative to the anatomical structure. This includes imaging the anatomical structure and displaying a working portion of the instrument relative to the image data by an electromagnetic tracker. The projection and reception portions use electromagnetic fields and can be used for multiple instruments. The tracking system includes an electromagnetic localizer and is operable to obtain data from the identification member. There is a communication system that interconnects the tracking system and the navigation processor. The navigation processor determines the identification of the surgical instrument based on the data transferred from the identification member to the navigation processor.

[0009] Please refer to JP2020106844, which discloses a medical training equipment and method for training in minimally invasive surgery. It includes a display device for visualizing a medical device, a medical device structure positioned relative to a physical simulation area, and a virtual surgical site, which is a simulated surgical field. The medical device is manipulated with a manipulator arm, and the surgical device is positioned in the virtual environment relative to a reference arm. This provides feedback to the user by comparing the position of the manipulator arm with the reference arm. The simulation area is manipulated based on the position of the medical device. In the simulated surgery, the virtual environment is updated based on the physical interaction, and the position is mapped by the manipulator arm.

[0010] Yet another reference refers to Russian Patent No. 2726476, which teaches a method for facilitating the training of otolaryngologists for basic skills of endoscopic endonasal surgery with a Sinus Model Otorhino-Neuro Trainer simulator (SIMONT). This is a simulation model for providing skills training for sinus surgery. It helps to practice skills with biological materials made of silicone, which lends itself to repeated practice in the laboratory. It provides hands-on collaborative training with instruments and endoscopes.

[0011] Reference is further made to WO2012106706, which discloses a system and method for facilitating training of clinical procedures by mixed reality simulation. Such a system may include a physical model and a virtual model of an anatomical region associated with the procedure, the virtual model associating tissue types with positions in the physical model. The system may include a tracking component that tracks the position of at least one clinical device relative to the model, and an anatomical feedback component that may generate perceptible changes in the physical model based on an interaction between the device and the virtual model. The clinical device interface may detect an output of the clinical device, such as an electrical signal, pressure or flow, and feedback to the physical model depends on the tracked position of the clinical device, and on the output from the same or a different clinical device. Another component may generate a feedback effect to the clinical device. An embodiment may simulate an anesthesiology procedure, such as a regional nerve blockade.

[0012] WO2012106706 is a mixed reality simulation model including a physical and virtual model for anesthesiology simulating a regional nerve block in the arm. The present invention, on the other hand, is a navigation-based simulation and training system that maps a physical model related to neuroendoscopy to a virtual model of the area as well as to a CT / MRI of the corresponding anatomical area. The visualization and training motivations of both simulators are clearly different. The training simulator of the present invention involves the training of the neuroendoscopic surgeon to navigate well in the brain and skull area as he normally does inside the operating room. The visualization provided is also identical to the navigation system. In the present invention, the simulation is aimed at procedure-based skill training to provide a realistic physical model for training with the help of image guidance.

[0013] The tracking component of WO2012106706 concerns a commercially available electromagnetic tracking system for locating the instrument, which is susceptible to interference with nearby motorized systems and instruments. In the present invention, the tracking method is based on a single camera and AruCo marker-based tracking system that maps the position of the instrument tip into the 3D CT space of the respective physical model. The camera-based tracking system provides the instrument pose (rotation and translation) relative to a reference AruCo tracker. The tracking system is robust against any motorized obstructions.

[0014] The anatomical feedback component of WO2012106706 generates changes in the physical model based on the interaction of the equipment in the virtual model. In the present invention, on the other hand, the feedback mechanism works differently. The physical feedback is direct, since the trainer uses the clinical equipment for training, and the visual and haptic feedback is received directly with the help of the endoscope and the physical model, respectively. Tracking and visualization with 3D CT planes are additional information, signaling the surgeon to locate himself, learn where he is anatomically, and operate with precise movements. The present invention provides a user interface similar to that of the operating room.

[0015] WO2012106706 includes tracking information to be used to provide simple feedback to trainees such as time, speed, etc. In the present invention, 3D position of instruments and endoscopic camera inputs are captured and stored for feedback on performance skill. These sensor data are synchronized so that the inputs can be analyzed using deep learning based skill assessment techniques to provide specific skill feedback to the surgeon in terms of skill assessment metrics such as a Likert scale. The present invention includes surgical image guidance / navigation and artificial intelligence / deep learning based skill assessment. The present invention develops an algorithm to assess surgeons based on a global assessment measure of endoscopic video inputs and 3D tracking information for tasks performed in the simulator.

[0016] Reference is further made to US Patent Publication No. 20100167253, which discloses a surgical simulator including at least one tracking system configured to track movements within the body form, including movements of an instrument partially positioned within the body form, and a computer that receives data from the tracking system and outputs the data to a display. The display displays a virtual background including a virtual image of at least a portion of the instrument, a virtual movement of the instrument, and a virtual image of at least one organ. The simulator also includes a physical model corresponding to the at least one organ and positioned within the body form, the physical model providing haptic feedback when contacted by the instrument.

[0017] US20100167253 includes a non-anatomical physical model for providing haptic feedback. Our invention includes an anatomically relevant physical model for training neurosurgeons. One tracking system mentioned in US20100167253 includes an instrument to be partially placed within a body form, with the tracker present within the body form. Whereas in the present invention, the tracking system is external to the trainer and can also be extended to surgical scenarios without relying on a casing or body form. The instrument is free to move and does not need to be fixed with the body form. US20100167253 displays a virtual background that includes images of parts of the instrument, its movement and organs. In the present invention, the visualization includes CT / MRI slices, with the position / tip of the instrument mapped into the CT space. The physical model included in US20100167253 is non-anatomical and there is no relevant anatomical mapping to the virtual model. In the present invention, the same CT / MRI used for visualization is segmented and recreated as a physical model to replicate the exact anatomy for training. The present invention further includes surgical image guidance / navigation and artificial intelligence / deep learning based skill assessment.

[0018] See also US Patent Publication No. 20100159434, which provides a composite simulator system that combines the benefits of both physical objects / simulations and virtual representations. In-context integration of virtual representations with physical simulations or objects can facilitate education and training. Two modes of composite simulation are provided. In the first mode, the virtual representation is combined with the physical simulation or object by using a tracking display capable of displaying appropriate dynamic virtual representations as the user moves around the physical simulation or object. In the second mode, the virtual representation is combined with the physical simulation or object by projecting the virtual representation directly onto the physical object or simulation. In further embodiments, the user's actions and interactions can be tracked and incorporated within the composite simulator system to provide mixed reality post-action review. The subject composite simulator can be used in many applications, including, but not limited to:

[0019] US20100159434 is a hybrid simulator that combines physical and virtual representations. The present invention is a navigation-based simulation and training system that maps a physical model to the CT / MRI of the corresponding anatomical region, rather than the virtual representation. US20100159434 includes tracking using an optical sensor (infrared camera), and there are two modes of visualization. On the one hand, the virtual representation moves dynamically as the user moves and is displayed. In the second mode, the virtual representation is projected onto the physical simulation. In the present invention, the visualization mode is the CT space of the physical model, and the tip of the instrument inside the physical model is mapped to the relevant anatomical structure. The tracking in US20100159434 is mainly focused on post-action review. In the present simulator, tracking focuses on image guidance to the trainee and capturing the movement for skill evaluation as a 3D tracking instrument. US20100159434 does not include any specific feedback to the trainee for skill improvement and is a generalized module. The present invention is a dedicated simulator for skill assessment in neurosurgery, providing feedback to the trainee with the help of our custom designed deep learning module, including surgical image guidance / navigation and artificial intelligence / deep learning based skill assessment.

[0020] Yet another reference is WO2014116278, an embodiment of which discloses a surgical training system (100) comprising an anatomical training model (102), three or more telemetry sensors (104) attached to the training model, a training instrument (106), a display (112), and a controller (108). The anatomical training model physically simulates human anatomical features. The training instrument comprises at least one transmitter (122) configured to emit a signal. The controller includes one or more processors configured to determine a position of the training instrument relative to the training model using the sensor in the signal and generate on the display a virtual image of the training instrument and the anatomical features simulated by the training model based on the position of the training instrument. In an embodiment of a surgical training method, the training instrument (106) is positioned (160) near an anatomical training model (102) that physically simulates human anatomical features. The position of the training instrument relative to the training model is determined (162) using the controller (108). An orientation of the training instrument relative to the training model is determined using the controller (164). A virtual instrument (130) corresponding to the training instrument is positioned (166) within a virtual model (128) corresponding to the training model based on the position and orientation of the training instrument using the controller. The virtual instrument and virtual model are displayed (168) on the display (112) using the controller.

[0021] WO2014116278 includes a surgical training system that simulates human anatomy for physicians to practice electrode implantation without the need for a cadaver. In the present invention, the simulation training focuses on surgical training with accurate anatomical features of the patient's CT / MRI. The simulation is aimed at procedure-based skill training to provide a realistic physical model to train with the aid of image guidance. WO2014116278 includes the use of a telemetry sensor attached to the training model. In the present invention, a lightweight tracker is attached to any one of the surgical instruments, eliminating the need for a camera to be present inside the training model. The camera can be placed outside to track the instrument tip, and does not need to clearly see the instrument tip. In WO2014116278, the orientation of the training instrument relative to the training model is determined using a controller for display purposes. In the present invention, the training instrument is 3D tracked with the aid of AruCo markers and the instrument tip is projected into the CT / MRI space of the anatomy. The input of the endoscopic camera and the camera tracking the instrument are also captured for skill assessment. In WO2014116278, no specific feedback is provided regarding the trainee's performance. In the present invention, feedback is provided to the trainee according to his / her performance, including surgical image guidance / navigation and artificial intelligence / deep learning based skill assessment.

[0022] In addition to the above cited prior art, there are commercially available neuronavigation systems that work with the help of stereotactic frames, optical tracking methods or electromagnetic tracking methods to identify the location of a particular probe being inserted into the brain. These systems are very expensive and are not available for laboratory training. Commercially available neuroendoscopic simulators are synthetic simulators for hands-on procedural training. They do not provide objective metrics for skill evaluation. An integrated system that provides image-guided surgical training, neuroendoscopic procedural training with objective evaluation is lacking in scenarios for neurosurgical skill training.

[0023] Therefore, considering the existing prior art, there is a dire need for a neuroendoscopic surgery training module with 3D tracking and low-cost navigation system to train neurosurgeons for image-guided surgery with endoscopic simulation training.

[0024] Summary of the Invention The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key / critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description of the invention that is presented later.

[0025] The objective of the present disclosure is to provide a neuroendoscopic surgery training module with 3D tracking and low-cost navigation system for training neurosurgeons for image-guided surgery, accompanied by endoscopic simulation training and artificial intelligence-based skills assessment.

[0026] The objective of the present disclosure is to provide a neuroendoscopic surgery training module with a 3D tracking and low-cost navigation system that integrates computer vision technology to track ArUco-based markers using a machine vision camera unit and locate the tip of a surgical instrument within the CT / MRI space of a human anatomical model.

[0027] The objective of the present disclosure is to provide machine learning and deep learning based methods for image guidance, methods for anatomical and endoscopic orientation of a neurosurgeon, and methods for evaluating endoscopic skills based on performance, including a neuroendoscopic skills training simulator that provides 3D tracking of the trainee along with objective evaluation metrics of the tasks performed.

[0028] The objective of this disclosure is to provide a neuroendoscopic multipurpose simulator for image-guided neurosurgery and neuroendoscopic simulation with 3D tracking-based skill assessment and to develop an algorithm to evaluate the surgeon based on a global assessment measure of the endoscopic video input and 3D tracking information for the tasks performed in the simulator.

[0029] A first aspect of the present invention is a three-dimensional tracking and navigation simulator system for image-guided neuroendoscopy, comprising: a standards-based physical model of an anatomical region associated with neuroendoscopy, the standards-based physical model being mapped with CT / MRI slices of the anatomy; a virtual model of the anatomical region associated with neuroendoscopy; at least one surgical instrument operable to perform neuroendoscopy, the at least one surgical instrument being associated with a 3D tracking component; a camera unit for tracking a position of the at least one surgical instrument with the 3D tracking component relative to a reference unit of the physical model; one or more displays for displaying the position of the at least one surgical instrument in the CT / MRI slices and the endoscope output; and a computer workstation operably coupled to the virtual model and the camera unit, the computer workstation comprising: a localization and tracking module configured to track tasks performed by a user with at least one surgical instrument and to estimate a position and rotation of the at least one surgical instrument in six degrees of freedom (DOF); a virtual space mapping module configured to map positions to corresponding anatomical positions in the virtual model to provide navigation of the surgical instrument within the virtual model; and an evaluation module configured to evaluate the 3D tracking of the performed tasks based on the localization metrics and the 3D tracking metrics and provide feedback to the user based on the performance using machine learning and AI based algorithms.

[0030] Other aspects, advantages and salient features of the present invention will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses illustrative embodiments of the invention.

[0031] The above and other aspects, features, and advantages of the embodiments of the present disclosure will become more apparent in the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a schematic diagram illustrating a 3D tracking and navigation simulator for neuroendscopy, according to one embodiment of the present invention. [Diagram 2] FIG. 1 is a schematic diagram of a 3D physically simulated anatomical model for training purposes, according to one embodiment of the present invention. [Diagram 3] FIG. 13 is a schematic diagram of a 3D tracker design for an instrument with an instrument tracker having ArUco markers attached to a biopsy forceps, according to an embodiment of the present invention. [Figure 4] FIG. 2 is a detailed block diagram of a 3D tracking and navigation simulator for neuroendscopy, according to one embodiment of the present invention. [Diagram 5] FIG. 1 illustrates anatomical points defined within the skull for experiments, according to one embodiment of the present invention. [Figure 6] FIG. 1 illustrates a setup of a neurosurgeon practicing on a simulator system, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and may not be drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help improve understanding of various exemplary embodiments of the present disclosure. Note that like reference numbers are used throughout the figures to depict the same or similar elements, features, and structures.

[0034] The following description, with reference to the accompanying drawings, is provided to aid in a comprehensive understanding of exemplary embodiments of the present invention. To aid in the understanding, various specific details are included, but these should be considered as merely examples.

[0035] Features described and / or illustrated with respect to one implementation may be used in the same or similar manner in one or more other implementations and / or in combination with or instead of features of the other implementations.

[0036] Neuroendoscopy is a surgical specialty that requires bimanual, visual, cognitive and psychomotor skills. Endoscopic surgery is performed with the help of a 2D display, and the surgeon creates a mental picture of the 3D anatomical structures using depth cues. In neuroendoscopic surgery, tumor resection is assisted with the help of image guidance to precisely locate the tumor and remove it. However, no specific training simulator exists based on navigation / image guidance, or neuroendoscopic simulation along anatomical CT orientation. Although several commercially available neuroendoscopic synthetic simulators exist, no image guidance-based training exists, and they do not provide 3D tracking of the surgeon's actions.

[0037] Embodiments of the present invention disclose a multi-purpose simulator for neuroendoscopic surgical skills simulation and skills assessment using 3D tracking and image guidance.

[0038] Various embodiments of the present invention describe a 3D tracking and navigation simulator (3DTN simulator) for neuroendoscopy. These embodiments include one or more of the components to provide a 3DTN simulator, which includes a physical anatomical model, a virtual model of the anatomical structure, registration software for mapping the physical model with CT / MRI slices or virtual models of the anatomical structure, 3D tracking of any instruments used in the surgery fitted to our tracker, calibration of coordinate systems for physical and virtual spaces, navigation of instruments in the virtual anatomy, localization of the anatomical structure, simulation environment for neuroendoscopic surgery, 3D tracking of the tasks performed, evaluation of the tasks based on localization and 3D tracking metrics, and feedback to the trainee based on performance using machine learning and AI-based algorithms.

[0039] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of the present disclosure. The term "module" as used in this application refers to a computer-related unit, either hardware or a combination of hardware and software.

[0040] The term "feedback" as used in this application includes physical feedback provided directly to trainees using clinical equipment for training, and / or visual and tactile feedback received with the aid of endoscopes and physical models. It also includes feedback provided in the form of a summary of their actions, including objective metrics on their performance and specific feedback on their skill level.

[0041] This involves training neuroendoscopic surgeons to navigate successfully through the brain and skull regions, similar to in the operating room. The visualization provided is identical to the navigation / image guidance system. It is aimed at procedure-based skill training on physical models, with realistic visual and haptic feedback, with the help of image guidance and feedback modules for the individual development of specific skills.

[0042] According to one embodiment, a 3D tracking and navigation simulator for neuroendoscopy includes a camera unit, a surgical instrument based tracker, a fiducial unit, a standard based physical simulation model, its CT / MRI scan, and software for the process of virtual to physical model registration, 3D tracking of any neurosurgical instrument fitted to our tracker, tri-planar CT / MRI visualization, and evaluation software for surgical skill analysis. Software is developed for registration of CT / MRI images to the physical simulator. The model can be replaced by any simulation model of interest, such as an ETV simulator, a spine surgery simulator, or a general surgery simulator.

[0043] In one embodiment, the present invention relates to a 3DTN simulator having a standard-based anatomically relevant physical simulation model of a neuroendoscopic surgery with corresponding CT / MRI scans. This anatomical model with standard points is CT / MRI scanned with 0.60 mm XY resolution and 1 mm Z resolution positioned according to the simulation requirements.

[0044] Another embodiment is a platform for positioning an anatomical model for different training scenarios. The platform includes a housing for the camera unit, the reference unit, and also for the anatomical model. It also includes a fixture for securing the model and accessories to cover the model for training.

[0045] Yet another embodiment includes a localization and tracking module with a three-plane instrument tracker consisting of planes at different angles to each other, which can be adapted to any neurosurgical instrument. The tracker is affixed with ArUco markers. ArUco markers are synthetic square markers consisting of a wide black border and an inner binary matrix that determines its identifier (id). The three-plane structure is designed to estimate at least two planes from any viewpoint. To determine the different planes, ArUco markers with different identifiers were affixed. The three-plane tracker was modeled and its corner 3D points were known with respect to the origin.

[0046] Yet another embodiment includes a reference tracker that uses two planes positioned at different angles to each other.

[0047] Another embodiment includes a camera unit for tracking the ArUco markers. The camera is positioned on the platform such that in a given frame both the instrument tracker and the reference unit are visible.

[0048] Yet another embodiment is a physical localization module that collects data from the vision camera and locates the instrument tip relative to a reference unit. This module provides the spatial position and orientation of the instrument. The tracking module then tracks the instrument tip frame by frame and is robust to any motorized obstructions. The localization and tracking module can monitor the instrument or instrument at any given time. This includes 6 degrees of freedom (DOF). There are no wired sensors attached to the instrument, which does not interfere with the surgeon's actions.

[0049] Yet another embodiment is a virtual space mapping module that aligns the instrument tip of the physical space instrument with respect to the virtual space and locates the tip position on the CT / MRI slices. The instrument tip position estimated with respect to the reference coordinate system is projected onto the CT / MRI of the simulation model for visualization and mapped to the respective anatomical location. The simulation model is assumed to be fitted with standard markers and the corresponding CT / MRI coordinate system associated with it. The CT / MRI data was resampled to 0.60 mm (XY resolution) in all three directions and a new CT / MRI image set was created using the VTK library. The CT / MRI points were visualized in all three planes, i.e. horizontal, coronal, and sagittal. Another embodiment is localization of anatomical structures in the virtual space and 3D model.

[0050] One embodiment is a simulator system comprising one or more displays (6, 12) for displaying the position of at least one surgical instrument in CT / MRI slices and the endoscope output, as shown in Fig. 1. Furthermore, the system comprises a computer workstation (11) operatively coupled to the physical model and the camera unit (1), the workstation comprising a localization and tracking module configured to track a task performed by a user with at least one surgical instrument and the endoscope (7) and estimate the position and rotation of the at least one surgical instrument in six degrees of freedom (DOF), a virtual space mapping module configured to map positions to corresponding anatomical locations in the virtual model to provide navigation of the surgical instrument in the virtual model, and an evaluation module configured to evaluate the 3D tracking of the performed task based on the localization metrics and the 3D tracking metrics and provide feedback to the user based on the performance using machine learning and AI based algorithms.

[0051] Another embodiment is a synthetic simulation environment for neuroendoscopic surgery with any 3D printed physical / synthetic models such as skull / skin / dura / brain / tumor structures with interchangeable modules after training.

[0052] Another embodiment is a training method that includes data collected following surgical simulation experiments such as anatomical structure identification, ear piercing, and pituitary tumor resection.

[0053] Another embodiment is a method to validate the usefulness of the developed system by establishing face validity, content validity, construct validity, and concurrent validity measures. The action was to use a 3D tracking instrument, starting from a reference point placed outside the simulator, and reach the anatomical structures inside the simulator using an endoscope on the one hand and an instrument on the other.

[0054] Yet another embodiment is to identify relevant anatomical structures to simultaneously train the anatomical structure and the handling of the endoscope (Figure 5).

[0055] Another embodiment is localization metrics and 3D tracking metrics for neuroendoscopic skill assessment. Another embodiment is endoscopic camera input synchronized with 3D tracking to provide additional sensor information for skill assessment. 3D information of the instrument and endoscopic camera input are captured and stored for feedback on performance skills. These sensor data are synchronized and provided as input to an artificial intelligence (AI)-based skill assessment module.

[0056] Another embodiment is an artificial intelligence (AI)-based skill assessment module that uses machine learning and deep learning-based algorithms to segment and track the equipment and includes an automated assessment of skill to estimate the trainee's level of expertise in the form of an objective Likert-scale-like metric that correlates with the subjective assessment of the expert. The tracking data and endoscopic video output can be utilized for the expert's assessment. It can also be used by the trainee to assess his or her performance to date and can be accessed using a web interface or on demand. The web module can include interactions by the trainee and by the expert for assessment. Another embodiment is an interface for providing trainees with personalized and specific feedback and skill scoring based on performance and skill level.

[0057] Another embodiment is the use of an AI-based skill assessment module directly for surgical video analysis, as it also includes specific paths and embeddings that can be modified for real surgical scenarios to assess skill levels, which can include magnetic field sweeps and interactions with the background as part of the assessment.

[0058] According to one embodiment, a single simulator serves as image guidance, 3D tracking and localization of instruments in CT / MRI slices and virtual models, as well as for skill evaluation of performance. It provides training for neuroendoscopy with anatomical orientation, CT visualization, anatomy of relevant structures, and hands-on skill of procedure-based neuroendoscopic skills. 3D tracking leads to accurate estimation of posture, localization of instruments during operation, helps in evaluation of surgeons with 6 degree of freedom data, and differentiates between expert and novice performance. This is done with a lightweight tracker wirelessly attached to one or more instruments. This tracker also helps in accurate localization in CT / MRI, thereby identifying the exact reach position by different users. The invention includes an artificial intelligence / deep learning based skill evaluation of image guidance / navigation and trainee skills. It includes segmentation and tracking of instruments, as well as automatic skill evaluation using objective Likert scale and specific feedback on individual skill level. This evaluation module includes specific paths and embeddings that can be modified for real surgical scenarios for evaluation of surgical skills.

[0059] The developed system is independent of the camera position by providing an external reference. The camera also tracks the external reference and estimates its rotation and translation relative to the camera center. The coordinate geometry is then mapped to obtain the position of a tracker attached to the rear end of the surgical instrument relative to the external reference.

[0060] Some of the non-limiting advantages of the 3D tracking and navigation simulator for neuroendscopy are:

[0061] 1. A multi-purpose simulator for simulation of neuroendoscopic surgery with image guidance, 3D tracking of instruments, and machine learning / AI based skills assessment. 2. Utilized for development of hands-on skills and visual orientation for neuroendoscopic surgery, accompanied by an image-guided system, providing training for neuronavigation / image guidance along with skill assessment with the help of 3D XYZ-based tracking of surgeon's actions; enabling procedure-based skills development along with CT and 3D visualization of relevant anatomical structures. 3. Objective assessment of surgeon's performance based on virtual localization metrics and 3D tracking metrics based on precise localization of anatomical structures, with evaluation parameters such as hand-eye coordination, instrument-tissue manipulation, dexterity, procedural flow and effectiveness. 4.3D models can be obtained from CT / MRI of the human anatomy, with the CT visualization including three planes (horizontal, coronal, and sagittal). 5. Providing 3D tracking and evaluation based on XYZ coordinates. It can also be combined with other sensors. It can also be extended to real surgical scenarios.

[0062] Although the 3D tracking and navigation simulator for neuroendoscopy with image guidance and skill assessment for neuroendoscopic surgery skills training and the method thereof are described in language specific to structural features and / or methods, it should be understood that the implementations disclosed in the above sections are not necessarily limited to the particular features or methods or apparatus described. Rather, the particular features are disclosed as examples of implementations of the 3D tracking and navigation simulator for neuroendoscopy with image guidance and skill assessment for neuroendoscopic surgery skills training and the method thereof. [Explanation of symbols]

[0063] 1 Camera unit 2 Fixing platform 3 Standard conformity simulation model 1 4. Fixture Tracker 5 Reference Units 6 Visualization screen 7 Endoscopy 8 Flaps / attachments for simulation models 9 Standard Compliant Simulation Model 2 10 Platform Fixtures 11 Computer Workstations / Systems 12 Endoscope Display 13 Endoscope System 14 Intervertebral foramen 15 Left cribriform plate 16 Right anterior clinoid tip 17 Right optic root canal 18 Midpoint of the tubercle sellae 19 Inferior border of left superior orbital fissure 20 Left foramen ovale 21 Right foramen rotundum 22 Right foramen spinosum 23 Tip of right posterior clinoid process 24 Saddle back point 25 Right external auditory canal 26 Left jugular foramen 27 Right hypoglossal foramen 28 Midpoint of the anterior edge of the foramen magnum

Claims

1. 1. A three-dimensional tracking and navigation simulator system for image-guided neuroendoscopy, comprising: - a standard-based physical model (3 and / or 9) of an anatomical region associated with neuroendoscopy, the standard-based physical model being mapped with CT / MRI slices of said anatomical structure; a virtual model of the anatomical region associated with neuroendoscopy; at least one surgical instrument operable to perform neuroendoscopy, the at least one surgical instrument being associated with a 3D tracking component (4); a camera unit (1) for tracking the position of said at least one surgical instrument with said 3D tracking component relative to a reference unit (5) of said physical model; one or more displays (6, 12) for displaying the position of the at least one surgical instrument within the CT / MRI slices and an endoscope output; a computer workstation (11) operatively coupled to said physical model and said camera unit (1), a localization and tracking module configured to track a task performed by a user with at least one surgical instrument and an endoscope (7) and to estimate the position and rotation of the at least one surgical instrument in six degrees of freedom (DOF); a virtual space mapping module configured to map locations to corresponding anatomical locations within the virtual model to provide navigation of the surgical instrument within the virtual model; and an evaluation module configured to evaluate 3D tracking of the performed task based on localization metrics and 3D tracking metrics and provide feedback to the user based on performance using machine learning and AI based algorithms.

2. 2. The simulator system of claim 1, wherein the tracking component is a three-plane instrument tracker, the three-plane tracker consisting of planes at different angles to one another, operably adapted to the at least one surgical instrument, the tracker having ArUco markers affixed thereto.

3. The simulator system of claim 1, comprising said synthetic simulation environment for neuroendoscopic surgery with any 3D printed physical / synthetic models such as skull / skin / dura / brain / tumor structures with exchangeable modules after training.

4. The simulator system of claim 1 , wherein the camera unit is configured to track ArUco markers, and the camera is mounted on a platform such that both the instrument tracker and the reference unit are visible in a given frame.

5. 2. The simulator system of claim 1, wherein the localization and tracking module is configured to collect data from the camera and localize a position of a tip of the surgical instrument relative to the reference unit to provide a spatial position and orientation of the surgical instrument.

6. 2. The simulator system of claim 1, wherein the virtual space mapping module is configured to align the instrument tip of the surgical instrument in the physical space with respect to the virtual space and locate the tip on the CT / MRI slices, and the position of the tip of the instrument estimated with respect to the reference coordinate system is projected onto the CT / MRI of the virtual model for visualization and mapped to a respective anatomical location, and the CT / MRI is visualized in all three planes: horizontal, coronal, and sagittal.

7. The simulator system of claim 1 , comprising data collected in accordance with surgical simulation experiments, such as anatomical structure identification, ear piercing, and pituitary tumor resection, regarding actions performed by the user.

8. The simulator system of claim 1 , wherein the system is configured to provide visual and tactile feedback received with the aid of the endoscope and the physical model.

9. 2. The simulator system of claim 1, wherein input from an endoscopic camera is synchronized with the 3D tracking to provide sensor information to the evaluation module, and 3D information of the instrument and the endoscopic camera input is captured and stored in the workstation for feedback and scoring on the skill of the task performed by the user.

10. 2. The simulator system of claim 1, wherein the assessment module is an artificial intelligence (AI) based skills assessment module configured to use machine learning and deep learning based algorithms to segment and track the equipment and automatically assess the skills to estimate the user's level of expertise in the form of an objective Likert scale-like metric that correlates with an expert's subjective assessment.

11. A method for three-dimensional tracking and navigation for image-guided neuroendoscopy in a simulator system according to claim 1.

Images