Camera Navigation System

Abstract

A camera navigation system is provided. The camera navigation system includes an insert or grid with multiple markers configured to assist in tracking a simulated laparoscope. The camera navigation system generates a digital environment containing computer-generated elements for different camera navigation training, which are displayed on a monitor for the user to view. The position of the simulated laparoscope relative to the training environment is simulated as a cursor in the digital environment, and the movement of the simulated laparoscope relative to the training environment corresponds to the movement of the cursor in the digital environment. The camera navigation system is compatible with both zero-degree and angled laparoscopes and includes camera navigation training provided to train the user on different camera navigation skills. Feedback is also provided characterizing the user's performance in the camera navigation training.

Description

TECHNICAL FIELD

[0001] 〔CROSS-REFERENCE TO RELATED APPLICATIONS〕 This application claims priority and the benefit of U.S. Provisional Patent Application Serial No. 63 / 587,009, filed September 29, 2023, entitled "Camera Navigation System", and U.S. Provisional Patent Application Serial No. 63 / 571,243, filed March 28, 2024, entitled "Camera Navigation System", both of which are hereby incorporated by reference in their entirety into the present specification.

[0002] The present application relates to surgical training, and in particular, to an apparatus and method for training camera navigation skills in a digitally generated laparoscopic environment.

BACKGROUND ART

[0003] During laparoscopic or endoscopic surgery, someone manipulates the laparoscope or endoscope to provide a view within the patient. This view is displayed on a nearby video display screen or monitor. In various situations, someone other than the surgeon performing the laparoscopic or endoscopic surgical procedure may control the laparoscope or endoscope. For example, a medical student or intern may be assigned the task of navigating the laparoscope or endoscope, and these individuals must quickly learn the skills necessary to provide optimal visibility, such as recognizing and centering the surgical field, maintaining the correct horizontal axis, grasping the timing for zooming in or out, maintaining a stable image, and tracking the surgical instruments used by the surgeon as they move. An experienced camera operator with a thorough grasp of the situation can anticipate the surgeon's next movement and move the laparoscope or endoscope accordingly. Therefore, camera navigation is extremely critical for properly performing laparoscopic or endoscopic surgery. Accordingly, one important part of laparoscopic or endoscopic camera navigation skill training is to provide the ability to practice complex camera movements for the purpose of following the movements of a surgeon performing a surgical procedure, while overcoming the difficulties described above.

[0004] While camera movement can vary depending on the specific surgical procedure being performed, there is a need for simple and universal methods to train and evaluate camera navigation skills. Several studies have been initiated to investigate the impact of poor camera navigation in surgical cases, predicting that suboptimal imaging can lead to surgeon frustration and inefficiency. These studies have also shown that poor visibility of the surgical field can significantly disrupt the flow of surgery, forcing surgeons to interrupt procedures. Furthermore, delays caused by poor visibility during surgical procedures increase the time spent in the operating room.

[0005] Not only do new doctors need to learn camera navigation skills, but even doctors with some experience need to either 1) practice further / refining the camera navigation skills they have already acquired, or 2) learn and practice new camera navigation skills that are specific to newly introduced surgical procedures.

[0006] While training can be acquired in the operating room, there is growing interest in devising faster and more efficient training methods, preferably outside the operating room. Surgeons who have achieved a reasonable skill level outside the operating room are well-prepared when they enter the operating room. As a result, surgeons can gain optimized and valuable operating room-related experience, which can reduce patient risk and lower costs. Therefore, there is a need for a specialized camera navigation system that can be used to acquire and train camera navigation skills useful for assisting surgeons in surgical procedures. Such training tools would enable trainees to acquire the camera navigation skills necessary to provide surgeons with the best possible view before entering the operating room. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] U.S. Patent Application Serial Number 13 / 248,449 [Overview of the project]

[0008] According to various embodiments of the present invention, a camera navigation system is provided which includes a simulated laparoscope having a camera, a training environment having markers provided for tracking the position of the simulated laparoscope, a scope view generator, and / or a digital environment and a monitor for displaying feedback on camera navigation training. The scope view generator can generate a digital environment having computer-generated elements corresponding to the training environment, track the position of the simulated laparoscope relative to the training environment by processing image data acquired from the simulated laparoscope including markers at regular time intervals, calculate positional information of the simulated laparoscope from the image data, update the digital environment using the positional information, monitor the user's performance during camera navigation training by comparing the positional information of the simulated laparoscope with computer-generated elements in the digital environment, and generate feedback based on the user's performance during camera navigation training.

[0009] According to various embodiments of the present invention, a device for tracking the position of a simulated laparoscope is provided. The device includes a simulated laparoscope having a camera to be tracked, a training environment including an insert or grid having many unique markers, and / or a scope view generator used to determine the position of the simulated laparoscope. The scope view generator includes a memory storage device that stores the position of each of the unique markers, an executable application such as a computer vision application that can identify which unique markers are captured in the image data and determine the position information of the simulated laparoscope based on the identified markers, and a scope view generator that can generate a digital environment and computer-generated elements embedded in the digital environment.

[0010] According to various embodiments of the present invention, a camera navigation system is provided. The camera navigation system is used with a simulated angled laparoscope having a camera and a rotation sensor. The camera navigation system also has a training environment which includes markers used to track the position of the simulated angled laparoscope. It also includes a scope view generator which is provided to generate a digital environment which includes computer-generated elements based on camera navigation training selected by the user, corresponding to the training environment. The scope view generator tracks and determines the position of the simulated laparoscope based on image data and angles measured from the rotation sensor. The scope view generator updates the digital environment using the position information of the simulated laparoscope and monitors the performance of the camera navigation training by the user. The scope view generator can detect whether there is a collision between a cursor (a digital environmental representation of the position of the simulated laparoscope) and one or more computer-generated elements (such as targets).

[0011] According to various embodiments of the present invention, an insert or grid for tracking a simulated laparoscope is provided. The insert or grid has markers that are unique to each other. The markers are arranged in a predetermined pattern covering a predetermined space. The position of the simulated laparoscope can be determined based on which markers are captured. Information regarding the positions of the markers on the insert or grid is stored in memory and can be retrieved by a scope view generator when determining the position of the simulated laparoscope.

[0012] According to various embodiments of the present invention, a device for tracking the position of a simulated angled laparoscope is provided. The device includes a simulated angled laparoscope, which has a camera and a sensor designed to detect the rotation of the simulated angled laparoscope. The device also includes a training environment, which includes an insert or grid on which many unique markers are placed. The device also includes a scope view generator provided for identifying the position of the simulated angled laparoscope, which includes a memory for storing the position of each unique marker and an executable application, such as a computer vision application, for identifying the unique markers captured as image data by the simulated angled laparoscope and determining the position of the simulated angled laparoscope from the image data and angle rotation from the sensor. The device also includes a scope view generator that generates a digital environment corresponding to the training environment and computer-generated elements incorporated into the digital environment.

[0013] This invention can be better understood by interpreting it in relation to the accompanying drawings, where the reference numerals throughout the figures indicate the same parts. [Brief explanation of the drawing]

[0014] [Figure 1A] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1B] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1C] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1D] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1E] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1F] This figure shows various embodiments of a camera navigation box and a part thereof. [Figure 1G]This is a diagram illustrating various embodiments of a camera navigation box and a portion thereof. [Figure 2A] This is a diagram illustrating various embodiments of a camera navigation system. [Figure 2B] This is a diagram illustrating various embodiments of a camera navigation system. [Figure 2C] This is a diagram illustrating various embodiments of a camera navigation system. [Figure 3] This is a diagram illustrating an embodiment of an insert or grid having a plurality of markers. [Figure 4A] This is a diagram illustrating various embodiments of a computer-generated menu for a camera navigation system or a portion thereof. [Figure 4B] This is a diagram illustrating various embodiments of a computer-generated menu for a camera navigation system or a portion thereof. [Figure 4C] This is a diagram illustrating various embodiments of a computer-generated menu for a camera navigation system or a portion thereof. [Figure 5] This is a diagram illustrating various embodiments of feedback generated by a camera navigation system and displayed in a digital environment. [Figure 6] This is a diagram illustrating various embodiments of feedback generated by a camera navigation system and displayed in a digital environment. [Figure 7] This is a diagram illustrating various embodiments of feedback generated by a camera navigation system and displayed in a digital environment. [Figure 8] This is a diagram illustrating various embodiments of feedback generated by a camera navigation system and displayed in a digital environment. [Figure 9] This is a diagram illustrating various embodiments of feedback generated by a camera navigation system and displayed in a digital environment. [Figure 10A] This is a diagram illustrating an exemplary calculation of simulated laparoscopic rotation performed by a camera navigation system. [Figure 10B] FIG. 2 is a diagram showing an exemplary calculation of simulated laparoscope rotation performed by a camera navigation system. [Figure 11] FIG. 3 is a diagram showing an exemplary embodiment of a meter in a digital environment used to quantify a user's performance. [Figure 12] FIG. 4 is a diagram showing an exemplary calculation of the observation distance between a simulated laparoscope and a target. [Figure 13] FIG. 5 is a diagram showing an exemplary embodiment of a distance meter. [Figure 14A] FIG. 6 is a diagram showing an exemplary embodiment of trace camera navigation training. [Figure 14B] FIG. 7 is a diagram showing an exemplary embodiment of trace camera navigation training. [Figure 15] FIG. 8 is a diagram showing an exemplary calculation for determining the proficiency level of trace camera navigation training. [Figure 16A] FIG. 9 is a diagram showing an exemplary embodiment of follow camera navigation training. [Figure 16B] FIG. 10 is a diagram showing an exemplary embodiment of follow camera navigation training. [Figure 17A] FIG. 11 is a diagram showing an exemplary embodiment of framing camera navigation training. [Figure 17B] FIG. 12 is a diagram showing an exemplary embodiment of framing camera navigation training. [Figure 18A] FIG. 13 is a diagram showing exemplary camera navigation training for a simulated angled laparoscope. [Figure 18B] FIG. 14 is a diagram showing exemplary camera navigation training for a simulated angled laparoscope. [Figure 18C] FIG. 15 is a diagram showing exemplary camera navigation training for a simulated angled laparoscope. [Figure 18D] FIG. 16 is a diagram showing exemplary camera navigation training for a simulated angled laparoscope. [Figure 18E] FIG. 17 is a diagram showing exemplary camera navigation training for a simulated angled laparoscope. [Figure 19] This figure shows an exemplary embodiment of a camera navigation system. [Figure 20] This diagram illustrates an example of RGB to grayscale conversion. [Figure 21] This figure shows an example of a binary image. [Figure 22] This is a diagram illustrating an example of contour calculation. [Figure 23] This diagram illustrates a quadrilateral-shaped filtering process. [Figure 24A] This figure illustrates an exemplary filtering process using the corners of a quadrilateral. [Figure 24B] This figure illustrates an exemplary filtering process using the corners of a quadrilateral. [Figure 24C] This figure illustrates an exemplary filtering process using the corners of a quadrilateral. [Figure 25] This figure shows an exemplary transformation matrix for determining distortion. [Figure 26] This figure illustrates the exemplary steps for adding a corner point. [Figure 27] This diagram shows an example of the steps involved in labeling each corner. [Figure 28] This figure shows an exemplary step of reprojecting a corner point having an identified corner. [Figure 29] This figure shows an exemplary embodiment of a camera navigation system. [Figure 30] This is a diagram illustrating an exemplary surgical trainer. [Figure 31] This figure shows a portion of an example of a simulated angled laparoscope. [Modes for carrying out the invention]

[0015] According to various embodiments, a camera navigation system is provided. The camera navigation system is designed to train the user in various camera navigation-related skills. The user can also learn and practice using different types of simulated laparoscopes, such as zero-degree laparoscopes and angled laparoscopes. The camera navigation system is designed to have a training environment housed in a camera navigation box and / or surgical trainer to simulate actual surgical situations in which a user (e.g., a surgeon) manipulates a laparoscope within a patient during surgery. The simulated laparoscope is used to acquire image data from the training environment. In various embodiments, a scope view generator is provided that uses the image data acquired from the simulated laparoscope to determine the current position data of the simulated laparoscope relative to the training environment. The scope view generator is also configured to generate a digital environment and corresponding computer-generated elements that utilize the acquired image data. The generated digital environment is then sent to a monitor for the user to see. Similar to how a surgeon relies on a monitor to view the surgical field within a patient, the user views a digital environment on a monitor while manipulating a simulated laparoscope in relation to the training environment.

[0016] According to various embodiments, the reference to the training environment corresponds to an insert or grid configured to facilitate tracking the position of the simulated laparoscope relative to the training environment. The “position” of the simulated laparoscope referred to below is characterized by the corresponding six degrees of freedom, and in various embodiments, the “position” of the simulated laparoscope is identified as the distal end of a camera having x, y, z coordinates and rotational values ​​(e.g., roll, pitch, and yaw) associated with the simulated laparoscope.

[0017] In various embodiments, the digital environment corresponds to the space defined by the training environment. The digital environment is displayed on a monitor for the user to see. In various embodiments, a scope view generator is configured to incorporate computer-generated elements into the digital environment to provide various menus and camera navigation training functions. In various embodiments, the digital environment may include an augmented reality embodiment in which computer-generated elements are superimposed on real-time image data acquired from a simulated laparoscope (i.e., image data acquired from the training environment) and displayed on the monitor.

[0018] In various embodiments, computer-generated elements (generated by a scope view generator and included in the digital environment) include elements such as buttons, targets, cursors, meters, and obstacles. In various embodiments, computer-generated elements are used in conjunction with the digital environment to provide menus and various camera navigation training for the purpose of teaching and training camera navigation-related skills.

[0019] In various embodiments, all information relating to computer-generated elements is stored in the camera navigation system's memory. For example, information about computer-generated elements stored in memory includes how these elements are used in camera navigation training, the specific location of these elements in the digital environment when used in camera navigation training, and / or characteristics (i.e., whether the elements are stationary, opaque, moving, or their shape). In various embodiments applicable to augmented reality, computer-generated elements can be used to enhance image data captured by a simulated laparoscope to provide an augmented view displayed on a monitor. For example, various augmented embodiments provide the incorporation of various computer-generated elements, such as buttons and targets, superimposed on the image data displayed on the monitor. The placement of each computer-generated element intended for image data augmentation is also stored in memory to identify when the augmentation element is available, where it should be placed, and / or how it behaves.

[0020] In various embodiments, the memory used to store, for example, information for camera navigation training and computer-generated elements used in training, may be a memory storage device physically located at the same location as the scope view generator. In various embodiments, the memory may reside on a remote server or in the cloud (i.e., at a location away from the scope view generator), or the memory may be included in or attached to such a remote server or in the cloud. In various embodiments, the scope view generator is configured to retrieve appropriate information stored in memory for menu generation and / or for performing camera navigation training or related operations on the camera navigation system.

[0021] In various embodiments, the camera navigation system includes at least one monitor. Through the use of the monitor, the user of the camera navigation system can view the training environment captured by the camera of the simulated laparoscope. In various embodiments, the training environment can be housed within a camera navigation box and / or surgical trainer. Therefore, according to various embodiments, it is impossible or limited to directly view the training environment without relying on a scope or camera. In various embodiments, the monitor is configured to provide the user with a field of view for viewing the training environment in embodiments where direct viewing is generally impossible. By viewing the training environment (housed within the camera navigation and / or surgical trainer) through the use of the monitor, the surgeon's need to view the surgical field within the patient's body cavity during a surgical procedure is simulated.

[0022] Generally, surgical personnel such as surgeons require highly skilled surgical techniques, and this is especially true for performing laparoscopic surgery. In laparoscopic surgery, several small incisions are made in the abdomen to insert trocars or small cylindrical tubes, and surgical instruments and a laparoscope are inserted into the abdominal cavity through these incisions. During the surgery, the laparoscope is used to illuminate the surgical field and then to capture magnified images from inside the patient's abdominal cavity and transmit them to a monitor. The magnified images displayed on the video monitor show the surgeon a magnified view not only of the surgical field but also of the surrounding organs and tissues. The surgeon performs the laparoscopic surgery by manipulating the surgical instruments positioned through the trocars while viewing a live video feed on the monitor transmitted via the laparoscope.

[0023] Because surgeons rely on live video feeds on a monitor rather than directly observing organs and tissues in the surgical field with the naked eye, problems arise because visual information of the three-dimensional space related to the surgical field is instead obtained from two-dimensional images on the monitor. Presenting a three-dimensional environment through two-dimensional images results in a significant loss of information; for example, viewing two-dimensional images as a guide when manipulating surgical instruments in three dimensions reduces depth perception.

[0024] Furthermore, the movement of the simulated laparoscope is constrained by the patient's anatomical structure. For example, since the trocar is inserted through a small incision and supported by the abdominal wall, the manipulation of the instrument and laparoscope is restricted by the abdominal wall. Specifically, the abdominal wall creates a fulcrum effect on the instruments and laparoscope used in laparoscopic surgery. This fulcrum effect defines a point of angulation that restricts the range of motion of the instrument and laparoscope.

[0025] Furthermore, when the hand moves in one linear direction along with the laparoscope, the movement of the tip in the opposite direction is amplified when viewed on the monitor. Not only do the movements of the instrument and laparoscope appear in opposite directions on the screen, but the amplified tip movement depends on the ratio of the length of the instrument and laparoscope above the abdominal wall. The lever effect not only amplifies the movement but also increases the force at the tool tip that is reflected based on the movement. Thus, the manipulation of laparoscopes and surgical instruments using a fulcrum by the surgeon is not intuitively obvious and therefore requires intentional learning, which can be provided by the camera navigation system described herein.

[0026] In various embodiments, surgical instruments and laparoscopes are positioned through ports with seals. These seals induce stick-slip friction. Stick-slip friction can occur, for example, when the tool direction reverses rapidly, such as when quickly changing from pulling to pushing tissue. During stick-slip motion, the rubber components of the seal rub against the tool shaft, causing friction or movement of the tool against the seal. If the friction is overcome, the instrument slides against the seal. Stick-slip friction (also known as oil-canning) at the contact surface between the seal and the instrument and / or laparoscope imposes nonlinear forces on the instrument and / or laparoscope, resulting in jarring of the image on the live video feed displayed on the monitor. During surgical procedures, the jarring resulting from stick-slip friction can be distracting for the surgeon. Therefore, practice is needed to vary the insertion depth of the laparoscope and surgical instruments to prevent or minimize the occurrence of stick-slip friction.

[0027] Hand-eye coordination skills are also necessary and must be practiced. Hand-eye coordination during actual surgical procedures correlates the movement of the hand, accompanied by the movement of the tool tip within the patient's body cavity, with the movement of the tool tip shown via a live feed on a monitor. In laparoscopic surgery, the surgeon cannot directly palpate the tissue with their hands, resulting in reduced tactile sensation through the tools. Because tactile sensation is diminished and distorted, surgeons must hone a set of core haptic skills, which are fundamental to improving laparoscopic surgical skills. Acquiring these skills is one of the main challenges in laparoscopic training, and according to various embodiments of the present invention (described in further detail below), this disclosure describes various embodiments aimed at providing a method by which users can improve their performance with camera navigation technology and other related surgical skills.

[0028] In various embodiments described in further detail below, a camera navigation system is provided having a digital environment that includes various different computer-generated elements. The digital environment is influenced by captured image data acquired by the simulated laparoscope from the training environment. A camera navigation box and / or surgical trainer is used to simulate the patient's body cavities. A correspondence is established between the position of the simulated laparoscope relative to the training environment and the display of a cursor in the digital environment, specifically by tracking the position of the simulated laparoscope using markers related to the training environment and providing a correlation between this position and the cursor position in the digital environment. The camera navigation system provides a view using computer-generated elements displayed on a monitor, allowing the user to simulate actual surgical procedures that indirectly view the surgical space and practice where and how to move the laparoscope from one position to the next.

[0029] In various embodiments, a simulated laparoscope (having a camera sensor) is configured to capture images of the training environment. The images captured by the simulated laparoscope can then be used to generate a digital environment and other computer-generated elements (e.g., a cursor), which can then be displayed on a monitor. In various embodiments, the simulated laparoscope has one or more sensors.

[0030] In various embodiments, the camera navigation system includes a camera navigation box configured to simulate the surgical space within a patient, a surgical trainer configured to simulate the patient's torso, multiple markers within the camera navigation box, LED (light-emitting diode) illumination, and / or a simulated laparoscope used to capture images of the markers from within the camera navigation box.

[0031] In various embodiments, the camera navigation system includes a scope view generator that generates and updates a digital environment using information about the position of a simulated laparoscope. In various embodiments, the simulated laparoscope is represented as a cursor in the digital environment. The cursor is positioned in the digital environment based on the position of the simulated laparoscope, indicated by markers captured by the simulated laparoscope. The cursor can also be used as an actuation mechanism to initiate and / or interact with one or more computer-generated elements (e.g., buttons, targets) in the digital environment, and the digital environment is displayed on a monitor for the user to see. A series of camera navigation training is provided for instruction and practice of camera navigation or surgical skills, such as manipulating a laparoscope in a patient's body form.

[0032] According to various embodiments, the training environment can incorporate the use of a surgical trainer (3010). The surgical trainer (3010) is configured to simulate the torso of a patient. The training environment is housed within the surgical trainer (3010), which is configured to require the user to be informed of how to manipulate the simulated laparoscope relative to the training environment by relying on a digital display on a monitor, by obstructing a direct view of the training environment. According to various embodiments, the surgical trainer (3010) defines an internal cavity (3024) by having a top cover (3016) spaced apart from the base (3018). Further details of the surgical trainer (3010) are shown below in reference to Figure 30.

[0033] According to various embodiments, the camera navigation system tracks the position of a simulated laparoscope using a training environment (i.e., an insert or grid with multiple unique markers). The training environment can be housed within a camera navigation box (100), an example of which is shown in Figure 1A. The insert or grid includes multiple markers (e.g., QR codes®) specifically designed and placed within the training environment to facilitate the identification of the simulated laparoscope's positional information. Figures 1A to 1G show various embodiments of the camera navigation box. In various embodiments, the camera navigation box is designed to house the insert or grid internally. For example, the insert or grid can be placed on the base (102) of the camera navigation box and / or on one or more side walls (104). Furthermore, the camera navigation box can be housed within a surgical trainer, as shown in Figure 30, to further simulate procedures within a patient's body cavity. Further details regarding the camera navigation box are provided below.

[0034] As described above, according to various embodiments, the insert or grid can be configured to be positioned on the base and / or side wall within the camera navigation box (100). The digital environment (which is a digital simulation displayed on a monitor for the user to see) is configured to have a size or area corresponding to the size or area covered by the insert or grid. Furthermore, the cursor movement and the training environment (e.g., the insert or grid) correspond directly (e.g., one-to-one) to the position of the simulated laparoscope. In various embodiments, the position of the simulated laparoscope relative to the training environment is represented as a cursor in the digital environment. Thus, the user can move the simulated laparoscope and expect the cursor to move similarly in the digital environment.

[0035] Figures 1A to 1G show various embodiments of a camera navigation box. The camera navigation box is designed to accommodate an insert or grid. In one embodiment, as shown in Figure 1A, the camera navigation box (100) has a base (102) and one or more side walls (104). The base (102) and the one or more side walls (104) have apertures (103) used to attach the base (102) and the side walls (104) to each other to form and maintain the shape of the camera navigation box (100). The base (102) and the side walls (104) can be connected via screws, pins, or other connecting structures that are harmonized with the apertures (103) associated with the base (102) and the side walls (104). In various embodiments, the camera navigation box (100) is designed to minimize the effects of external illumination and reflections on the simulated laparoscope used inside it.

[0036] Many of the figures and related descriptions contained herein use and / or refer to specific embodiments of camera navigation boxes, but camera navigation boxes used in relation to inserts or grids can have any number of different shapes depending on the need to simulate the space in which a simulated surgical procedure is being performed. In some embodiments, the camera navigation box (100) may correspond to a rectangular box having a base that defines the rectangular box shape and one or more side walls. In some embodiments, the camera navigation box (100) may have any other quadrilateral (e.g., square, trapezoid) defined by the use of its bottom and one or more side walls. In some embodiments, it may also have geometric shapes with different structures (e.g., triangle, pentagon, circle).

[0037] The choice of shape for the camera navigation box (100) can be useful in facilitating the foldability or collapsibility of the entire structure. For example, if the camera navigation box (100) is foldable or collapsible, it can be designed to reduce the space occupied by the entire system, and therefore to be easily transported, thus facilitating the transport of the entire system.

[0038] As described above, the camera navigation system is designed to track the position of the simulated laparoscope using inserts or grids housed within the camera navigation box and to display the corresponding position via a cursor in a digital environment viewed on a monitor. Therefore, for this tracking to function, the inserts or grids must be properly positioned within the camera navigation box. Specifically, the edges of the inserts or grids positioned on the base must be aligned with the edges of the inserts or grids positioned on one or more side walls so that the markers on each insert or grid are aligned in a predetermined pattern and order. The placement and position of the markers relative to the training environment are stored in memory, and therefore deviations from the expected placement and position of the markers may result in errors in the tracking that provides the positioning of the simulated laparoscope. Further details regarding the inserts or grids and associated markers are described below.

[0039] In other embodiments, such as an angled laparoscope, it may be necessary to define a horizontal plane using an insert or grid (300). The horizontal plane can be customized based on training during practice. In various embodiments, the horizontal plane can be established as the bottom of the image sensor of the angled laparoscope. In another embodiment, a camera or image sensor can be introduced into a camera navigation box and / or surgical trainer to view the training environment in various different directions, and these can be inserted through the top or front to simulate different surgical procedures, for example.

[0040] In various embodiments, the camera navigation box (100) is designed to house an insert or grid (300) and may further include an upper or ceiling portion. The upper or ceiling portion can be used to further make it difficult for the user to directly observe the insert or grid (300) from the top of the camera navigation box. The upper or ceiling portion may be useful in embodiments that do not include a surgical trainer.

[0041] In various embodiments, the base (102) of the camera navigation box (100) can be surrounded by one or more side walls (104). In various embodiments, each side wall (104) is positioned perpendicular to the base (102). In various embodiments, the base (102) of the camera navigation box (100) can have one or more sides that do not have side walls (104). The absence of side walls on one or more sides of the base (102) allows for an opening in the camera navigation box (100), as shown in Figure 1A, through which a simulated laparoscope can be inserted to access the training environment.

[0042] In various embodiments, the camera navigation box (100) is designed to be portable. The camera navigation box (100), having inserts or grids, is movable to different locations and can be used with different configurations. The portability of the camera navigation box (100), facilitated in part by the foldability / collapseability of the camera navigation box (100), facilitates the transport of the camera navigation box (100) by making it easier to store and / or move parts of the camera navigation system. By making the camera navigation box (100) foldable or collapsible, the overall footprint of parts of the camera navigation system can be reduced or minimized when transporting or storing them when not in use. In various embodiments, the camera navigation box (100) is also configured to be robust, easily manufacturable, and reusable.

[0043] In various embodiments, the insert or grid housed within the camera navigation box (100) includes a non-reflective surface that prevents or minimizes reflections that may occur, for example, due to lighting provided within the camera navigation box (100) or related to the entire camera navigation system. In various embodiments, the non-reflective surface of the insert or grid is coated or formed with a material such as a mat that minimizes or eliminates reflections. In various embodiments, the camera navigation box (100) can define the training environment instead of the insert or grid. Specifically, instead of having multiple markers placed on the insert or grid (for example, printed on adhesive paper that can adhere to or otherwise attach to one or more surfaces of the camera navigation box (100) (i.e., the base and / or sidewalls)), multiple markers can be placed directly on the base and / or sidewalls of the camera navigation box (100).

[0044] In various embodiments, the insert or grid can be printed on a sheet cut to the dimensions of the base (102) and / or sidewalls (104) of the camera navigation box (100) so that markers on the insert or grid can be reliably and correctly aligned. In various embodiments, the insert or grid used with the camera navigation box (100) can be configured not only to be removable but also interchangeable with other different inserts or grids. As a result, the same camera navigation box (100) can be fitted with different inserts or grids, and each different insert or grid can be used in different situations (e.g., different laparoscopes, different camera navigation training, different spatial volumes). For example, the insert or grid can also have different shapes (e.g., square, rectangular, oval) based on the shape and size of the camera navigation box. In another example, different sized markers may be required for specifications or features associated with a particular type or brand of laparoscope so that the simulated laparoscope can correctly capture image data used to track the position of the laparoscope relative to the training environment.

[0045] In various embodiments, markers are placed on the surface of an insert or grid. These markers, such as QR codes, can be printed, etched, or otherwise applied to the insert or grid. In various embodiments, instead of providing the markers on the insert or grid, they can be printed, etched, or otherwise applied directly to the base (102) and / or sidewall (104) of the camera navigation box (100).

[0046] Referring again to Figures 1A to 1G, these figures show examples of embodiments of camera navigation boxes designed to be foldable or crushable. As described above, the foldability or crushability of the camera navigation box provides an easy way to transport at least a part of the camera navigation system, or an easy way to store a part of the camera navigation system when not in use.

[0047] In one embodiment, Figure 1B-1 shows two differences in the assembly of a camera navigation box (110) formed of separate components (e.g., a base (102), side walls (104), and a front section (106)). These separate components (e.g., the base (102), side walls (104), and front section (106)) can be manufactured and / or supplied separately. However, these separate components are configured to be assembled and mounted together. As shown in Figure 1B-1, each of these components has slits (108, 112) that allow for a sliding connection between the components. In various embodiments, for example as shown in Figure 1B-1, two side walls (104a and 104b) have a slit (108) near the rear of the camera navigation box (110). The slit (108) opens from the top of the side walls (104a, 104b) and extends to the central area of ​​the side walls (104a, 104b). A third sidewall component (104c) is provided to connect to these two sidewalls (104a and 104b). The slit in the third sidewall (104c) starts from the bottom of the third sidewall (104c) and extends to the central area. As a result, the slit in the third sidewall (104c) can be joined to the slit (108) of the two sidewalls (104a and 104b), as ultimately shown in the configuration of the camera navigation box (120). By providing the third sidewall component (104c), the upright arrangement of the two sidewalls (104a and 104b) relative to the base (102) can be maintained.

[0048] The side walls (104a-104c) are configured to attach to the base (102), in addition to having slits that facilitate attachment between the side walls. To facilitate this connection, the base (102) has a plurality of slits (112) associated with the base (102). These slits (112) are arranged along the periphery of the base (102) and can ultimately define the space of the training environment. In various embodiments, the side walls (104a-104c) may have various hook portions (114) at their bottoms for connecting them to the base. These hook portions (114) are designed to be repositioned so that, after being inserted into the slits (112) of the base (102), the side walls (104a-104c) cannot be pulled out unless the hook portions (114) are first realigned with the slits (112) of the base (102). A fully connected arrangement can be seen in Figure 1B-2.

[0049] Furthermore, as shown in Figures 1B-1 and 1B-2, the camera navigation boxes (110, 120) may also include a front section (106). In various embodiments, the front section (106) facilitates defining the periphery of the training environment (together with the other side walls (104a-104c)). In various embodiments, the front section (106) is removable, allowing for the insertion and removal of inserts or grids (described later) used for camera navigation training. The front section (106) of the illustrated camera navigation boxes (110, 120) may utilize complementary slits (116) that allow the front section (106) to slide-engage with the side walls (104a, 104b), similar to how the third side wall (104c) slides into the side walls (104a, 104b). In some embodiments, adhesive can be used not only for the connection between the base and the side walls, but also to attach and secure one of the side walls to the other side walls.

[0050] (For example, in Figure 1B-1) when the components of the camera navigation box (110, 120) are disassembled, the camera navigation box (110, 120) has a minimal footprint. As a result, at least the camera navigation box portion of the camera navigation system can be easily transported or stored.

[0051] In various embodiments of the camera navigation box (shown in Figures 1A-1G), the camera navigation box can be formed from separate elements or components. Therefore, in manufacturing, the elements or components can be standardized using templates to maintain consistency. The advantage is that the manufacturing of the camera navigation box is consistent, and the ability to ensure that the dimensions of the camera navigation box allow for satisfactory alignment of inserts or markers on a grid is provided. In various embodiments, the elements or components that make up parts of the camera navigation box (e.g., base, side walls) can be machined, laser cut, or die-cut.

[0052] Furthermore, by manufacturing the components separately and allowing the user to assemble the camera navigation box, the camera navigation box (or at least its components) can be shipped with a smaller footprint. Upon receiving the camera navigation box, the user can assemble the components together using various different types of fasteners (e.g., screws, pins) that interlock with holes or apertures (103) to secure the base and sidewalls to each other. In addition, in various embodiments, the fasteners (e.g., screws, pins) can be removable so that the camera navigation box can be disassembled. In various embodiments, the sidewalls can have dowel pins that are press-fitted into the base. Corresponding mating holes in each sidewall facilitate the connection between the base and the sidewall.

[0053] In various embodiments, components of the camera navigation box (e.g., the base and side walls) can be attached and secured to each other via various hinges (135). The hinges (135) also allow the components of the camera navigation box to be folded or crushed into a flat configuration during storage or transport, as shown in the embodiment of the camera navigation box (130). In various embodiments, the hinges (135) are configured to be used to attach the side wall (104) to the base (102). Generally, the hinges (135) are located on the outside of the camera navigation box to make the inside of the camera navigation box more planar or obstruction-free.

[0054] In various embodiments, a side wall (such as a rear side wall (104c)) can be attached to an adjacent side wall (104a and 104b) via complementary hooks and slits (116), similar to the method in which the front section (106) is attached to the side walls (104a and 104b) shown in Figure 1B-2.

[0055] Referring to Figures 1C-1 and 1C-3, the base 102 may include a surgical trainer aperture 118 used to secure the camera navigation box to the surgical trainer. In various embodiments, this surgical trainer aperture 118 is not included. In this way, the camera navigation box can be placed inside the surgical trainer and secured by other means (e.g., clips), or left inside the surgical trainer without being secured. In various embodiments, if the surgical trainer aperture 118 is not used to secure the camera navigation box to the surgical trainer, the user may also use the surgical trainer aperture 118 as a means of handling the device. For example, the user may insert their fingers into the surgical trainer aperture 118 and hold and carry the camera navigation box.

[0056] As will be explained below, various different types of hinges can be used to attach the side wall to the base. Examples of hinges (140a and 140b) are shown in Figures 1D-1 and 1D-2. With regard to the first hinge (140a) shown in Figure 1D-1, the first hinge can consist of two hinge plates (141, 142) held together by pins (144). In this configuration, the elements of the hinge (140a) can be manufactured separately and assembled later. These two hinge plates (141, 142) also have a structure (143) that extends upward away from the surface of the hinge plates (141, 142). These structures (143) can be used to attach the hinge to a corresponding aperture or hole on the base and / or side wall of the camera navigation box. Regarding the second hinge (140b) shown in Figure 1D-2, a similar arrangement can be observed, having two hinge plates (141, 142) connected to each other. However, instead of pins (144), epoxy overmolding (145) is used to attach the two hinge plates (141, 142). Such an implementation provides a different (potentially cleaner) appearance compared to the use of pins (144). However, overmolding (145) does not always cure flat, and the implementation may require additional processes compared to using pins (144) as described for the first hinge (140a).

[0057] In various embodiments, the hinge (135) may be formed and / or provided separately from the base (102) and side wall (104) of the camera navigation box (100). In this case, for example, the user needs to attach the hinge (135) to the base (102) and / or side wall (104). In various embodiments, attachment between the hinge (135) and the base (102) and / or side wall (104) of the camera navigation box can be done using adhesive, overmolding, and / or a hardware interface (e.g., screws). In various embodiments, the hinge (135) may be designed together with the base (102) and / or side wall (104) as a single monolithic component. In various embodiments, the hinge (135) may be designed as separate plates (141, 142) connected to the base (102) and / or side wall (104). The separate plates (141, 142) containing the hinges can be connected to each other (i.e., snap-fitted) during the subsequent assembly of the camera navigation box (100). Other types of hinges are also possible, as long as the movement of components of the camera navigation box (100), such as the side walls (104), allows the camera navigation box (100) to be folded and unfolded.

[0058] In various embodiments, the hinge (135) can be designed to open by a predetermined amount (e.g., 90 degrees). In various embodiments, the hinge (135) can be designed to lock in its current position when it reaches a predetermined angle during extension. The hinge (135) can be provided with a release mechanism to allow the hinge (135) to be folded again.

[0059] Referring to Figure 1D-1, the first hinge (140a) used to attach the base (102) to the side wall (104) can be a butt hinge. The reason a butt hinge configuration is desirable is that the base (102) can be removed from the side wall (104) by removing the pin (144) that holds the two plates (141, 142) of the butt hinge together. Thus, the base (102) and the side wall (104) can be separated from each other without having to disassemble the entire hinge (140a) from the base (102) and / or the side wall (104). In various embodiments, other types of hinges may also be considered, including, but are not limited to, continuous hinges, strap hinges, spring-loaded hinges, leaf hinges, and countersunk mount hinges.

[0060] Figures 1E-1 and 1E-2 show further embodiments of the camera navigation box using "living hinges" (155). Specifically, these illustrated embodiments provide a configuration of a camera navigation box that can be transported in an unfolded state (150b), and therefore in a flat state, as shown in Figure 1E-2. Figure 1E-1 shows an embodiment of the camera navigation box in constructed form (150a).

[0061] As shown in Figure 1E-2, the base (102) and side walls (104a-c) of the camera navigation box are attached to each other or otherwise formed from a single monolithic material piece. As described herein, the base (102) and side walls (104a-c) of the camera navigation box are attached to each other via a “living hinge” (155). In various embodiments, the portion of the camera navigation box between the base (102) and the side walls (104a-c) is modified to form an inclined groove (155). In various embodiments, the base (102) and side walls (104a-c) can be separate parts. These separate materials can then be used to connect the base (102) and side walls (104a-c) to each other so as to function as a “living hinge” (155). These separate materials can then be configured to have a shape similar to that of the inclined groove (155).

[0062] In various embodiments, inclined grooves (155) are positioned at corner junctions between the base (102) and / or side walls (104a-c). The inclined grooves (155) allow the base (102) and side walls (104a-c) to flex or bend. Specifically, the inclined grooves (155) allow for the positioning (150a) of the camera navigation box when in use, and then convert to a basically flat, unfolded state (150b) that allows for the transport or storage of the camera navigation box.

[0063] In various embodiments, hooks 152 and corresponding slots 154 can be provided between adjacent side walls (104a-c) to facilitate connection between them. The use of hooks 152 and slots 154 facilitates the maintenance of the shape of the camera navigation box, for example, as a box or rectangular configuration.

[0064] In various embodiments, the deflection and bending provided by the inclined groove (155) can be limited (for example, to 90 degrees). In various embodiments, the physical limitations of the separate materials used to form the living hinge (155) can also affect the flexibility of the "living hinge". In various embodiments, the inclined groove (155) can be added to further braces or structures to ensure that the deflection and bending are limited to 90 degrees or less. In various embodiments, the degree of deflection and bending of the "living hinge" (155) can be controlled by the side wall maintaining the upright configuration of the living hinge (155) using structures such as hooks 152 and slots 154.

[0065] In various embodiments, the “living hinge” (155) between the base (102) and the side walls (104a-c) may include a material that is sufficiently elastic to bend without shattering, such as polypropylene. In various embodiments, an insert or grid may be placed on the base (102) and / or side walls (104a-c), and further parts related to the insert or grid may be used as a “living hinge” to not only attach the base (12) to the side walls (104a-c) but also to provide the folding and unfolding capability of the camera navigation box. The reason why the insert or grid is suitable is that it can be formed from an elastic material (e.g., vinyl) that provides the necessary flexibility. The insert or grid may be designed so that the side walls (104a-c) are positioned perpendicular to the base (102), and markers on the insert or grid may be designed to align when the side walls are properly configured. In various embodiments, a "gap" or space is provided between the visible portion of the insert or grid that displays the markers and a portion of the insert or grid that functions as a "living hinge" (155) that is hidden when the base (102) and one or more side walls (104a-c) are folded into a desired configuration to form a camera navigation box.

[0066] Referring again to Figure 1E-1, the hook (152) and slot (154) provide the ability to attach adjacent side walls (104a-c). In various embodiments, other structures such as magnets, Velcro®, and complementary snaps (described later) can also be used. In any case, Figure 1F-1 shows the hook (152) and slot (154) in closer detail. Specifically, in various embodiments, the hook (152) is formed of two parts: a toothed first part (152a) and an expanding second part (152b). When engaged with the slot (154), the teeth of the toothed first part (152a) come into contact with the top of the slot (154). As a result, the slot (154) engages with one of the grooves of the toothed first part (152a), preventing the hook (152) from moving backward while engaged. The second expansion portion (152b) is biased outward, applying pressure to the tooth portion (152a) to engage with the slot (154). To insert the hook (152), both the tooth portion and the expansion portions (152a and 152b) of the hook (152) are pressed to insert the hook (152) into the slot (154). The expandability of the second portion (152b) prevents the hook (152) from coming out of the slot (154). The insertion of the hook (152) into the slot (154) can be seen in Figure 1F-2. To remove the hook (152) from the slot (154), the two portions (152a and 152b) of the hook (152) are pressed again before removing it from the slot (154).

[0067] The attachment mechanism for the side walls (104a-c) helps define the space for the training environment by maintaining the periphery corresponding to the side walls (104a-c). In various embodiments, the side walls (104a-c) provide an additional surface on which the training environment is positioned, providing a wider three-dimensional space that can be tracked. Furthermore, this attachment mechanism also helps maintain the configuration of the side walls (104a-c) perpendicular to the base (102). In various embodiments, connectors and / or associated attachment mechanisms for smoothing, holding, and / or locking the side walls (104) together can be manufactured separately from the base and / or side walls and then assembled. In various embodiments as described throughout, the attachment mechanism can be embedded in the base (102) and / or side walls (104), or manufactured at least together with the base (102) and / or side walls (104).

[0068] Figures 1G-1 and 1G-2 show further embodiments of the camera navigation box. Specifically, Figure 1G-2 shows an embodiment in which the camera navigation box is assembled using magnets (175) that are used to attach the side walls (104a-c) together and maintain the vertical configuration of the side walls (104a-c) relative to the base (102). In various embodiments, the magnets (175) may be embedded in the side walls (104a-c), fixed using adhesive, or a combination of both. In contrast, Figure 1G-1 shows an embodiment of the camera navigation box in a folded / collapsed state in which the side walls (104a-c) are folded relative to the base (102). In the folded / collapsed state, the camera navigation box is easily transported and stored when it is not being used with the rest of the camera navigation system.

[0069] Various embodiments show two different configurations of an example camera navigation box. In various embodiments, the first configuration provides a camera navigation box in a folded / flat configuration (170a), and the second configuration provides an unfolded (170b) camera navigation box. In various embodiments, two side walls (104a-104b) may have magnets, or a third side wall (104c) may have a magnetic material with opposite polarity to the magnet (175) associated with it. In various embodiments, the two side walls (104a-104b) may be formed of a material attracted to the magnet (175) of the third side wall (104c).

[0070] As shown in the figure, an insert or grid (300) is arranged inside the camera navigation box. In various embodiments, the insert or grid (300) is arranged on the base (102) of the camera navigation box. In various embodiments, the insert or grid (300) is arranged on one or more of the side walls (104a-c). In various embodiments, the insert or grid (300) can cover all or at least part of the base (102) and / or the side walls (104a-c). In any case, the area defined by the insert or grid (300) corresponds to the training environment and further corresponds to the space represented via the digital environment displayed on the monitor.

[0071] As shown in Figures 1A to 1G, the camera navigation box or a part thereof is foldable or collapsible. In various embodiments, the camera navigation box is configured such that multiple markers on an insert or grid (300) are properly aligned across each inner surface of the camera navigation box (e.g., the base and / or one or more side walls). Specifically, the multiple markers on the insert or grid (300) are arranged clearly and without interruption. In various embodiments, the multiple markers follow a consistent pattern on different inner surfaces of the camera navigation box. Furthermore, the transition from one surface (e.g., the base (102)) to a different surface (e.g., side walls (104(a~c))) is configured not to disrupt the pattern provided by the multiple markers on the insert or grid (108). For example, Figure 1G-2 shows a specific arrangement example of markers on an insert or grid (300) on multiple surfaces, where the camera navigation box in an unfolded state (170b) has multiple markers that are present on the base (102) and seamlessly transition to other surfaces such as other side walls (104a~c).

[0072] The alignment of markers associated with the insert or grid (300) ensures that the camera navigation system can accurately process a group of adjacent markers to identify the position of the simulated laparoscope relative to the insert or grid (300). In various embodiments, the simulated laparoscope needs to acquire at least four markers. These four markers allow the camera navigation system to calculate / identify various types of information related to the position of the simulated laparoscope relative to the training environment. If fewer than four markers are used, some of the information related to the position of the simulated laparoscope relative to the training environment can be identified, but it may not be possible to correctly identify all or any of the relevant information. If more than four markers are used, the camera navigation system needs to perform more processing, which can be time-consuming and result in slower updates of the digital environment. In various embodiments, the required number of markers can be based on the size of the markers, the type of simulated laparoscope (e.g., its field of view), and / or the types of markers used. The combination of markers can uniquely correspond to a specific position within the training environment (i.e., the insert or grid). In various embodiments, to determine where / in which direction the simulated laparoscope is oriented within the training environment, the camera navigation system uses computer vision to identify and determine which individual markers are present in the image data captured by the simulated laparoscope. Once the marker locations are known, the camera navigation system can estimate the position of the simulated laparoscope relative to the training environment (via a PnP process in various embodiments), and in various embodiments, this position is characterized by the six degrees of freedom of the simulated laparoscope (e.g., x, y, z, and roll, pitch, yaw). Using the acquired position information, the camera navigation system can determine how the simulated laparoscope is currently held relative to the training environment.

[0073] If some markers are misaligned (for example, if a marker on the base is misaligned with a marker on one side wall) or if the alignment is outside the acceptable margin of error, the camera navigation system may not be able to recognize the marker combination and determine the orientation of the simulated laparoscope within the training environment. This can result in an error condition that prevents the camera navigation system from determining the current position of the simulated laparoscope. Furthermore, this error condition may interfere with or cause errors in the digital environment generated for the user to see on the monitor.

[0074] In various embodiments, similar errors can occur when the simulated laparoscope is facing an area where no markers are placed (e.g., a side wall without markers), as the camera navigation system may be unable to identify the current position of the simulated laparoscope. However, the content that can be displayed on the monitor can differ. For example, the digital environment may provide notification that the simulated laparoscope is outside the training environment and instruct the user to reposition the simulated laparoscope toward the training environment. In another embodiment, the monitor may instead provide a real-time image of what information is currently being captured by the camera sensors of the simulated laparoscope (e.g., an empty space or empty side wall of the camera navigation box without markers). In this way, the camera navigation system provides clear instructions to the user that they need to operate the simulated laparoscope toward the training environment.

[0075] In various embodiments, errors may include the inability to maintain a stable digital environment (including computer-generated elements embedded within the environment). Such scenarios hinder user interaction with the camera navigation system. For example, the view of the digital environment displayed on the monitor may fluctuate back and forth between frames because the camera navigation system attempts to update the digital environment based on a miscalculation that the simulated laparoscope is simultaneously in two different positions due to misplaced markers. Alternatively, the view of the digital environment may not change (or may freeze) despite the movement of the simulated laparoscope because the camera navigation system has not received an update. As described above, notifications can be provided instructing the user to move the simulated laparoscope in a specific direction (i.e., point the simulated laparoscope towards one of the markers in the training environment) to enter a predetermined area that can be tracked again.

[0076] According to various embodiments, the camera navigation system can be operated locally, remotely, or partially locally and partially remotely. In various embodiments, the remote application is implemented with at least a portion of the camera navigation system, such as the scope view generator portion, being implemented on a cloud-based server or remote server, while the remote portion is implemented at least physically away from the training environment and the user. In various embodiments, data related to camera navigation training and / or multiple markers can also be stored locally, remotely, or partially locally and partially remotely.

[0077] According to various embodiments, the camera navigation system provides different surgical training for practicing surgical skills corresponding to the use of laparoscopes or endoscopes during surgical procedures in relation to a training environment (e.g., insert or grid). In various embodiments, the training environment for the camera navigation system is adapted to different laparoscopes (e.g., third-party made or having different features such as zero-degree or angled (e.g., 30 degrees)).

[0078] According to various embodiments, camera navigation training examples provided by a camera navigation system used with a zero-degree laparoscope include follow exercise, track exercise, and / or framing exercise. In follow exercise, the user must manipulate a cursor in the digital environment to follow a path displayed on a monitor. In track exercise, the user must manipulate a cursor in the digital environment to follow a moving target displayed on a monitor. In framing exercise, the user must manipulate a cursor to overlap one or more targets in the training environment displayed on a monitor. The manipulation of the cursor in the digital environment is performed by using a simulated laparoscope to perform similar movements relative to the training environment.

[0079] In various embodiments, the same camera navigation training can be used for angled laparoscopes. However, in various embodiments, further training is provided for simulated laparoscopes with angled features. Examples of training provided for angled laparoscopes include tube targeting and / or star pursuit. Tube targeting training requires manipulating the cursor's "viewpoint" to a planar target and the center of a tube extending perpendicularly from this target on the viewing surface in the digital environment. Star pursuit training requires manipulating the cursor's "viewpoint" within the digital environment to track and follow the position of a star moving from one location to another within the digital environment. Again, the manipulation within the digital environment is performed by manipulating a simulated angled scope or a part thereof relative to the training environment. Further details of each of these camera navigation trainings are provided below.

[0080] One reason for the difference in camera navigation training types between zero-degree laparoscopes and angled laparoscopes lies in the physical characteristics of each laparoscope. In zero-degree laparoscopes, the camera / image sensor is aligned with the longitudinal axis of the simulated laparoscope. In contrast, in angled embodiments, the camera / image sensor is not aligned with the longitudinal axis of the simulated laparoscope, but rather presented at a fixed angle (e.g., 30 degrees). This introduces further complexity regarding how to rotate and manipulate the camera / image sensor relative to the training environment.

[0081] Figure 31 is an example diagram showing some of the angled laparoscopes (3100) according to various embodiments. Specifically, the angled laparoscope (3100) can be rotated via two different operating points, with a first point (3120) located at the proximal end of the angled laparoscope. Rotating the angled laparoscope (3100) using the first operating point (3120) produces the effect of rotating the image acquired by the angled laparoscope, similar to when the user physically rotates a zero-degree laparoscope. Therefore, rotating 180 degrees using this first point (3120) results in the acquired image being upside down.

[0082] With respect to the second operating point (3110), this point is located between the distal and proximal ends of the angled laparoscope. The second operating point (3110) is configured to physically rotate the camera / image sensor. The physical rotation of the camera / image sensor is used to change the direction in which the angled portion of the angled laparoscope, such as the distal end of the angled laparoscope, is facing. When the camera / image sensor is physically rotated (and thus the direction in which the angled portion is facing changes), the angled laparoscope gains the ability to view different areas of the training environment, even if the position of the distal end of the angled laparoscope does not change (other than its rotation). This movement allows the angled laparoscope to "look around" at objects. In some zero-degree laparoscopes, such changes in the field of view are impossible or limited and are not provided in some embodiments.

[0083] In angled laparoscopes, the complexity arises in terms of the types of rotation present, and a rotation sensor / encoder (3130) is provided that measures the amount of rotation using one or both of the operating points (3110, 3120). In various embodiments, such as the illustrated embodiment, the rotation sensor (3130) can be housed within the handle of the angled laparoscope.

[0084] In embodiments relating to zero-degree laparoscopes, the scope view generator can calculate 6 degrees of freedom based on image data of markers in the training environment. The same calculation can be performed for angled laparoscopes. In various embodiments of angled laparoscopes or scopes with rotation sensors, the camera navigation system is configured to take into account measurements made by the rotation sensor. In various embodiments, for example, the roll value of such a laparoscope is further modified by data acquired via the rotation sensor (3130).

[0085] In various embodiments, the camera navigation system is configured to detect that a simulated laparoscope has been connected to the camera navigation system and to identify which training sets should be loaded and provided to the user. In various embodiments, the camera navigation system can be configured so that an angled laparoscope has access to all available camera navigation training. However, a zero-degree laparoscope is prevented from accessing training designated as skills relevant to the use of a simulated angled laparoscope due to its physical limitations (i.e., inability to "look around" corners). Because zero-degree laparoscopes and angled laparoscopes may differ in how image data is acquired and used, in embodiments using a simulated angled laparoscope, the computer navigation system is configured to process and calibrate data acquired from image data so that the display and updating of the digital environment are processed and provided uniformly.

[0086] Figures 2A to 2C show examples of embodiments relating to a camera navigation system (200). Referring to Figure 2A, the camera navigation system (200) includes a camera navigation box (210), a simulated laparoscope (220) (e.g., a simulated 0° or angled laparoscope) having a camera navigation box (210), a plurality of markers on an insert or grid (215) (e.g., markers), a scope view generator (230), a monitor (240), and / or a corresponding camera (225). In various embodiments, the camera navigation system (200) is configured to identify the type of training to be presented to the user (250) via the monitor (240) and to identify how the captured image data from the simulated laparoscope (220) will be used based on the type of laparoscope to which it is connected (e.g., a zero-degree or angled laparoscope). In various embodiments, the camera navigation system (200) identifies the type of simulated laparoscope (220) to which it is connected, and then retrieves the camera navigation training set corresponding to the connected simulated laparoscope and displays it via a menu (for example, as shown in Figure 4A). In various embodiments, the user (250) can submit or provide the camera navigation system (200) (for example, through selection from the user interface) with the type of simulated laparoscope (220) in use, so that the appropriate training is retrieved, selected, and / or displayed. In various embodiments, this identification is performed automatically as the camera navigation system identifies the simulated laparoscope to which it is connected and retrieves the corresponding information (e.g., camera navigation training, calibration).

[0087] As described above, the camera navigation system (200) tracks the position of the simulated laparoscope through the use of markers on inserts or grids which can be placed on one or more of the flat inner surfaces within the camera navigation box (210). In various embodiments, the inserts or grids (215) may be removed and multiple markers may be placed directly (e.g., printed) on the inner surface of the camera navigation box (210). In various embodiments, the inserts or grids (215) may be placed on the inner surface of the camera navigation box (210) or otherwise attached to the inner surface of the camera navigation box (210) (e.g., via adhesive). In various embodiments, the inserts or grids (215) may be compatible with or replaceable with other inserts or grids which may have different marker arrangements used for different camera navigation training.

[0088] In various embodiments, the simulated laparoscope (220) includes a camera (225) used to capture image data of a subset of markers from a plurality of markers associated with an insert or grid (215) (also referred to herein as an image sensor or camera sensor). In various embodiments, a scope view generator (230) is provided, specifically configured to estimate the current position or scope view / viewpoint of the simulated laparoscope and to generate a representation of this current position or scope view in a digital environment. In various embodiments, the scope view generator (230) calculates position-related information of the simulated laparoscope from the captured image data, specifically by identifying combinations of markers and confirming the positions of the markers relative to the training environment. In various embodiments, the scope view generator is configured to perform a number of calculations to further extract position data about the simulated laparoscope (e.g., a PnP process). Specifically, the scope view generator can obtain information from the acquired image data that can be represented as six degrees of freedom of the simulated laparoscope. This information can "recreate," or at least determine, how the simulated laparoscope is held or positioned within the three-dimensional space defined by the training environment.

[0089] In various embodiments, the scope view generator is configured to generate and update a digital environment that simulates a three-dimensional space corresponding to the training environment. In various embodiments, the scope view generator is configured to generate menus and camera navigation training for identified simulated laparoscopes connected to and used with a camera navigation system. In various embodiments, the current position of the simulated laparoscope relative to the training environment has a corresponding representation in the digital environment. In various embodiments, this representation (e.g., a circle) is used as a designator or cursor in the digital environment. Thus, the movement of the simulated laparoscope relative to the training environment is also represented as cursor movement.

[0090] The digital environment and the computer-generated elements implemented therein (such as a cursor and targets or obstacles related to the displayed camera navigation training or menu) are provided on a monitor for the user to see. In this application, user interaction (via the use of a cursor) with computer-generated elements (e.g., buttons, targets) enables a simulation of camera navigation training that the user can practice outside the operating room by manipulating a simulated laparoscope in the training environment. According to various embodiments, the camera navigation system is configured to provide the user with an opportunity to perform training outside the operating room.

[0091] Figures 2A to 2C show various embodiments of the camera navigation system. Figure 2A shows an embodiment of the camera navigation system. Specifically, this figure shows a data flow for generating a digital environment and displaying it on a monitor (240) after processing image data acquired via a simulated laparoscope (220) during interaction with a training environment (215) via a scope view generator (230).

[0092] In various embodiments, image data captured by the simulated laparoscope (220) via a camera sensor (225) is transmitted to a scope view generator (230). The scope view generator (230) is configured to process this image data to identify the position of the simulated laparoscope (220) relative to the training environment (215) and / or to generate and update a digital environment (incorporating various computer-generated elements) displayed on a monitor (240). The digital environment is generated and / or updated to provide menu and camera navigation training. In various embodiments, computer-generated elements such as background images, text, cursors, buttons and / or meters incorporated into the digital environment can provide feedback on current performance. In various embodiments, all information or data regarding the computer-generated elements is stored in memory and can be retrieved by the scope view generator (230).

[0093] In various embodiments, as shown in Figure 2A, the camera navigation system (200) is configured at a physical location (e.g., a school, a hospital) and can operate locally at this physical location. In other embodiments, as shown in Figure 2C, parts of the camera navigation system can be configured and operated remotely (e.g., via the cloud, a remote network, and / or a remote system).

[0094] In various embodiments, a user means an individual who uses or interacts with the camera navigation system in connection with practice and / or training involving various camera navigation exercises. In various embodiments, the user operates a simulated laparoscope around the training environment to capture images of inserts or markers on a grid. The user can view the corresponding digital environment and computer-generated elements (such as a cursor corresponding to the position of the simulated laparoscope relative to the training environment) on a monitor. By performing the various camera navigation exercises provided by the camera navigation system, it is possible to improve proficiency in camera navigation, which is useful for surgical procedures with limited field of view and / or space, which are actually performed laparoscopically.

[0095] In various embodiments (for example, as shown in Figure 2A), the camera navigation box (210) of the camera navigation system (200) corresponds to a confined or partially confined space (for example, a limited surgical space such as the pelvis). In various embodiments, the camera navigation system (200) may also further simulate a confined space through the use of a surgical trainer that simulates the patient's torso (details of one embodiment are shown below in relation to Figure 30).

[0096] In various embodiments, inserts or grids are provided to facilitate tracking the position of the simulated laparoscope. The spatial representation defined by the inserts or grids is a digital environment displayed on a monitor. The monitor displays a representation of the laparoscope's position relative to the training environment as a cursor within the digital environment. The camera navigation system is configured to periodically (e.g., 60 times per second) acquire image data from the simulated laparoscope so that the cursor position in the digital environment can be continuously updated.

[0097] In various embodiments, the camera navigation system is configured to perform camera navigation training in real time. Whenever the display of the digital environment on the monitor is updated, the latest cursor position (corresponding to the current position of the simulated laparoscope relative to the training environment) is already calculated and displayed. If for any reason the camera navigation system is unable to complete the processing necessary to identify and provide the latest position information of the simulated laparoscope to be implemented in the digital environment, it can skip the specific frame in which the position of this laparoscope is updated, hold the current frame being displayed, and retrieve and process the next image data.

[0098] In various embodiments, the digital environment is configured to provide a view of a simulated laparoscope relative to the training environment. The purpose is to simulate a real-time scenario in which a surgeon views the surgical field on a monitor while a surgical laparoscope is being manipulated within a patient.

[0099] The digital environment displayed on the monitor is based on image data captured by the simulated laparoscope. This image data is processed by the camera navigation system, and the scope view generator is configured to generate and update the digital environment accordingly. Furthermore, any subsequent calculations related to quantifying user performance in relation to target tracking, path following, distance perception, and collisions during camera navigation are also performed by the camera navigation system and / or directly updated in the digital environment. In various embodiments, references to the physical settings related to the simulated laparoscope or the training environment are not made until at least the next update of the digital environment is required. For example, collision calculations relating to the user's viewpoint in relation to the position of the simulated laparoscope between the target and the tubes surrounding the target are performed based on information related to camera navigation training within the digital environment and computer-generated elements included in the digital environment. Camera navigation training has data on where the target is located, where the tubes are located, and the current cursor position and viewpoint within the digital environment. Calculations relating to the user's current viewpoint in relation to the position of the simulated laparoscope with respect to the target and tubes are performed within the digital environment and in stored data related to computer-generated elements (e.g., the target and tubes). This calculation can be represented, presented, and / or determined by generating a line between the cursor and the target and detecting whether any computer-generated element (e.g., a tube) has positional information that intersects that line.

[0100] In another embodiment, a camera navigation system may utilize star-tracking training (detailed in this application) in relation to an angled laparoscope. Once the position and viewpoint of the angled laparoscope are known in the digital environment, the position of the angled laparoscope can be compared with data stored in conjunction with the star-tracking training, which corresponds, for example, the current position of a star, how the star moves from one location to another, and / or the position / arrangement of obstacles in the digital environment. Within the camera navigation training, calculations are performed, for example, by comparing coordinates or other position comparison means in the digital environment between a cursor and a star to determine at least whether the simulated angled laparoscope is tracking the star, or whether the star is otherwise correctly visible to the simulated angled laparoscope.

[0101] In various embodiments, similar functionality exists for zero-degree laparoscopy and its associated camera navigation training. For each camera navigation training (e.g., follow, track, framing), the camera navigation system has information on how the camera navigation training is performed, such as the action steps, action states and / or action conditions, and how computer-generated elements (e.g., objects, trajectories, targets) are defined (e.g., shape), positioned (e.g., x, y, z coordinates), and / or behave (e.g., movable, stationary). Once the camera navigation system identifies the current position of the simulated laparoscope relative to the training environment, this information is transformed (via a PnP process to achieve the 6 degrees of freedom of the zero-degree laparoscope), and the digital environment (e.g., the placement of a cursor representing the position of the simulated laparoscope relative to the training environment) is updated accordingly. The current position of the simulated laparoscope can be characterized relative to the training environment as a combination of (x, y, z) coordinates corresponding to the coordinates in the digital environment where the cursor is located. The camera navigation system can use the cursor's coordinates to compare the cursor's current position with stored data about one or more computer-generated elements (e.g., trajectories, targets, objects) in the digital environment. Calculations can be performed between the cursor's coordinates and stored information about the digital environment and / or computer-generated elements (e.g., position-related information for each computer-generated element) to determine user performance (e.g., whether the user is following the trajectory, whether the user has framed the target correctly, and / or whether the user has collided with an object).

[0102] In various embodiments, the position of the simulated laparoscope can be updated and calculated at regular intervals (e.g., 60 times per second). Alternatively, in various embodiments, only the position information of the laparoscope is extracted and used to update the digital environment. In various embodiments, subsequent calculations generally related to user performance are performed using data extracted from the position information and stored information about the digital environment or its computer-generated elements related to selected camera navigation training.

[0103] Figure 3 shows an embodiment of the insert or grid (300). The insert or grid (300) includes an arrangement of special markers that can be placed on part or all of the floor of a surgical trainer and / or camera navigation box, or on other internal surfaces (e.g., side walls and / or ceiling). As shown in Figure 3, the markers (305) are displayed in a checkboard arrangement in which markers occupying bright squares are alternated with dark squares (310). The dark squares (310) are provided to create spacing between adjacent markers (305) to facilitate the identification of individual markers (305). In various embodiments, different arrangements from the checkboard described above may be used, depending on the shape and size of the markers used. For example, the insert or grid (300) may also be configured to remove all dark squares and instead place all markers adjacent to each other. One advantage of placing all markers adjacent to each other is that it reduces the area that needs to be image-captured to identify a minimum number of markers.

[0104] In various embodiments, the insert or grid (300) or markers may also be applied to a physical object (e.g., an object placed on the insert or grid (300)). Such embodiments may have the ability to distinguish markers (305) associated with the base and / or side walls of the surgical trainer and / or camera navigation box from markers associated with objects placed on the training environment, so that the camera navigation system can distinguish between the surface of the training environment and the surface of an object.

[0105] In various embodiments, the insert or grid (300) can be configured as a plane or sheet including a specific arrangement of markers (305). Depending on the desired size of the training environment, the insert or grid (300) can be enlarged to include part or all of the corresponding inner surface of the camera navigation box, such as the entire bottom surface of the camera navigation box. In various embodiments, the insert or grid (300) can be placed only on part of the camera navigation box and / or surgical trainer. In such embodiments, camera navigation training can be directed to move the simulated laparoscope within a more constrained area.

[0106] In various embodiments, a special arrangement of markers used in relation to an insert or grid in a training environment may include a plurality of binary square markers (e.g., QR (quick response) codes). In various embodiments, the markers (305) may instead be integrated (e.g., printed) on the inner surface of a camera navigation box and / or surgical trainer, and / or on an object housed inside a camera navigation box and / or surgical trainer. In various embodiments, the insert or grid (300) is removable from the camera navigation box and / or surgical trainer. In various embodiments, the markers (305) may be printed on one or more separate sheets, the sheets may be planar, and / or the sheets may have at least one flat or planar surface relative to the camera sensor of a simulated laparoscope. In various embodiments, the sheets facilitate the removableness of the insert or grid (300) from the camera navigation box. In various embodiments, the same insert or grid (300) may be used in various different camera navigation boxes and / or surgical trainers. In various embodiments, different inserts or grids (300) may be provided, created and / or used. In various embodiments, the inserts or grids (300) may be provided and used without a surgical trainer and / or camera navigation box. In various embodiments, the aim is to enable the simulation of different actual surgical procedures or the practice of different surgical skills by relying on a camera navigation system to track a simulated laparoscope or similarly equipped instruments, tools, or accessories configured to reference the inserts or grids and capture image data.

[0107] As described in various embodiments herein, the markers (305) used with the insert or grid (300) may be squares, but may also have any number of different shapes, such as triangles or circles. Whatever the shape of the markers (305), information about such markers can be stored in memory and used to calculate the positional information of the simulated laparoscope.

[0108] While various embodiments described in this application use QR codes as unique markers for identifying the position of a simulated laparoscope within a training environment (e.g., a camera navigation box), other symbols may be used as long as each symbol is unique. If different shapes and / or symbols are used, the camera navigation system must be specifically designed to accommodate the different configurations so that the data can be correctly processed to accurately determine the position of the simulated laparoscope within the training environment. Furthermore, such information regarding the markers is stored in memory and retrieved to determine the position of the simulated laparoscope.

[0109] As described above, each marker (305) used in relation to the training environment of the camera navigation system is unique from all other markers (305) related to the same training environment. As a result, the position of the simulated laparoscope within the training environment can be properly identified. In various embodiments, the positions of markers within the training environment can be classified via x, y, and z coordinate sets. Each marker has a unique set of coordinates that helps to pinpoint the marker's position in the 3D space defined by the training environment. The position of each marker (305) on the grid (300) is stored in memory so that the camera navigation system can later identify the position to which the simulated laparoscope is pointed when using computer vision to identify one or more markers (305). As a result, the camera navigation system can calculate the position information of the simulated laparoscope based on the image data of the markers.

[0110] With regard to the digital environment, the 3D space associated with the digital environment corresponds to (for example, is the same as) the 3D space defined by the training environment. Therefore, for each marker defined in the training environment, a similar point exists in the digital environment, if not identical, and both 3D spaces have the same reference point (0,0,0) coordinates. Thus, a marker associated with the training environment that has x, y, and z coordinates also has the same point in the digital environment that has x, y, and z coordinates. These two points correspond to each other between the training environment and the digital environment.

[0111] The following scenario is presented to illustrate the correlation between the training environment and the digital environment, and to define a three-dimensional space for these two environments.

[0112] In various embodiments where the marker (305) is positioned on the base of the camera navigation box, the coordinates of the marker (305) are stored in memory using a reference point (320) at the lower left corner of the insert or grid (300) (i.e., the point corresponding to the (x,y,z) coordinate (0,0,0)). Each marker has an x,y coordinate pair where z=0. The same applies to the digital environment, as the cursor position at any of these same spots also shares the same x,y coordinate pair where z=0.

[0113] In various embodiments where a marker (305) is placed on either of the side walls of the camera navigation box (as shown on 104a and 104b in Figure 1G-2), the markers have the same x-coordinate value, and their y and z coordinates are variable. In practice, the x-coordinate value of the marker on side wall 104a is 0. Similarly, all markers placed on the other side wall 104b have the same x-value (except that they are farther from the reference point (0,0,0) and therefore different from the x-value of the markers on side wall 104a), and their y and z coordinates are variable. In the corresponding digital environment, the points in the digital environment corresponding to these markers on side walls 104a and 104b share the same coordinates (i.e., x,y,z).

[0114] In various embodiments where markers (305) are positioned on the rear side wall of the camera navigation box (as shown in 104c of Figure 1G-2), all markers have the same y-coordinate value, but the x and z values ​​vary based on the placement of the markers on the insert or grid. In the corresponding digital environment, the corresponding points in the digital environment related to the markers on the rear side wall (104c) share the same variable x and z coordinates, while the y-value is constant.

[0115] In various embodiments in which markers (305) are positioned on the base of a camera navigation box (as shown above (102) in Figure 1G-2), all markers have the same z value, and their respective x and y values ​​are variable based on their position. In the corresponding digital environment, the corresponding points in the digital environment related to the markers on the base (102) share the same variable x and y coordinates, but their z values ​​are constant.

[0116] In other words, regarding the placement of markers related to the training environment, each marker has a unique identifier corresponding to a specific position relative to a reference point within the training environment. Furthermore, a corresponding position with the same specific location is also defined within the digital environment. Both the positions within the training environment and the positions within the digital environment are defined by the same x, y, and z coordinates.

[0117] The camera navigation box can be designed to notify the user of errors if the image captured by the simulated laparoscope is not recognized, or if the simulated laparoscope is pointed at an area where no markers exist. This notification can be provided in several different ways. Firstly, the user can be notified in the digital environment that the simulated laparoscope is untrackable and that the user should proceed to return to the training environment. Secondly, the camera navigation can display in real time on the monitor that what the simulated laparoscope is currently looking at is not the training environment. When this is seen, the user is informed that they need to reposition the simulated laparoscope back towards the training environment to resume tracking its position. Furthermore, hints and feedback can be provided to guide the user on where to move the simulated laparoscope.

[0118] In some embodiments, an error can be triggered to temporarily remove the user from camera navigation training if no identifiable marker is present (for example, if the user is attempting to point the laparoscope outside the training environment). In some embodiments, as a way of notifying that the simulated laparoscope is not pointed towards the training environment, the camera navigation system can be configured to display real-time images captured from the simulated laparoscope instead of the digital environment.

[0119] In various embodiments, when two or more images of the marker (305) are acquired by the simulated laparoscope, the camera navigation system can analyze and identify the acquired image data to determine a set of positional information regarding the current location of the simulated laparoscope. Specifically, the camera navigation system can determine where the simulated laparoscope is located in a three-dimensional space related to the training environment (characterized by a set of x, y, and z coordinates corresponding to the endpoints of the camera sensor), and / or the endpoints of how the simulated laparoscope is positioned in a three-dimensional space related to the training environment, characterized by roll, pitch, and yaw values. As used herein, the roll value corresponds to rotation (about the longitudinal axis of the simulated laparoscope), the pitch corresponds to an angle with respect to the insert or grid, and the yaw corresponds to rotation about a vertical axis (perpendicular to the longitudinal axis of the simulated laparoscope).

[0120] In various embodiments, the camera navigation system may calculate several values ​​based on a comparison between image data and data about markers stored in memory. For example, if the image of marker (305) appears distorted, the camera navigation system is configured to determine a specific angle or pitch of the simulated laparoscope by calculation transformations between the distorted version and a normal or predetermined version of the same marker. In various embodiments, the camera navigation system uses a stored transformation algorithm to process the distortion and convert such information into angle or pitch values ​​for the simulated laparoscope. Further details on how to calculate the positional information (i.e., 6 degrees of freedom) of the simulated laparoscope are described in more detail below.

[0121] In various embodiments, the simulated laparoscope needs to capture a predetermined minimum number of markers (e.g., 4) from an insert or grid to provide sufficient information so that the scope view generator can calculate and determine the position of the simulated laparoscope in a 3D space defined by the training environment. In one example, the simulated laparoscope needs to capture at least two markers in the same image. However, there may be embodiments where one marker is sufficient, or embodiments where three or four or more markers are required. The more markers captured by the simulated laparoscope in the same image, the more accurate the determination can be. For example, if seven markers are captured in the same image, the determination can be more accurate than if only two markers are captured in the same image.

[0122] However, there is an upper limit to the number of markers that can be captured and used to determine the position of the simulated laparoscope. Specifically, the more markers that are captured, the longer it takes to process all of them, and therefore the longer the camera navigation system can determine the position of the simulated laparoscope over that period. If the information regarding the position of the simulated laparoscope is not updated in time, the camera navigation system may need to stop the current processing and move on to the next updated set of information for the simulated laparoscope.

[0123] In various embodiments, the optimal size and number of markers mounted on the insert or grid may depend on the simulated laparoscope used. A balance must be struck by ensuring that the camera associated with the simulated laparoscope can identify the markers at both near and far distances. In one embodiment, the markers are approximately 1.63 cm wide. As a result, a particular simulated laparoscope can produce clear images visible through it within a range of 3 to 7 inches from the markers. This range may depend on the type of simulated laparoscope used, as well as other factors such as the size and number of markers associated with the insert or grid. Outside the desired range, the image captured by the simulated laparoscope may appear blurred on the monitor. In various embodiments, blurred image captures of the markers can still be used to identify the position of the simulated laparoscope. However, using such blurred image captures is undesirable because it can lead to errors or failure to identify the markers, reducing the accuracy of determining the position of the simulated laparoscope.

[0124] In various embodiments, the number of markers that a simulated laparoscope can capture in the same image may depend on the angle of the simulated laparoscope and the viewing distance from the insert or grid. In various embodiments, at least two adjacent markers are required to determine the position (i.e., location and / or orientation) of the simulated laparoscope. However, the more markers that are captured in addition to the first two adjacent markers, the more accurate the position determination of the simulated laparoscope becomes.

[0125] Another consideration related to the structure of the insert or grid is the total number of markers to be included. In one embodiment, the insert or grid may have 216 markers, but in other embodiments, there may be more or fewer. The number of markers depends, for example, the space of the training environment available for marker placement (e.g., the insert or grid), the size of the individual markers themselves, and / or how the markers are positioned. In various embodiments, even more markers may be used in connection with placement on the side walls and / or ceiling of the camera navigation box.

[0126] Increasing the number of visible markers improves the stability of tracking the position of the simulated laparoscope. However, it is thought that increasing the number of markers also increases the computations that the camera navigation system needs to perform in relation to identifying and distinguishing different markers. Increased computation can affect the overall responsiveness of the camera navigation system when providing the information to be used to update the position of the simulated laparoscope in the digital environment. Therefore, according to various embodiments, the number of markers used aims to balance the desired responsiveness and speed of marker identification with the stability that can be obtained by using more markers.

[0127] In various embodiments, markers are placed and / or integrated on a flat or relatively planar surface (e.g., an insert or grid) relative to the front of the simulated laparoscope, lens, and / or image sensor used to capture images of the markers. Placing one or more markers on a non-planar / non-horizontal or curved surface can cause distortion of the markers when viewed from the simulated laparoscope, resulting in errors in the recognition of the corresponding markers and / or the calculation of the simulated laparoscope's position, thus leading to errors in tracking the simulated laparoscope's position in the training environment. Various embodiments use a rigid board to maintain or help maintain the flat or planar shape of the insert or grid in order to ensure that the markers are placed on a flat or planar surface.

[0128] In various embodiments, markers can be placed on the camera navigation box and / or surgical base rather than on the insert or grid. Furthermore, markers can be placed directly on the side walls and / or ceiling of the camera navigation box and / or surgical trainer. Adding markers to the side walls and / or ceiling increases the opportunities to always orient the simulated laparoscope towards a traceable surface when used in conjunction with the camera navigation system. This is particularly useful when using other surgical devices and / or when the simulated laparoscope is an angled laparoscope. In various embodiments, the use of an angled laparoscope or articulated laparoscope, for example, is designed to allow observation of areas of the training environment that cannot be observed or have poor observation capabilities with a zero-degree laparoscope, for example. Furthermore, having markers on the ceiling and / or walls of the training environment can also facilitate other entry points into the training environment (e.g., insert or grid), such as having multiple openings into the camera navigation box and / or surgical trainer, not just from above or the ceiling. In various embodiments, entry points can be placed on the opposite side of the surface or location where the markers are placed. In various embodiments, markers can be provided on the posterior or distal wall of the camera navigation box and / or surgical trainer to enable tracking when using the camera navigation system in a simulated vaginal approach or procedure. In various embodiments, a simulated vaginal approach or procedure can be performed by inserting the simulated laparoscope into the camera navigation box and / or surgical trainer from the anterior or proximal wall, rather than from the top or ceiling.

[0129] In various embodiments, the markers used in the training environment are implementations specifically designed from the use of OpenCV, an open-source computer vision library. In one embodiment, the markers are implemented using QR codes arranged in a checkerboard pattern, for example, as shown in Figure 3. As mentioned above, the markers can be any symbols, as long as each symbol is unique from other symbols placed on the insert or grid and is recognizable by the camera navigation system. Furthermore, the markers can be arranged in configurations other than the checkerboard arrangement shown in Figure 3. In various embodiments, the camera navigation device stores the location of each unique marker related to the training environment in memory. As a result, it is possible to refer to the stored information regarding the correlation between the image of the markers acquired by the simulated laparoscope and their location relative to the training environment.

[0130] In various embodiments, the simulated laparoscope includes or is attachable to a camera / image sensor. The camera / image sensor is used to acquire image data. The image data may include image captures of markers from the training environment. The simulated laparoscope used in conjunction with the camera navigation system may be either a zero-degree laparoscope or an angled laparoscope (e.g., 30 degrees). Other embodiments may include other / further surgical instruments or devices equipped with or attachable to a camera or image sensor. Such embodiments allow for visual tracking (as seen on a monitor) of the positioning of the surgical instrument relative to the training environment. In various embodiments, frame rate, field of view, and / or image clarity may affect the camera navigation system's ability to observe and recognize one or more markers to track the position of the simulated laparoscope. For example, in some embodiments, the frame rate of the camera sensor may be 30fps (frames / second) to 60fps, but frame rates below 30fps may result in choppy images, and 60fps or higher is supported for smoother motion graphics.

[0131] The field of view of the camera or image sensor used to capture image data affects the size of various computer-generated elements displayed in the digital environment. Since the image data of the marker being captured already occupies a large portion of the existing visual area, a narrow field of view reduces the ability to capture extreme zooming in and out. Therefore, the objects that can be resized depending on the amount of zooming in and out may be limited.

[0132] Furthermore, the accuracy of identifying the position of the simulated laparoscope may be affected by the image clarity provided or determined by the camera or image sensor, which impacts the ability to confirm and recognize markers. Factors that may affect image clarity include camera resolution, depth of view, and shutter speed settings. In various embodiments, resolutions ranging from 640×480 to 1280×720 are used. On the other hand, resolutions lower than 640×480 may result in increased monitor jitter and shaking. Moreover, resolutions higher than 1280×720 require more computational power for analysis while offering little return in terms of tracking stability.

[0133] In various embodiments, the camera's field of view depth setting can be optimized to suit the training environment being used. If one or more portions of the insert or grid are outside the field of view depth of the simulated laparoscope, these portions of the insert or grid will be blurred, making it difficult to use them for tracking the position of the laparoscope. On the other hand, portions of the insert or grid that are within the depth range of the simulated laparoscope will have sharper, higher-contrast edges, thus enabling clearer tracking of the simulated laparoscope's position.

[0134] In various embodiments, the shutter speed of the simulated laparoscope camera affects the degree to which motion blur is captured in the image data. Increasing the shutter speed reduces the change in the position of the simulated laparoscope between subsequent captured images, thus reducing blur. The presence of distortion in the image data degrades the quality of position tracking of the simulated laparoscope.

[0135] Image data of the target acquired by the simulated laparoscope is transmitted to the scope view generator. In various embodiments, the image data can be transmitted via a wired connection (e.g., USB). In various embodiments, the information can also be transmitted wirelessly (e.g., Bluetooth®). The scope view generator uses image data from the simulated laparoscope, including captured images of markers acquired with respect to the training environment, to generate or update the digital environment and / or generate or update computer-generated elements corresponding to the position of the laparoscope. In various embodiments, the markers enable the scope view generator to determine the position information (i.e., 6 degrees of freedom) of the simulated laparoscope relative to the training environment. The camera navigation system tracks the position of the simulated laparoscope using markers in the training environment, thereby enabling the user to interact with computer-generated elements for purposes such as selecting different training and participating in different camera navigation training by mapping cursor movements to the movements of the simulated laparoscope. This interaction is achieved by reflecting the movements that the simulated laparoscope performs relative to the training environment in the cursor within the digital environment. By manipulating the cursor to overlap one or more computer-generated elements within the digital environment, the camera navigation system can identify user interaction with those computer-generated elements.

[0136] In various embodiments, the cursor is an example of a computer-generated element used in conjunction with a digital environment displayed on a monitor. The cursor position displayed on the monitor within the digital environment corresponds to the position of the simulated laparoscope relative to the training environment. Specifically, the cursor is the point where a virtual ray, extending from the distal end of the simulated laparoscope and parallel to its longitudinal axis, intersects the training environment. In other words, the point where the cursor is located relative to the simulated laparoscope is similar to the scenario where the simulated laparoscope is replaced with a laser point, and the endpoint to which the laser pointer is directed (in this case, the training environment) is the same type of point on the simulated laparoscope.

[0137] For illustrative purposes, the following observations regarding the position of the simulated laparoscope and the cursor position within the digital environment are presented. The following three scenarios illustrate examples of the relationship between the position of the simulated laparoscope relative to the training environment and the cursor position within the digital environment.

[0138] First, if the simulated laparoscope is perfectly perpendicular to the training environment-related markers placed on the base of the camera navigation box (such as (102) in Figure 1G-2), the x and y coordinates of the simulated laparoscope will be the same as the x and y coordinates of the cursor in the digital environment.

[0139] Furthermore, using the same situation as described above, when the simulated laparoscope is in contact with the training environment, the z-coordinate value of the simulated laparoscope perpendicular to the marking placed on the base of the camera navigation box (such as (102) in Figure 1G-2) will be the same as the z-value of the cursor in the digital environment.

[0140] Finally, in scenarios where the simulated laparoscope is perpendicular to the portion of the training environment located on the posterior side wall of the camera navigation box (such as 104c in Figure 1G-2), the z-value of the simulated laparoscope will be equal to the z-value of the cursor only when the simulated laparoscope is vertically aligned.

[0141] In various embodiments, a dummy laparoscope can be inserted through the top or upper part of the camera navigation box or the upper cover of the surgical trainer to simulate the insertion of a laparoscope through an incision in the patient's abdominal wall. However, as described above, other types of surgical procedures can also be simulated by inserting the dummy laparoscope through a different opening (e.g., the front).

[0142] In various embodiments, the scope view generator includes a processor and / or computer device that can be connected to a simulated laparoscope. In various embodiments, the scope view generator is configured to have a dedicated local application and / or a web browser-based application that processes information acquired from the simulated laparoscope related to markings on an insert or grid, and / or outputs information regarding the position of the simulated laparoscope relative to the training environment. In various embodiments, this process involves the scope view generator using a PnP process to calculate information regarding the six degrees of freedom of the simulated laparoscope (e.g., x, y, z coordinates and roll, pitch, yaw). The exemplary computer device may include a laptop or a desktop. Furthermore, the surgical trainer may include or be communicatively connected to a processor and / or computer device. The scope view generator may be communicatively connected to a monitor and a simulated laparoscope. Further details regarding embodiments of the scope view generator are shown in Figure 2B. Such embodiments can accommodate local implementations of camera navigation systems. Other embodiments are also conceivable in which the processing of the simulated laparoscope position relative to the training environment and / or the generation of supplemental graphic elements can be performed remotely via a cloud-based server (e.g., via the Internet) and / or via a remote server. Such embodiments are also described (see Figure 2C).

[0143] Figure 2B shows an example of the data flow of the camera navigation system shown in Figure 2A. The steps performed by the scope view generator (230) can be performed locally (e.g., via a local processor, desktop, laptop, etc.), remotely (e.g., via the cloud, or via a remote processor or computer device (i.e., related to a web browser-based implementation or application)), or a combination thereof. In various embodiments, the scope view generator (230) of the camera navigation system may have various applications and / or access in-memory libraries or stored data that provide corresponding data for digital environments and computer-generated elements used in various camera navigation training, facilitating the generation and processing of relevant data from a simulated laparoscope.

[0144] For example, a computer vision library (234) is provided, as shown in Figure 2B. In various embodiments, the computer vision library (234) is a set of programming functions that modify or analyze images used by the scope view generator (230) to determine the position data of the simulated laparoscope (220). In various embodiments, the images captured by the simulated laparoscope (220) correspond to images of markers on an insert or grid. The computer vision library (234) is specifically designed to track and identify the position of the simulated laparoscope in the three-dimensional space of the training environment using the markers on the insert or grid.

[0145] The application logic (236) corresponds to workflow logic that processes application data and logic for various simulated surgical training exercises that can be performed using the training environment. For example, using positional information of a simulated laparoscope (e.g., information on the 6 degrees of freedom), the application logic (236) compares the cursor position in the digital environment with the position of computer-generated elements in the digital environment to implement and execute any of the functions necessary to perform virtual button presses, menu transitions, and various camera navigation training exercises (described in further detail later).

[0146] As shown in the figure, the scope view generator (230) of the camera navigation system can also access a graphics library (238). The graphics library (238) is specifically designed to assist in rendering the digital environment and computer-generated elements output to the monitor for camera navigation training. In various embodiments, the graphics library (238) may rely on associated application logic (236) used to render the digital environment displayed for various camera navigation training purposes. Exemplary embodiments may use various libraries, such as OpenGL (Open Graphics Library) for desktop applications and WebGL for web browser-based applications.

[0147] Referring again to Figure 2A, the camera navigation system (200) includes a training environment (210) in which the tracking of a simulated laparoscope is performed. In various embodiments, the training environment (210) is implemented via an insert or grid. In various embodiments, the insert or grid can be sealed / housed within the camera navigation box and / or surgical trainer. In various embodiments, the camera navigation system may also function without using an insert or grid, with markers associated with the inner surface of the camera navigation box and / or surgical trainer, etc.

[0148] In various embodiments, the camera navigation box provides a controlled lighting environment used with the insert or grid. In various embodiments, the controlled lighting environment is provided by one or more lights, such as LEDs, that are connectable to the camera navigation box and / or surgical trainer and configured to illuminate the entire insert or grid. In various embodiments, one or more lights are controlled and / or actuated by a scope view generator (230). In various embodiments, the scope view generator (230) can adjust the brightness, direction and / or position of the lights. If the lighting environment changes unpredictably, changes in light conditions may cause the camera settings associated with the simulated laparoscope to change dynamically, potentially altering image quality and making tracking with the insert or grid difficult. In various embodiments, the camera navigation box and / or surgical trainer may also provide a natural or fixed pivot point when the simulated laparoscope is inserted apex or lateral. Camera navigation systems, such as full virtual reality (VR) solutions that do not use enclosed structures such as a simulated laparoscope or surgical trainer, may include a mechanically simulated pivot point.

[0149] In various embodiments, the camera navigation system may also include a lightboard that operates in conjunction with the insert or grid. In such embodiments, the insert or grid may be printed on a clear or transparent sheet, and this sheet may be illuminated from behind using a light source such as the lightboard described above. The light source works in conjunction with the camera of the simulated laparoscope, which may have limited or no control over automatic exposure, causing motion blur that can affect the ability to track the position of the simulated laparoscope using the insert or grid. Using a lightboard can increase the shutter speed by flooding the camera and / or sensors associated with the simulated laparoscope with light. By increasing the shutter speed, the amount of motion blur captured by the simulated laparoscope can be reduced.

[0150] In various embodiments, control of the exposure settings of the simulated laparoscope camera and / or control of the light source of the training environment are provided. However, in various embodiments, these actions may not be possible during actual surgical procedures and can only be used strictly for the simulation of surgical procedures.

[0151] In various embodiments, the camera navigation system may also utilize other types of markers besides the QR codes described above in relation to the insert or grid. For example, a unique non-QR code can be used. However, the symbols used as markers must be unique in such a way that the camera navigation system can uniquely identify the location associated with the symbol or combination of symbols captured by the simulated laparoscope, and must be distinguishable from all other symbols associated with the training environment. In various embodiments, the symbols (i.e., non-QR codes) used as markings on the insert or grid are white and black with no grayscale in between. Furthermore, as a different special library (or further modification of an existing library), there are non-QR code symbols specifically designed to identify any of the symbols used in other embodiments and any associated information such as a specific location relative to the training environment (e.g., x, y, z coordinates). Furthermore, this new or modern library is used in relation to a unique non-QR code image acquired by the simulated laparoscope to help determine the position of the simulated laparoscope. Exemplary symbols that can be used in these non-QR code embodiments may include various emojis, alphabets, or photographs of different objects.

[0152] In various embodiments, an insert or grid can be implemented using a computer device having a monitor. Specifically, the computer device (such as a computer tablet) has a monitor with a bright background. In various embodiments, markers can be generated by using a specially programmed application and displayed on the computer device's monitor. As a result, the simulated laparoscope can interact with the markings generated on the computer device's monitor, which functions as the insert or grid described above. In various embodiments, another similar embodiment can replace the computer device with a flat monitor connected to and / or incorporating a processor in a computer device such as a desktop or laptop. The connected computer device and / or processor is configured to generate and display markers designed to provide suitable illumination for the simulated laparoscope on the flat monitor. In various embodiments, a dedicated display (e.g., a display with or capable of memory installation) is configured to access and display the markers for the simulated laparoscope.

[0153] In various embodiments, the insert or grid can be completely replaced by the use of a body form or enclosure (e.g., a surgical trainer) having holes or narrow spots of different shapes that are illuminated from the back. The holes or spots are designed to mimic the use of tracking markers, and therefore each hole must have a unique shape that is easily distinguishable from one another. When the simulated laparoscope interacts with a particular hole or narrow spot, these holes or spots provide a response (reflecting light to the simulated laparoscope). When the simulated laparoscope is not interacting with a hole or narrow spot, the space within the body form or enclosure remains black. In such embodiments, the camera navigation system includes different applications and computer vision logic to correctly identify what images are being captured within the body form or enclosure and translate them to the corresponding positions on the simulated laparoscope.

[0154] In various embodiments, a camera navigation system determines the position of the simulated laparoscope relative to the training environment using images of markers captured by the simulated laparoscope. In various embodiments, the camera navigation system is configured to receive and analyze captured image data for generating and updating the digital environment and associated computer-generated elements. The digital environment and computer-generated elements are displayed on a monitor. As described herein, tracking of the simulated laparoscope is performed using inserts or grids. A scope view generator determines the position of the simulated laparoscope using information from the simulated laparoscope along with a computer vision library. Once the scope view generator has determined the position, this information is passed to the application logic, which uses this position information to evaluate whether to perform any application-related actions, such as pressing a button, navigating a menu, performing camera navigation training, and any other functions necessary for performing and displaying menus and / or camera navigation training. In various embodiments, the computer-generated elements displayed on the monitor are rendered using an open graphics library (e.g., OpenGL). Computer-generated elements can be rendered remotely via local and / or remote processors, via local and / or remote computer devices (e.g., desktops / laptops), and / or via web-based graphics libraries (e.g., WebGL).

[0155] In various embodiments, a computer vision library (234) provides computer vision algorithms for a camera navigation system. The camera navigation system uses the computer vision algorithms to determine the position of the simulated laparoscope relative to an insert or grid by identifying which (single or multiple) markers are currently captured in the image data of the simulated laparoscope's image sensor / camera.

[0156] For example, the embodiments shown in Figures 2A and 2B generally relate to the local implementation of a camera navigation system. For instance, a camera navigation system can be implemented in a school to allow students to conduct various camera navigation training exercises in the classroom.

[0157] On the other hand, parts of the camera navigation system can also be implemented remotely (for example, via the internet). As shown in Figure 2C, the scope view generator can be run at a location away from where the training environment is physically located. In various embodiments, parts of the camera navigation system can be run locally and other parts remotely. In further embodiments, one or more steps for determining the position of the simulated laparoscope can be performed both locally and remotely. As described herein, remote execution can be performed on a remote processor, computer device and / or server at another physical location independent of the training environment, as well as via a cloud-based server (for example, on the internet).

[0158] Figure 19 shows an exemplary embodiment of the camera navigation system. Specifically, the figure shows an example flowchart (1900) detailing different operations that the scope view generator uses to identify the current position of the simulated laparoscope relative to the training environment. Specifically, the scope view generator identifies what markers are present in the image data acquired by the simulated laparoscope. These steps or operations include converting the acquired images of inserts or grids obtained from the simulated laparoscope into a format that can be filtered and analyzed to determine the current position of the simulated laparoscope.

[0159] Referring to Figure 19, when the scope view generator receives acquired image data from the simulated laparoscope, it converts the acquired image (which can often be in color) from a color (i.e., RGB) image to grayscale (1910). Figure 20 shows an exemplary RGB to grayscale conversion. This figure shows the arrangement of a checkerboard pattern of multiple markers (2000) with black space (2010) used to space the multiple markers (2000) apart from each other. Generally, the multiple markers (2000) are represented in lighter tones (e.g., white or gray), and the other space is black (2010). In various embodiments, if the markers (2000) have non-white / black tones, the camera navigation system can characterize different colors with a certain threshold so that the marker colors are converted to different tones of white, black, and gray. If the markers associated with the insert or grid are already grayscale (i.e., black and white), there is no need to use color information in subsequent steps.

[0160] When the images acquired from the simulated laparoscope are converted to grayscale, the scope view generator converts the acquired images to a binary image (1920). Figure 21 shows an exemplary binary image. As shown, the binary image has multiple markers (2100) and black space (2110). Specifically, Figure 21 shows the final result of grayscale-to-binary conversion using adaptive thresholding. For each pixel in the binary image, it is determined whether the pixel is completely on or completely off, with completely on corresponding to white areas and completely off corresponding to black areas. Adaptive thresholding is a computer vision algorithm that easily determines which pixels are "on" or "off" based on a grayscale image. It compares adjacent pixels and uses a difference larger than a predetermined threshold to distinguish between pixels that should be "on" and pixels that should be "off".

[0161] In various embodiments, a computer vision algorithm in the camera navigation system samples pixels in a grayscale image and turns on or off the corresponding pixels based on an evaluation of whether the value is higher or lower than the average value of adjacent pixels by comparing the values. By using adaptive thresholding, the binary image can be enhanced to highlight the bright and dark areas of the previous grayscale image. The camera navigation system can use this information along with the binary image to identify the captured images of one or more markers that have been captured therein.

[0162] Using this binary image, the computer vision algorithm of the camera navigation system is used to analyze the binary image and find contours associated with bright areas (1930). As shown in Figure 21, bright areas can be considered to correspond to groups of pixels that are "on". A predetermined number of adjacent pixels must be determined to be "on". If the number of pixels falls below a predetermined threshold, this contour corresponds to a contour that is too small to be used and can therefore be ignored.

[0163] Each contour corresponds to a list of pixels used to define the boundary of a bright area. Based on the position of the contour, the shape of each area within it can be calculated. Figure 22 shows an exemplary contour calculation. Multiple markers (2200) with contours are shown in the same arrangement as the multiple markers before the transformation. The black space (2210) between the multiple markers (2200) is empty.

[0164] The computer vision algorithm of the camera navigation system filters the image to include only quadrilaterals using information related to the contours of bright areas (1940). Figure 23 shows an exemplary filtering operation for quadrilaterals. Specifically, only convex contours or shapes with four sides are considered potential markers of interest. As shown in Figure 23, after filtering is complete, each quadrilateral (2310) with its corresponding contour (2300) is shown in the image. After filtering (2320), inserts or grid portions that are not recognized as having four sides, such as when multiple markers or black spaces are removed, are not represented. Comparing the image in Figure 22 with the filtered image in Figure 23, contours located within quadrilaterals are ignored.

[0165] In the next step or operation, the corners (2400) and associated edges (2410) of each quadrilateral can also be plotted in a graph. Figures 24A to 24C show exemplary filtering using quadrilateral corners. The graph used for determination should include only the corners (2400) that are most likely to be one of the four corners of the marker. As shown in Figure 24B, all corners associated with a marker have two corners connected by an edge (at 2420). However, not all corners are connected to other corners, as shown in Figure 24A, for example. Referring to Figure 24A, the corner (2400) is not connected, and such an arrangement corresponds to a shape that is not part of the quadrilateral in the checkerboard arrangement and therefore should not be considered.

[0166] The connected edges (2420) are identified as corresponding to different markers arranged in a checkerboard pattern. These adjacent corners are merged so that four edges (2410) form a single point that forms an "x" pattern (2420). An example of corner merging is shown in Figure 24B. As shown in Figure 24C, all corners found inside the quadrilateral are incorrect corners and do not correspond to the quadrilateral, so the camera navigation system identifies and removes these corners (2430).

[0167] The final result after filtering out the incorrect corners can be seen, for example, in Figure 24C. Specifically, the final result should have identified rectangles (2440). Rectangles (2440) that are mainly light in color (i.e., on) correspond to multiple markers, while rectangles (2450) that are mainly dark in color correspond to black space.

[0168] Figure 25 shows an exemplary transformation matrix for determining distortion. Specifically, the figure shows a transformation matrix (2500) performed by the scope view generator to correctly identify quadrilaterals associated with inserts or markers on a grid where distortion may exist. In various embodiments, the position of the simuloscope is such that the captured view (2510) of one or more markers (shown here as quadrilaterals) appears distorted when, for example, the simuloscope is held at an angle to the training environment. Since a predetermined shape (e.g., a square) (2520) of the markers captured by the simuloscope is known and stored in memory, the camera navigation system can correct the distortion based on predetermined information about whether each of the four uniquely labeled corners (2530) (corresponding to Id=1 to Id=4) of the distorted shape should be positioned as part of the predetermined shape (e.g., a square). The scope view generator solves a system of linear equations using the four unique points of the corners as input. The transformation equation generates a transformation matrix representing the translation, rotation, scaling, and distortion necessary to transform the distorted points of the distorted quadrilateral (2510) into a predetermined shape (e.g., a square) (2520) that the marker should have. This transformation removes the distortion from the viewpoint of the simulated laparoscope, thus obtaining an image of the marker as if the simulated laparoscope were viewing it from, for example, directly above the marker or aligned perpendicularly to the insert or grid.

[0169] In various embodiments, other shapes are possible, but their corresponding transformation matrices must be implemented. For example, shapes with more or fewer than four sides (e.g., circles) can also be used, but different mathematical methods will be employed.

[0170] To verify that a particular contour (e.g., a quadrilateral) contains a marker (1950), pixels within that contour are sampled in a grid pattern. Bright pixels represent "1" and dark pixels represent "0". Combining all the "1"s and "0"s, these combinations form unique identification information for a particular marker. If this identification matches a known marker (information about known markers is stored in memory), the contour is confirmed to be a valid detection. "Known" markers are based on all markers stored in memory that are used to correlate with specific locations in the training environment. In various embodiments, a specific location is characterized as a set of x, y, and z coordinates that identify a marker in a 3D area related to the training environment.

[0171] The next step or operation (1960) performed after confirming the valid detection of the markers is to add corners (2600). Figure 26 shows an exemplary operation of adding corner points, which specifically shows the corresponding two-dimensional point set (1960). Corners (2600) are located at all intersections of edges formed by multiple markers (2610) and multiple black spaces (2620). These two-dimensional points are used in another step or operation (1970) to determine the positional information of the simulated laparoscope relative to the training environment. Specifically, each marker has a corner point with a unique label, as shown, for example, in Figure 27. The orientation (e.g., roll) can be calculated by comparing the corners (2600) and their unique labels (2700) with information about each marker (including information about each unique corner) stored in memory.

[0172] Figure 27 shows an exemplary step or operation for labeling each corner. As shown, each corner identified in Figure 26 is labeled (2700). Each marker (2710) and each black space (2720) can be defined by four sets of uniquely labeled corners (2700). This is compared with the information for each corner of each marker stored in memory. As described above, the offset of each corner is used to determine the orientation between the marker captured via the image sensor and the assumed marker orientation (i.e., how much rotation is detected around the longitudinal axis of the simulated laparoscope). The offset corresponds to the “roll” value, which is one of the six degrees of freedom of the simulated laparoscope.

[0173] Once one or more markers (2810) are identified, the scope view generator then performs calculations to determine the position of the simulated laparoscope relative to the training environment (e.g., insert or grid) and provides this information to the digital environment (1980). As described above, an example of the two-dimensional point processing step is shown in Figure 26. The process of how the position of the simulated laparoscope in three-dimensional space can be obtained from the two-dimensional points of markers related to the training environment is a problem that can be solved using a process known as "perspective-n-point" or (PnP). In various embodiments, the PnP problem is solved through an iterative approach based on the Levenberg-Marquardt algorithm.

[0174] Specifically, the PnP process is particularly useful in calculating the six degrees of freedom for the positional information of the simulated laparoscope, which cover the x, y, and z coordinates of the simulated laparoscope, as well as the roll, pitch, and yaw in three-dimensional space related to the training environment. Marker identification is used to determine the position of the simulated laparoscope in three-dimensional space (defined by the set of x, y, and z coordinates) as defined by the insert or grid, and each corner of the identified marker is used (via the PnP process) to extrapolate the roll, pitch, and yaw values ​​of the simulated laparoscope, which describe how the simulated laparoscope pointing to the marker is oriented in three-dimensional space. The roll value characterizes the degree of rotation of the simulated laparoscope around the longitudinal axis, the pitch value characterizes the relative angle of the simulated laparoscope to the insert or grid, and the yaw value characterizes the rotation of the simulated laparoscope around the vertical axis.

[0175] According to various embodiments, the roll value of an angled laparoscope is determined in a similar manner to that described above. However, in various embodiments, the roll value is further complemented by a rotation value measured by the angled laparoscope or via a rotation sensor provided with the angled laparoscope. Specifically, in various embodiments, the roll value calculated using a PnP process of marker image data acquired by the angled laparoscope is further modified by the angle detected by a rotation sensor associated with the angled laparoscope. The further addition to the roll value calculated by the PnP process takes into account the complexity of operations that can be performed using an angled laparoscope, which can bring about camera / image sensor rotation via two different operating points. Exemplary details regarding the angled laparoscope are shown with respect to Figure 31.

[0176] Figure 28 shows an exemplary step of reprojecting a corner point using an identified corner. As illustrated, this step involves moving a 3D dot (2800) to coincide with a 2D point (2830) located at the corner. Once the reprojected 3D dot (2800) coincides with the 2D point (2830) acquired in Figure 26, the camera navigation system determines the positional information of the simulated laparoscope in the 3D space related to the training environment by solving this information as a PnP problem. The points (2830) and dots (2800) used in this calculation are based on all the multiple markers (2810) and black space (2820) captured in the image data. The more markers (2810) and black space (2820) captured, the more dots (2800) and points (2830) can be used, thus improving accuracy. As a result, the scope view generator can appropriately use solutions (e.g., obtained using the Levenberg-Marcaert algorithm) to generate the corresponding cursor position shown together with the digital environment displayed on the monitor. In various embodiments, the positional information of the simulated laparoscope can also be used to modify the digital environment to provide different viewpoints (i.e., simulate the viewpoint from the simulated laparoscope). Furthermore, the positional information of the simulated laparoscope can also be used to perform other calculations useful for camera navigation training, such as determining whether collisions caused by other computer-generated elements (e.g., tubes) exist between the cursor and the target.

[0177] In various embodiments, a monitor is used in conjunction with a camera navigation system to provide a user interface for the user to view a digital environment, similar to how a surgeon relies on a monitor to view the surgical field within a patient during a surgical procedure. The user can interact with the digital environment and the computer-generated elements displayed within it through the movement of a cursor that mimics the movement of a simulated laparoscope. As described above, the position of the simulated laparoscope relative to the training environment is indicated as such a cursor, correlated with the cursor in the digital environment, and subsequently displayed on the monitor. In this way, the user can use the simulated laparoscope and monitor to select various camera navigation training from a menu and perform these camera navigation training aimed at teaching and refining useful camera navigation skills or surgical procedures. In various embodiments, navigation and selection options (e.g., via the pressing of virtual buttons) are provided through the use of the simulated laparoscope. In various embodiments, other hardware (e.g., controller, keyboard) or other manual button presses are not required or provided. Rather, the scope view generator generates and displays a cursor (e.g., a small circle or arrow) on the monitor to perform specific selections (via the pressing of virtual buttons) or actions related to camera navigation training through the digital environment and computer-generated elements. The position of the cursor in the digital environment displayed on the monitor corresponds to the position of the simulated laparoscope relative to the training environment. The user performs the corresponding action via the simulated laparoscope to move the cursor to a different position in the digital environment displayed on the monitor. This correlation between the movement of the cursor displayed on the monitor and the movement of the simulated laparoscope simulates how a surgeon relies on the image on the monitor to manipulate the laparoscope and other surgical instruments within the patient during actual surgery.

[0178] In various embodiments, the distance the cursor moves within the digital environment displayed on the monitor corresponds to the distance the simulated laparoscope moves relative to the training environment. In various embodiments, this correspondence may differ. For example, if the monitor provides a zoomed-in view of the digital environment, the movement shown on the monitor may be more pronounced (i.e., twice or three times) than the movement of the simulated laparoscope relative to the training environment. The reverse is also true; if the view is zoomed out, the movement on the monitor may be less pronounced (i.e., half or one-third) than the movement of the simulated laparoscope relative to the training environment. In various embodiments, a camera navigation system may provide the user with a notification of the difference between the magnitude of the cursor movement and the magnitude of the corresponding movement of the simulated laparoscope.

[0179] In various embodiments, a cursor is used in the digital environment to represent the position of the simulated laparoscope relative to the training environment at any given time. Specifically, the cursor represents a point in the training environment to which the simulated laparoscope is pointed. The cursor position is based on periodic updates of positional information about the simulated laparoscope, obtained via image data of markers captured in relation to the training environment being used. In various embodiments, computer-generated elements (such as buttons, targets, menus, and obstacles generated with the digital environment) are based on stored data related to selected camera navigation training or functions (i.e., homepage, menus). In various embodiments, the scope view generator retrieves and displays the appropriate computer-generated element based on the user's current usage. The cursor facilitates interaction with various computer-generated elements (e.g., buttons). Such interaction is determined based on a comparison between the position of a stored computer-generated element in the digital environment and the current cursor position in the digital environment. Specifically, if the cursor position is within a predetermined threshold range related to the position of a computer-generated element in the digital environment (i.e., overlapping), it is determined that the cursor is interacting with that computer-generated element.

[0180] An example of a computer-generated element is the menu (400) shown in Figure 4A. Specifically, this figure shows an exemplary digital environment (405) including a cursor (410) and various buttons (420-450) that can be interacted with using the cursor (410). For example, the buttons (420-440) may include various camera navigation training that can be performed using a camera navigation system. In various embodiments, these activities may include focus activities (420), follow activities (430), and trace activities (440). In various embodiments, these three trainings can be used for both zero-degree laparoscopy and angled laparoscopy.

[0181] To interact with the buttons (420-450) on the menu, a cursor (410), shown as a small circle in the figure, can be moved within the digital environment (405) and overlapped with at least a portion of the buttons (420-450). For example, as shown in Figure 4B, the cursor (410) is currently overlapping the focus activity button (420). In various embodiments, the cursor (410) may also appear as a different object / symbol, such as an arrow, an "X," or other shape.

[0182] In various embodiments, the camera navigation system uses a cursor (410) to facilitate user interaction with various computer-generated elements (e.g., buttons, targets) within a digital environment (405) displayed on a monitor. In various embodiments, the cursor (410) may point to or overlap a portion of a particular computer-generated element. In various embodiments, it may be necessary to hold the cursor (410) in the same position for a predetermined period of time so that the cursor (410) points to or overlaps a portion of a computer-generated element. In various embodiments, the time requirement for confirmation with a particular computer-generated element of interest (e.g., a button) may be set to 2 seconds. However, this predetermined time can be adjusted as needed. For example, the camera navigation system may also confirm the selection of an element faster or slower than 2 seconds. In various embodiments, the predetermined time can also be adjusted to be faster or slower as appropriate to accommodate cursor interactions with computer-generated elements related to different training that require shorter or longer interaction times.

[0183] If the time required to select an element using the cursor is too short, unintended selections may occur. Conversely, if the time required to select an element is too long, the user may lose interest or doubt whether the element was selected correctly. In various embodiments, visual elements (such as a status bar (460) shown in Figure 4B) can be used to indicate how long the cursor (410) must be held in the same position before an object (e.g., a button (420)) is selected. The status bar (460) can fill up, fading in and out as needed, when the cursor (410) is held in a particular position. When the status bar (460) is full, it can be used to indicate that the button (420) has been successfully selected and to notify the user that the button (420) has been successfully selected. In various embodiments, the status bar (460) can be reset or slowly emptied if the cursor (410) is not in the appropriate position to interact with a computer-generated element (i.e., a button (420)). However, once the cursor (410) returns to its correct position, the status bar (460) can begin to fill up again.

[0184] Various embodiments can reduce the need to incorporate further hardware control devices, such as keyboards or keypads, into the camera navigation system by using only a simulated laparoscope to interact with computer-generated elements displayed on a digital environment and monitor, thereby simplifying the camera navigation system and / or development process. Furthermore, using only a simulated laparoscope allows for a more accurate simulation of actual camera navigation in laparoscopic surgery, which does not necessarily involve the use of keyboards, mice, and other hardware. If other auxiliary control devices, such as keyboards or touchscreens, are used, a non-invasive experience is introduced that does not correspond to actual surgical procedures, such as pressing keys on a keyboard or tapping a screen. Therefore, to provide an immersive experience that is highly compatible with surgical procedures, it is desirable to use a simulated laparoscope control cursor displayed on a monitor to navigate and interact with computer-generated elements.

[0185] In various embodiments, it is necessary to move a simulated laparoscope relative to the training environment in order to find buttons displayed on a monitor in a digital environment and read text. For example, to select a virtual button displayed on the monitor, the cursor in the digital environment must be kept stationary on a computer-generated element for a certain period of time. To move the cursor displayed on the monitor from one point to the desired position of a computer-generated element, the user must move the simulated laparoscope relative to the training environment by a corresponding amount. Therefore, even the act of navigating and interacting with auxiliary graphic elements can be used as a form of camera navigation training, as the user needs to measure distances on the monitor and move the simulated laparoscope relative to the training environment by a corresponding distance. Thus, training is also provided using a camera navigation system and its associated camera navigation training to understand the relationship between movements shown on the monitor and actual movements via the simulated laparoscope.

[0186] In various embodiments, when an application for a camera navigation system is started, the user interface can generate and display various different computer-generated elements that function as buttons related to various different training exercises shown on the monitor. For example, as shown in Figure 4A, there may be three buttons (420, 430, 440) corresponding to three different training exercises: trace, follow, and framing. Below these first three buttons, there may be an "exit" button (450) that allows the application to be terminated during interaction. Once a training exercise is selected using the cursor (410), the application can generate and display a level / difficulty selection screen (470) as shown in Figure 4C.

[0187] In various embodiments, a level / difficulty selection screen (470) replaces a menu (400) generated within the digital environment (405) that displays the available levels or difficulty levels (475) for that particular training at that point. When the scope view generator receives a command based on the selection of buttons (420, 430, 440), it determines what menus need to be generated and displayed for the user next. The scope view generator retrieves information about the necessary computer-generated elements and updates the digital environment accordingly to provide the level / difficulty selection screen in this example. Different users that the camera navigation system can identify may have different progress regarding which levels or difficulty levels are available. Levels or difficulty levels that the camera navigation system determines and / or judges to be available to a particular user may be highlighted (475), while levels or difficulty levels that are not yet available may be highlighted (476), and / or a symbol may be placed (i.e., locked) (477) to indicate that a particular user has not yet selected that particular level or difficulty setting. In various embodiments, it may be necessary to meet certain conditions or proficiencies tracked and / or enforced by the camera navigation system (i.e., completing a previous level / difficulty and / or achieving a predetermined rating / score in a previous training session) before being able to access a higher level or difficulty.

[0188] In various embodiments, buttons related to different levels / difficulty levels provided by the camera navigation system may display previews or hints for specific tasks to be performed. For example, as shown in Figure 4C, a follow activity may include images of example routes to be practiced at a particular level / difficulty level.

[0189] In various embodiments, the level / difficulty selection screen (470) provided by the camera navigation system may also include “bonus” training (478). These “bonus” training (478) are not required to complete the course assigned by the instructor, but can be used to further test specific skills. In various embodiments, the “bonus” training (478) provides more challenging tasks that the user can take to hone their relevant camera navigation skills.

[0190] In various embodiments, a "Home" button (480) can also be placed on the user interface screen related to the training. The "Home" button (480) provides the ability to return to the main screen when selected. On the main screen, buttons for various training sessions are displayed again, allowing the user to select the training session to perform or to completely exit the application.

[0191] In various embodiments, a specific training session begins when a difficulty level or level is selected. The Scope View Generator retrieves relevant information about the training and the computer-generated elements (e.g., targets, obstacles) to be used in that training. In various embodiments, when a user selects camera navigation training, corresponding computer-generated elements are generated to facilitate practice of skills related to camera navigation and are displayed on the monitor for interaction during training. Information related to the computer-generated elements stored in memory includes, for example, information such as the placement or movement of these elements in the digital environment. Thus, for example, when a specific camera navigation training session is selected, targets are placed in the same positions, and any necessary information about the targets (such as the target's position) can be retrieved as needed for various decisions performed by the camera navigation system (e.g., whether the cursor is near or overlapping the target). In various embodiments, a “control” element is provided that helps standardize user performance feedback by using the same computer-generated element positions for all users. Thus, all users are given the same scenario each time, and all users are evaluated using the same criteria. In various embodiments, the computer-generated element may have variable information (e.g., multiple possible starting positions) that can give the user further challenges to perceive and adapt to a given camera navigation training during each trial. In various embodiments, interaction with the computer-generated element requires the manipulation and control of a simulated laparoscope in the training environment.

[0192] In various embodiments, different options can be selected from menu buttons at various points during camera navigation training. For example, the camera navigation system may predetermine providing navigation to return to the home screen, navigation to return to the difficulty or level selection screen, and / or reset and restart the current training.

[0193] In various embodiments, the camera navigation system presents a feedback screen upon completion of training. Figures 5 to 9 illustrate various embodiments of the feedback generated by the camera navigation system and displayed in the digital environment. In various embodiments, these feedback screens provide summary information about the performance of the completed camera navigation training, quantified by the camera navigation system on one or more criteria. Different training may have different predetermined types of feedback. In various embodiments, the type of feedback can be narrowed down to the skills that the user wishes to focus on.

[0194] In various embodiments, the same training can be restarted from the beginning from the feedback screen, and the camera navigation system can provide and be predetermined to move to a different difficulty or level, return to the difficulty or level selection screen, and / or return to the home screen.

[0195] Various embodiments of a camera navigation system may provide animations during the execution of one or more different camera navigation training exercises. Often, animations can serve to draw attention to specific details. In various embodiments, certain prompts provided to the user by the scope view generator may not be recognized, or the task at hand may not be easily understood. For example, a stationary yellow arrow may be used to indicate where to start a new camera navigation training exercise or where to proceed next. However, without prior instructions or details, it may not be clear what the stationary yellow arrow represents or that interaction with the stationary yellow arrow is necessary to proceed to the next training exercise. The camera navigation system may use such an arrow to provide animations (for example, by making the yellow arrow vibrate back and forth in its indicated direction) so that the stationary yellow arrow is more conspicuous as something that should be interacted with.

[0196] In various embodiments, the camera navigation system provides some form of animation related to a prompt whenever a prompt for interaction exists. Another example of animation use is providing guidance for correcting the orientation of a simulated laparoscope during training. For example, if the simulated laparoscope is rotated more than a given threshold, or if the horizontal plane is tilted, the camera navigation system may provide a visual element to inform the user that correction is needed and provide instructions, hints, or other computer-generated elements on how to make the correction (e.g., rotating the simulated laparoscope in the opposite direction to correct the previous rotation). Prompts can be useful in situations where prompts might be overlooked during training. In various embodiments, the camera navigation system may make prompts flash, and animated prompts may attract more attention to themselves.

[0197] In various embodiments, prompts provided by the camera navigation system can also be provided non-visually. For example, prompts can be provided via sound (e.g., beeps) and touch (e.g., tactile feedback). These prompts can be used to provide information about whether training is being performed correctly. For example, the camera navigation system can use audio sounds or vibrations to notify the user that the cursor has strayed too far from a given path or object. Similarly, different audio sounds or different types of vibrations can be used to notify the user that the cursor has correctly acquired a target.

[0198] Audio prompts can include various sounds (e.g., beeps) or recordings that provide status-based information as described above. In various embodiments, audio prompts can complement information provided from other sources (e.g., visual). However, the use of audio prompts may depend on the availability of speakers and may be affected by environmental factors (e.g., noise level). For example, tactile elements provided on the handles of a simulated laparoscope can also be an alternative or complementary feature. However, implementing tactile elements may require adding additional hardware, for example, within the simulated laparoscope. Furthermore, vibrations associated with tactile elements may directly affect (i.e., shake) the image being captured by the simulated laparoscope.

[0199] In various embodiments, the position of the simulated laparoscope is monitored through the cursor position in the digital environment to provide feedback on the user performance of one or more training sessions. For example, the camera navigation system is configured to provide feedback on the performance of the camera navigation training, including determining whether the simulated laparoscope has rotated, whether the simulated laparoscope is at an appropriate observation distance, whether the simulated laparoscope is following the correct path, and / or the speed at which the simulated laparoscope is being operated relative to the training and / or the digital environment. In various embodiments, one or more of these criteria are evaluated during the execution of different training sessions by the camera navigation system. The following describes various application examples of metrics used to evaluate the performance of the camera navigation system. The use and assumption of other metrics are also possible, not limited to those described below.

[0200] In various embodiments, metric examples may include ensuring the simulated laparoscope maintains the centrality of the surgical field relative to the training environment, keeping target biostructures within the field of view, and / or tracing the path of important structures. This is quantified by determining a point of interest and detecting whether the cursor is a predetermined distance away from that point of interest. These metrics correspond to the skills practiced in relation to the operation of the simulated laparoscope, leading to the ability to provide the surgeon with an optimal view of the surgical site during actual surgery. During suturing and dissection, it is often necessary for the camera navigator to manipulate the laparoscope and zoom in so that small needles, instrument tips, and individual tissue layers are clearly visible to the surgeon. Maintaining the surgical field corresponds to the ability to maintain an appropriate viewing distance using the simulated laparoscope, which is instructed and practiced through the camera navigation system. In procedures such as colectomy requiring dissection through a tissue plane, it is important to maintain the correct horizontal plane and stay on the appropriate dissection plane, corresponding to the ability to minimize rotation of the simulated laparoscope. If camera navigation during the procedure is not optimal, it can lead to inefficiencies and increased surgical time that are inconsistent with the ability to complete training quickly and efficiently.

[0201] Regarding the rotation of the simulated laparoscope, the camera navigation system monitors the degree to which the simulated laparoscope twists or rotates around its axis (i.e., how many degrees). The amount of rotation can be evaluated based on the ability to manipulate the simulated laparoscope in relation to the training environment without unnecessary twisting or rotation. Generally, the less twisting or rotation of the simulated laparoscope, the better the score. This evaluation is quantified by monitoring how much and how frequently the position of the simulated laparoscope changes during the training period.

[0202] In various embodiments, feedback is visualized at the end of relevant training performed by the camera navigation system. Figure 5A shows an example of rotation feedback (500) provided by the camera navigation system in relation to the rotation of a simulated laparoscope. The rotation feedback (500) corresponds to a graph (504) showing how much the simulated laparoscope rotated over a period of time during the camera navigation training, as well as the frequency and direction (i.e., left / right or counterclockwise / clockwise). The rotation feedback (500) tracks the rotation of the simulated laparoscope over a period of time and identifies whether the rotation was left or right (502). Furthermore, different thresholds can be used to indicate on the feedback provided by the camera navigation system where an appropriate amount of rotation and where an undesirable amount of rotation has been detected. For example, the camera navigation system can use different colors to indicate an appropriate amount of rotation (green) or an excessive amount of rotation (red) during the performance of the training. In various embodiments, the camera navigation system provides user interaction with a rotation graph (410) using a cursor (410) to highlight further details regarding rotation feedback (500), such as a specific period or degree during which rotation is detected.

[0203] In various embodiments, another criterion used by the camera navigation system relates to its ability to maintain an appropriate observation distance from a given target. The observation distance uses a metric that measures the change in distance between the simulated laparoscope and the training environment. Specifically, the “observation distance” can be calculated by determining the “hypotenuse” of a triangle whose sides and angles are determined by information relating to the x, y, and z coordinates relative to the training environment, using positional information about the simulated laparoscope. Parallel lines in the digital environment allow the distance between the cursor position in the digital environment and the position of the given target to be measured as the observation distance. The less the distance changes during training, the better the score. To determine whether the simulated laparoscope is positioned at an appropriate distance, the current position of the simulated laparoscope (e.g., x, y, and z coordinates) is monitored in relation to the target position to determine whether the simulated laparoscope is located around a desired predetermined observation distance from the target.

[0204] After training is complete, observation distance feedback (510) can be generated and displayed as a graph. As shown in Figure 5B, the observation distance feedback (510) provides a graph (512) that shows the measured distance of the cursor's current position from a given target (514) throughout the training. In various embodiments, the observation distance feedback (510) provided by the camera navigation system has a graph (512) that tracks the observation distance over a period of time and quantifies whether the observation distance is close or far from a given target (514). Furthermore, different thresholds can be provided to indicate what an acceptable distance is compared to a less desirable distance. For example, the variation in observation distance and when the current distance at a given point in time was too close, too far, or favorable can be shown in different colors.

[0205] In various embodiments, accuracy refers to how well a given task was completed. In the case of trace training, accuracy refers to how well the cursor remained within a predetermined path defined in the digital environment. As shown in Figure 5C, the camera navigation system provides a path accuracy feedback graph (520) associated with exemplary training, which allows the simulated laparoscope to be controlled to move the relevant cursor along a predetermined path. Once the trace training is complete, the path accuracy feedback graph (520) can be generated in the digital environment (405). Determining whether the cursor is along a predetermined path is based on monitoring the cursor's position over a period of time and verifying whether the cursor corresponds to a memory position associated with the predetermined path. In various embodiments, the path accuracy feedback (520) plots a path (528) showing the exact path of the cursor corresponding to how the simulated laparoscope was manipulated. The cursor path (528) can overlap with a predetermined assumed path (526). The camera navigation system can use color or other indicators to show when the cursor remained within or deviated from the predetermined path. Specifically, if the path lies within a predetermined path (526), ​​the cursor's path (528) can be shown in green. On the other hand, if part of the cursor's path (528) deviates from the predetermined path (526), ​​such part (530) can be highlighted or shown in a different color (e.g., red). Furthermore, the camera navigation system can provide a path accuracy feedback graph (520) that includes labels (522) identifying what the graph displays, and a percentage (524) summarizing how long the cursor remained within the predetermined path (526) throughout the entire training.

[0206] Figure 6 shows a user interface combining the three types of feedback described in Figures 5A to 5C into a single composite feedback display (600) in a digital environment, according to various embodiments. Specifically, as shown in the figure, feedback is provided for rotation (620), accuracy (630), and observation distance (640), along with a level difficulty identifier (615) used by the camera navigation system to indicate to the user which training has just been completed. Each of the feedback graphs provided to the composite feedback display (600) has a plot that quantifies the user's performance as determined by the camera navigation system with respect to rotation (625), accuracy (635), and / or observation distance (645) detected during training. Such a user interface can be viewed on a monitor.

[0207] Furthermore, in various embodiments, additional information (610) related to the illustrated composite feedback (600) can be provided at a predetermined location in the digital environment (e.g., on the side of the current view of feedback 600). For example, the additional information (610) may include threshold guidelines related to different parameters used to evaluate training performance, numerical scores / evaluations comparing that performance to the thresholds, and "best" scores achieved in past performance. In areas related to the parameters being evaluated (e.g., rotation (620), accuracy (630), observation distance (640)), plots generated and provided by the camera navigation system can show areas where improvement has been seen and areas where further improvement is possible. Examples of identifiers may include changing the colors of the graphs (620, 630, 640) to green (corresponding to acceptable performance), yellow (corresponding to performance that can be improved), and red (corresponding to unacceptable performance).

[0208] In various embodiments, the camera navigation system may provide various buttons (650, 660, 670) below the composite feedback display (600) that the user can interact with. A retry button (650) may be provided to allow the user to repeat a training session that has just been completed. A next button (660) may be provided to provide or start the next training session (i.e., the next difficulty level). A home button (670) may be provided to display the main menu (400).

[0209] Figure 7 shows another embodiment of the user interface of the composite feedback display (600) shown in Figure 6. This embodiment shares many of the same features already described in Figure 6, including various different feedback for rotation (620), accuracy (630), and / or observation distance (640), each plotting (625, 635, 645). However, the embodiment shown in Figure 7 (700) includes, instead of the additional information (610) that was displayed with the composite feedback display (600) in Figure 6, a list (680) of training sessions in which the user performance of the relevant skill reached "mastery" (i.e., the relevant performance exceeded a predetermined mastery threshold and the training was completed) using a specific threshold related to the level or difficulty determined by the camera navigation system. As shown in the mastery-related information (680) on the side, the score reflects a "mastered" score because the performance was within acceptable parameter (i.e., rotation, accuracy, distance) thresholds. In various embodiments, even if the score achieved by the user is not "perfect," the camera navigation system may provide a corresponding message highlighting the user's improvement if it determines that an improvement from past performance (i.e., a new high score) has been observed. For example, it may provide a notification via the user interface (e.g., "great job") and / or a graphical visual effect to indicate that improvement has been detected in one or more areas.

[0210] Depending on the camera navigation training, different types of feedback provided by the camera navigation system may be appropriate, and these can therefore be used to quantify performance during camera navigation training. For example, as shown in Figure 8, further types of feedback can be used to evaluate performance during tracking training. In this figure, the feedback can quantify the time the cursor was located within a location defined by a given path compared to the time the camera navigation system determined the cursor was outside the given path. In another embodiment, with respect to follow training, the accuracy criterion may include further feedback data, such as evaluating how well the cursor followed the target as the target moved along the path. The feedback may also depend on the type of path (e.g., whether the path is predetermined or random). In another embodiment, with respect to framing training, the accuracy criterion may further be used to evaluate how stably the cursor was held when focused on the target.

[0211] As mentioned above, one criterion for the camera navigation system to evaluate performance during training is monitoring the rotation of the simulated laparoscope. The rotation of the simulated laparoscope is measured in degrees by how much it twists around one or more axes. Specifically, rotation can be measured with respect to the longitudinal axis and / or vertical axis. A rotation of zero degrees means that the simulated laparoscope is aligned with the horizontal plane (i.e., the plane of the insert or grid). In various embodiments, the horizontal plane is defined as having a roll value of 0. Generally, this definition of the horizontal plane corresponds to the bottom side of the image sensor used to acquire image data of the markers. The reason this particular position is desirable is that it is the easiest for the surgeon to manipulate in this position. A rotation of 90 degrees means that the simulated laparoscope is twisted perpendicular to the horizontal plane. Generally, the position of the simulated laparoscope is undesirable if it is not aligned with the horizontal plane, as this makes perception and manipulation within the area difficult.

[0212] Figures 10A and 10B show exemplary calculations for determining the rotation of a simulated laparoscope in a digital environment. These figures show exemplary image captures (1000) of an insert or grid. The image captures (1000) received by the camera navigation system include marker 305 and a dark square (310) between markers 305. The image captures (1000) determine the rotation using the x and y axes. For illustrative purposes, Figures 10A and 10B are overlaid with x and y graphs (1005). In addition to the x and y axes shown on the x and y graph (1005), there is also a third line representing a line perpendicular to the plane of the insert or grid. The rotation of the simulated laparoscope is measured in degrees, expressed as an angle between the y axis and the third line. This rotation is shown as an arc between the y axis and the third line. This rotation metric is useful because the training objective is to keep the posture of the simulated laparoscope aligned with the horizontal plane. If the simulated laparoscope is rotated too much from side to side during actual surgery, the surgeon may lose their sense of direction, hindering the progress of the surgery. Figures 10A and 10B show different positions of the simulated laparoscope used during image data acquisition. Specifically, the image capture in Figure 10A shows excessive rotation to the right because the third line is between the x and y axes. On the other hand, the image capture in Figure 10B shows excessive rotation to the left because the third line is outside the x and y axes. Ideally, the third line should be aligned with the y axis so that the orientation of the image data is correct when viewed on the monitor.

[0213] In various embodiments, the values ​​used to quantify performance during training can be customized. In one embodiment, the camera navigation system can be configured to determine or rate the maximum rotation from a preferred 0-degree rotation, with ±30 degrees of rotation being "POOR," ±20 degrees being "OK," and ±10 degrees being "GOOD." By changing these values, the camera navigation system can adjust what can be evaluated as performance appropriate for different procedures or different experience levels. In various embodiments, the thresholds can also be based, for example, on the type of training being performed and / or the difficulty of the training (i.e., the more difficult the training, the less tolerance the threshold has for changes away from 0 degrees of rotation).

[0214] Figure 11 shows an exemplary embodiment of a meter (1110) provided by a camera navigation system to notify the user of their performance. In various embodiments, a graphic element (e.g., a meter) (1110) may be provided that appears when the camera navigation system detects that the simulated laparoscope has begun to rotate left or right during training. This figure shows an example of a meter (1110) related to the operation of a cursor (410) within a given path (1130). In various embodiments, the camera navigation system may provide different meters to quantify the user's performance on various different criteria.

[0215] In various embodiments, the meter (1110) may have different zones (1115) corresponding to different scores (e.g., poor, OK, good). In various embodiments, the middle zone may correspond to a "good" performance. The next zone adjacent to the middle zone may correspond to a different performance score (e.g., "OK" or "acceptable"). The last zone at the end of the meter may correspond to a "poor" performance score. These zones may also be color-coded, for example, green for the "good" zone, yellow for the "OK" zone, and red for the "poor" zone, so that they are easily referable to the user. A tracker (1120) provided by the camera navigation system may be configured to track or display where the user is currently located along the meter example (1110) with respect to the rotation of the simulated laparoscope. For example, if the user is not in a desired rotational orientation, the camera navigation system may, as needed, provide hints or instructions (1125) commanding the user to rotate in a particular direction. Since such information is not easily reflected solely through a cursor, these hints (1125) are intended to help guide the user to obtain the correct rotation of the simulated laparoscope.

[0216] In various embodiments, the meter can be configured to remain hidden as long as the measured rotation of the simulated laparoscope is controlled by the camera navigation system within the "GOOD" degree range (+ / 10 degrees). Alternatively, in various embodiments, the camera navigation system can be configured to reveal or display the meter if the simulated laparoscope rotates even slightly, or if it rotates beyond its "good zone" into other size zones (corresponding to "OK" or "poor"). In various embodiments, the meter indicates how far the simulated laparoscope has deviated from the desired orientation determined by the camera navigation system. Furthermore, the meter can also be used as a prompt or guide for how to perform corrective actions, such as how much the orientation of the simulated laparoscope should be rotated to the left or right. Since such a meter is unlikely to exist in actual surgery, the camera navigation system can prevent or limit reliance on the meter to maintain the correct orientation by only displaying the meter when necessary.

[0217] According to various embodiments, another criterion used to evaluate performance is the ability to maintain an appropriate observation distance between the simulated laparoscope (1210) and the target. Figure 12 shows an exemplary calculation of the observation distance of the simulated laparoscope (1210) as determined by the camera navigation system. As shown in Figure 12, the observation distance (1220) measured with respect to the training environment (1240) corresponds to the distance between the end of the simulated laparoscope (1210) and the training environment (1230) or target area (1240). The desired observation distance (1220) can be set shorter or longer depending on the procedure being simulated and / or the desired level or difficulty of practice as determined by the camera navigation system. Furthermore, the camera navigation system can also adjust different ranges and associated scores or performance qualifications related to the actual measured observation distance of the simulated laparoscope. For example, in one embodiment, the difficulty level or level set or identified by the camera navigation system may have measured observation distances of ±3 centimeters quantified as "poor," ±2 centimeters quantified as "OK," and / or ±1 centimeter quantified as "good," as defined by the camera navigation system.

[0218] In various embodiments, if the camera navigation system determines that the simulated laparoscope is positioned outside the desired or "good" observation distance range during training conducted within a digital environment (405), a range meter (1310) operating similarly to the simulated laparoscope tachometer (1110) described above (as shown in Figure 11) can be provided. Figure 13 shows an exemplary embodiment of the range meter. As shown, the range meter (1310) is configured to provide different zones (1315) corresponding to "poor," "OK," and "good" performance / scoring requirements. For example, the first central zone may correspond to "good" performance in maintaining the desired observation distance, the zones directly adjacent to the left and right of the "good" zone may correspond to "OK" zones, and the zones at both ends of the range meter may correspond to "poor" performance / scoring. A tracker (1320) provided by the camera navigation system is provided to track the user's performance based on a meter, which may also have relevant prompts or directions (1330) that can be used as a guide to instruct the user how to return to the ideal observation distance by moving the simulated laparoscope away from or near the target area within a given path (1335).

[0219] As described above, in various embodiments, there are various criteria (such as observation distance and / or rotation of the simulated laparoscope) that the camera navigation system determines and / or uses to quantify and / or provide corresponding feedback during the execution of camera navigation training. In various embodiments, the camera navigation system provides various different training aimed at training different skills related to camera navigation during surgical procedures. Below, some training examples such as target navigation, moving object tracking, and image stability are described. In various embodiments, the camera navigation system determines or quantifies image stability as the cursor remaining in a given position for a given time without changing or moving beyond a given movement threshold from that position. In other words, the camera navigation system determines that the image data being acquired is stable and free from large shakes, whether for identifying the position of the simulated laparoscope or during the execution of one or more camera navigation training sessions. In other embodiments of the camera navigation system, additional different training may be implemented, which may be aimed at camera navigation in general or other skills useful for specific laparoscopic surgery. Furthermore, although these camera navigation training exercises are described below in the context of applications using zero-degree laparoscopes, these camera navigation training exercises can also be applied to other types of laparoscopes (e.g., angled or 30-degree).

[0220] Figures 4A to 4C illustrate various embodiments of the computer-generated menu of the camera navigation system. In various embodiments, the camera navigation system provides one or more different camera navigation trainings that can be selected and / or performed by viewing them through a computer-generated menu on a monitor (as shown in Figure 4A). The camera navigation system can provide specific levels or difficulty levels for each camera navigation training, which the user can select via a simulated laparoscope (as shown in Figure 4C). In various embodiments, the camera navigation system can be configured to set or provide levels or difficulty levels that may have different metric parameters for evaluating the performance of the training. In various embodiments, the levels or difficulty levels provided by the camera navigation system may have different positioning requirements for the user relative to the training environment (e.g., standing beside or in front of the surgical trainer) that may present challenges to achieving correct positioning of the simulated laparoscope (e.g., position and / or orientation). Camera navigation training, combined with simulated surgery setup, manipulation of the simulated laparoscope within the trocar, and stick-slip friction and fulcrum effects associated with the manipulation of the simulated laparoscope in relation to the surgical trainer, provides laparoscopic operators with relevant and comprehensive dimensions of experience and practice outside the operating room that can be translated into actual laparoscopic procedures during surgery.

[0221] In various embodiments, camera navigation training examples that a camera navigation system can perform and provide are called “trace” training. “Trace” training measures how well a user can manipulate the position of a simulated laparoscope to move a corresponding cursor along a predetermined path to a designated position in the digital environment. In practice, surgeons often identify and trace important structures along paths such as the ureters or vascular system to avoid unintended damage or to help identify other structures. Furthermore, it is common practice for surgeons to examine the entire surgical field (through outlining) preoperatively and postoperatively to check for abnormalities. Skills such as dexterity, precision, and overall handling of the laparoscope are helpful in performing the above exemplary tasks and learning to overcome the challenges posed by lack of depth perception, the presence of stick-slip friction, and the fulcrum effect, and therefore, the trace training described below helps improve these skills.

[0222] Figures 14A and 14B show exemplary embodiments of trace camera navigation training provided by a camera navigation system. One objective of “trace” training is to practice moving a cursor (410) along a set path (1420) from a start point (indicated by a start identifier 1415) to an end point (1425) while maintaining the cursor (410) within the set path (1420) in a digital environment (405) generated, updated, and / or evaluated by the camera navigation system. Referring to Figure 14A, this figure shows an example embodiment (1410) of “trace” training as seen on a monitor. In various embodiments, the camera navigation system can give commands to the training environment to operate a simulated laparoscope to move the corresponding cursor (410) along a specific path (1420) within the digital environment (405). Trace training is useful for simulating laparoscopic surgery, allowing users to practice structuring the surgical field.

[0223] In various embodiments, once a level or difficulty level is selected, the camera navigation system can indicate a specific path (1420) to follow. Each level or difficulty level may have a different path with different characteristics. In other words, the difficulty level of the “trace” training predetermined by the camera navigation system depends on relevant metrics used to quantify, for example, the complexity of the path, the width of the path, and / or the performance of the training (i.e., the amount of roll that can be tolerated, the change in observation distance that can be tolerated). As shown in Figure 14A, the specific path (1420) that is generated and displayed may be a straight path, while another difficulty level may have a more complex path with respect to the specific path (1460) corresponding to the illustrated training, for example, as shown in Figure 14B. In various embodiments, the specific path to be followed in the “trace” training is stored in memory along with the camera navigation system. Furthermore, the camera navigation system may also provide access to create new levels and / or modify old levels.

[0224] During a "trace" training run, the camera navigation system periodically monitors the position of the simulated laparoscope relative to the training environment and updates the position of the cursor (410) in the digital environment. The camera navigation system compares the position of the cursor (410) with whether its position lies within a defined boundary associated with a particular path (1420). Thus, this determination is the same for each run of the same camera navigation training, regardless of whether it is performed by the user or another user, and uses stored information associated with the particular path (1420). In various embodiments, the camera navigation system can randomize the path, but can maintain the same number of elements (e.g., length, number of turns, type of turns) to give the user an additional challenge by not informing them in advance what needs to be done for the training.

[0225] Referring to Figure 14A, the “trace” training can also provide a computer-generated element that functions as a meter (1430) indicating a gauge of the user’s performance while performing the “trace” training. Specifically, the meter (1430) may have different zones that can be used to quantify whether the user is performing the training properly or whether the user needs instructions. In various embodiments, remarks (1435) can be given to inform the user of how to improve their performance in the current training. For example, in Figure 14A, the observation distance associated with the cursor (410) may be outside a given range. Thus, the remarks (1435) given by the camera navigation system from the “trace” training can instruct the user to “zoom in” the simulated laparoscope.

[0226] In various embodiments, the camera navigation system is configured to provide “trace” training, which includes training-related information (1445) that provides the user with additional information about the training as instructions to the user regarding how to perform the training and relevant thresholds for performance. In various embodiments, the camera navigation system may display the user’s performance (past and present) for reference.

[0227] In various embodiments, the "trace" training may also include a "home" button (1440). The "home" button (1440) is configured to end the current training and return to the main menu or to the level / difficulty selection.

[0228] Referring to the embodiment shown in Figure 14b, the “trace” training (1450) provided by the camera navigation system may include an arrow (1455) pointing to the starting point or the next point of interest (1465) of that level. This highlighted point of interest (1465) may include the “start” position of the training and any subsequent point to which the user is instructed to move the cursor (410). As a result, the user knows where to position the cursor (410) to begin the level of the “trace” training. In various embodiments, the camera navigation system may display one or both of the rotation meter and the observation distance meter while the cursor (410) is moving to reach the starting point or the next point of interest (1465). The rotation meter and observation distance meter are also used to help set up and maintain the “trace” training, for example, by ensuring that the simulated laparoscope is positioned at a specific distance from the surface and in the correct orientation. In various embodiments, before further training, the camera navigation system can prompt the user to ensure that the cursor is within the appropriate (i.e., "intermediate") zone for both the rotation meter and the observation distance meter to ensure that the appropriate starting conditions are met. The camera navigation system provides direct feedback on how to correctly set the training starting conditions by initially displaying both the rotation meter and the observation distance meter in real time. Furthermore, if rotation and / or observation distance errors occur, the system can use the meters to indicate when these errors occurred (and throughout the training) so that these errors can be corrected via the simulated laparoscope.

[0229] In various embodiments, the user interface relating not only to this “trace” training but also to other training described herein (such as the “follow” training described above and / or the “framing” training described later) is configured to have an information section (1445) containing details relating to the training being performed. As shown in Figure 14A, such information (1445) may include the name of the selected training, the difficulty / level being performed, instructions indicating what must be done in the training to complete it, and / or one or more guidelines relating to one or more parameters (e.g., rotation, distance, accuracy) being evaluated in relation to the performance of the training.

[0230] The camera navigation system, once it determines that the cursor (410) is in the correct position at the start of training, indicates the first checkpoint at some point along the path (1460). For example, as shown in Figure 14B, the training path (1460) may have multiple turning points and intersections where the cursor (410) is manipulated via the use of a simulated laparoscope. In various embodiments, the camera navigation system may indicate checkpoints (1465) one by one to clarify the target and prevent confusion about where the cursor (410) needs to be manipulated via the simulated laparoscope. In various embodiments, the checkpoints (1465) may be colored circles. Furthermore, by providing only one checkpoint at a time, the camera navigation system also simulates the situation where the surgeon cannot proceed until an optimal view of the surgical field is obtained. Once the cursor (410) is successfully positioned over a checkpoint (1465) determined by the camera navigation system, the camera navigation system causes that checkpoint (1465) to disappear and a new checkpoint to be indicated. In various embodiments, the camera navigation system may provide an arrow (1455) indicating the next checkpoint (1465). The arrow (1455) may disappear after a predetermined period of time determined by the camera navigation system, or after the cursor (410) has begun to move in the indicated direction. In various embodiments, the checkpoint (1465) cannot be retrieved if the cursor (410) is outside a predetermined path (1460) and / or outside an acceptable range of rotation of the simulated laparoscope and / or observation distance.

[0231] In various embodiments, the camera navigation system may also provide multiple checkpoints (1465) simultaneously. These multiple checkpoints (1465) can be given sequential instructions so that the user knows how to proceed from one checkpoint (1465) to another along a predefined path (1460). In various embodiments, the camera navigation system allows the user to proceed to the checkpoints (1465) in any order, as long as all checkpoints (1465) are covered.

[0232] Figure 15 shows an exemplary calculation for determining the proficiency level of trace camera navigation training as determined by the camera navigation system. In various embodiments, the path the cursor traveled in the digital environment is compared to each path segment along a given path in order to determine how well the cursor was maintained along the entire path. Figure 15 shows an exemplary analysis for determining how well the cursor was maintained along the entire path. In one embodiment, the center of the path (1510) associated with a given path (1505) is compared to the position of the cursor (410). The camera navigation system can use the distance (1530) between the center of the path (1510) and the cursor (410) to quantify the performance of the “trace” training performed in the digital environment (1520). The level is completed when the camera navigation system determines that all checkpoints along the given path have been collected. Overall feedback is then displayed. In various embodiments, the camera navigation system measures the accuracy of how well the user performed the trace training based on comparing the time spent within the path (1505) with the time spent outside the path (1505). In various embodiments, the degree to which the cursor moves outside a predetermined path (e.g., measurement distance 1530) can also affect the calculated feedback related to accuracy.

[0233] For example, providing summative feedback in the aforementioned "trace" training is useful for evaluating overall performance. Furthermore, the feedback can also be used to identify areas for improvement. In various embodiments, the feedback display provided by the camera navigation system includes various metrics used to quantify training performance. For example, in various embodiments, the feedback display may include graphs showing rotation and observation distance performance over the training range, bar graphs showing how accurately the cursor stayed within the path during training, the time taken to complete the training, and / or a leaderboard showing previous scores and the scores of other individuals who have completed the same training. These metrics represent the camera navigation skills required in surgical procedures. The camera navigation system may also determine and include further different metrics in the feedback display to further quantify performance.

[0234] Data used for feedback and / or leaderboards for "trace" training or any other training described in various embodiments can be stored locally in the camera navigation system's memory. Therefore, feedback and / or leaderboards can be limited to use associated with a specific configuration device or within a network of a certain physical location (i.e., a hospital, school). However, in various embodiments where part of the camera navigation system is implemented remotely (e.g., via a remote server, the internet), data from all users can be stored remotely from the users and combined with data from users in different locations. In various embodiments, it may also be possible to implement a camera navigation system that can present feedback based on a combination of data from all participants in an event on its respective device. Leaderboards can include all data from all users, even if users are physically separated and perform the same training on different trainers. Furthermore, it is possible to perform various trainings at different physical locations and manage the relevant data accordingly.

[0235] Figures 16A and 16B show exemplary embodiments of follow camera navigation training provided by a camera navigation system. "Follow" training is another type of training that a camera navigation system can provide, which provides training in a different skill set compared to the "trace" camera navigation training described above. In "follow" training, the camera navigation system is configured to measure the ability to control the position of a cursor (i.e., a ring) (410) within a digital environment (405) to follow a moving target (1610) indicated by a filled circle and a surrounding zone (1620) as shown in Figure 16A. In some embodiments, the cursor (410) must remain within a surrounding area (1620) indicated as an acceptable range in which the cursor (410) can be positioned, related to the moving target (i.e., a filled circle) (1610), and must not go beyond the surrounding area (1620) or the area indicated by the moving target (1610) for at least a predetermined period of time.

[0236] In various embodiments, the camera navigation system is configured to initiate “follow” training as well as “trace” training, relating to identifying the movement of a cursor (410) to a starting position and identifying or determining that the cursor is positioned at an appropriate observation distance and orientation. The starting position must be maintained for a predetermined time determined by the camera navigation system, which can indicate or display this state by having a meter (1640) filled or empty. The meter (1640) may have various zones that can be used to quantify how well the cursor (410) is positioned. The camera navigation system may use a marker (1650) to highlight where in the zone associated with the meter (1640) the user's performance is currently being quantified. Based on the user's performance, the camera navigation system may provide findings or hints (1660) that help improve the positioning of the cursor (410). Once the camera navigation system determines that the initial starting position is filled, it can give commands to guide the cursor (410) to follow a moving target (1610) along a predetermined path (1630). In various embodiments, the camera navigation system moves a moving target (1610) along a predetermined path (1630) at a set speed. Information about the predetermined path (1630) related to a specific training level or training difficulty is stored in memory. In some embodiments, the target (1610) can move at a variable speed and / or can start and stop at variable points, such features changing the difficulty of the camera navigation training being performed.

[0237] As described above, examples of “follow” training are shown in Figures 16A and 16B, where the moving target (1610) is shown as a black circle moving along the path (1630). In various embodiments, the path (1630) can be made invisible. In various embodiments, the moving target (1610) can move in a random manner not defined by a given path. However, in order to maintain continuity between different users using the same difficulty or level, the characteristics related to the path are in the same state and can have the same number of turns, start / stop points, etc. The order of the path features can be rearranged so that the user cannot memorize and predict the path, but rather must respond and react as appropriate. The reason this skill is particularly important is that a good camera navigator can predict the surgeon's next move during surgery in relation to a particular surgical or laparoscopic procedure. “Follow” training can improve the user’s ability to predict and adapt the movement of the cursor (410) relative to the moving target (1610) and maintain the optimal observation distance throughout. Therefore, "follow" training provides practice in which the user predicts the surgeon's real-time movements during laparoscopic procedures and moves the laparoscope accordingly.

[0238] In various embodiments, the camera navigation system is configured to slow down or completely stop the moving target (1610) if it determines that the cursor (410) (or part thereof) is moving outside the surrounding area (1620) associated with the moving target. In various embodiments, the camera navigation system may also slow down or stop the moving target (1610) if the rotation and / or observation distance range of the simulated laparoscope is outside the acceptable range. This action simulates real-world conditions in which a surgeon must interrupt surgery due to the inability to see the laparoscope properly positioned by the camera navigator. Thresholds related to the acceptable values ​​of rotation, position, and / or observation distance can be adjusted, for example, based on the difficulty level of the selected training. Thresholds can be lowered for more difficult training, while thresholds can be higher for easier training.

[0239] In various embodiments, a visual indicator (such as the illustrated meter (1640)) can be used to indicate in the digital environment that the cursor is not within the optimal view of the moving target. Such visual indicators can be used in “trace” training (for example, as shown in Figures 11 and 13). In the embodiment shown in Figure 16B, an example meter (1640) can be used to indicate that the cursor position (in this case, the observation distance) is outside the acceptable range. A meter (1640) can be shown next to the moving target (1610), providing a gauge indicating how far it is deviating from the acceptable range. The meter (1610) can be divided into several subsections, each representing the distance the cursor is from the appropriate observation distance range for the moving target. As the observation distance (corresponding to the insertion depth of the simulated laparoscope) is adjusted, the meter (1610) can be adjusted accordingly to indicate whether the observation distance has improved or worsened due to this adjustment of the simulated laparoscope. The camera navigation system can use a marker (1650) to visually indicate to the user which subsection the user is currently in with respect to the cursor position. Furthermore, in various embodiments, the ring associated with the cursor (410) can be made larger or smaller as the simulated laparoscope moves closer or further away, respectively. In situations where the meter (1640) is not visible for measuring the depth of the simulated laparoscope, this change in the ring can be used as an indicator.

[0240] Furthermore, the size of the meter (1640) of the cursor (410) provided by the camera navigation system can also affect how difficult or easy it is to keep the cursor (410) within the surrounding area (1620) of the moving target (1610). For example, a small ring can be more easily kept within the surrounding area (1620) than a large ring, but it still needs to be large enough not to encroach on the area indicated by the moving target (1610). On the other hand, a large ring can easily move with the moving target (1610) and avoid encroaching on the moving target's space, but it may be difficult to remain within the surrounding area (1620). Therefore, the challenge is to have a ring size that can adapt to the moving target while maintaining a balance in distance from the moving target (1610).

[0241] In various embodiments, the camera navigation system is configured to wait for the moving target (1610) until it is determined that the cursor (410) is at an appropriate predetermined position, orientation, and / or observation distance. The camera navigation system can provide visual indicators as described above to instruct how to correct the cursor's position, orientation, and / or observation distance. The camera navigation system is then configured to continue moving the moving target (1610) once the cursor (410) has achieved an optimal view (corresponding to the desired position, orientation, and observation distance). The level is considered complete and training is deemed finished when it is determined that the moving target (1610) has reached the end of the path (1630). Feedback related to the training performance is then displayed.

[0242] Similar to the feedback provided in "trace" training, the feedback provided by the camera navigation system for "follow" training can also include information used to quantify the performance of the user and / or other users. For example, the feedback could include graphs of simulated laparoscope rotation and observation distance performance. Other feedback could include bar graphs showing how accurately the target was tracked and the time taken to complete the training. Furthermore, the feedback could include a leaderboard showing a user's past scores, as well as the scores of other users who have completed the same training. These scores can be organized in various different ways (i.e., ranked based on performance).

[0243] In various embodiments, the difficulty level of this training, predetermined or set by the camera navigation system, can depend on a variety of factors. For example, the difficulty level can be based on the complexity of the path, the speed at which the target moves, the speed changes and / or stops performed by the target, the size of the target (i.e., the ring), and acceptable metrics related to the acceptable field of view acquired by the simulated laparoscope (e.g., how much rotation of the simulated laparoscope is acceptable, how much change in observation distance is acceptable), all or some of which can be predetermined or selectable via the camera navigation system.

[0244] Figures 17A and 17B show exemplary embodiments of framing camera navigation training, another type of camera navigation training that a camera navigation system can provide. "Framing" training measures and trains the ability to position and hold the cursor (1700) in a stable manner. Generally, this skill is applicable to actual surgical procedures, as surgical assistants need to change and hold the position of the laparoscope according to the surgeon's requests.

[0245] During "framing" training, one or more targets (1720) are presented by the camera navigation system. The targets (1720) can be placed at one or more random or predetermined locations within the digital environment (405). Furthermore, the targets (1720) may have numerical identifiers (1725) that help identify the order in which the targets (1720) need to be captured and how many targets (1720) are present in the training. Generally, the targets (1720) are oriented toward the simulated laparoscope. Thus, the placement of the targets (1720) in the digital environment for "framing" training is designed to simulate the surgical site. The user (i.e., the surgeon's assistant) can then be instructed to manipulate the cursor to obtain a view of a desired location within the training environment. The camera navigation system determines whether the simulated laparoscope has properly framed the target by comparing the cursor's position within the digital environment with one or more targets assigned to the digital environment.

[0246] While the simulated laparoscope is being operated relative to the training environment, a cursor (1700) is displayed on the monitor. An embodiment showing the cursor (1700) and one of the targets (1720) can be seen in Figure 17A. In various embodiments, the cursor (1700) corresponds to the desired view when the cursor is in the correct position relative to one of the targets (1720). A cursor (1700) is generated that can be transparent or partially translucent and remains in the center of the monitor. As shown in Figure 17B, the goal is to manipulate the cursor (1700) so that the target (1720) is positioned at the same location as the cursor (1700) and the cursor (1700) and the target (1720) directly overlap each other.

[0247] In various embodiments, the cursor (1700) and / or target (1720) may include alignment markers (1710). Generally, a camera navigation system may represent the alignment markers (1710) as parentheses within the cursor (1700) and / or target (1720). The alignment markers (1710) further assist in the "framing" or overlapping of the cursor (1700) and target (1720) by providing an additional reference point that the user can rely on to determine how to move the cursor (1700) to overlap the target (1720).

[0248] In various embodiments, the camera navigation system is configured to determine that a simulated laparoscope (and even a cursor (1700)) has been held for a predetermined period of time before moving to a different target (1720). In various embodiments, all targets (1720) can be generated and displayed at the start of training. In other embodiments, the camera navigation system can provide subsequent targets (1720) only after successfully capturing the current target (1720) determined by the camera navigation system.

[0249] The cursor position may not be the sole factor in ensuring the acquisition of the desired view (corresponding to the overlap / match between the overlay and the target). The positioning of the simulated laparoscope can also be another factor considered by the camera navigation system, as the cursor and target may not be perfectly aligned due to distortions resulting from different field-of-view angles. For example, the cursor and target may not be perfectly aligned if the simulated laparoscope is rolling, or if the simulated laparoscope (i.e., the observation distance) is too close or too far. Thus, the camera navigation system can store specific details regarding the correct position, orientation, and observation distance for the cursor to be considered correctly acquired within the digital environment (increasing or decreasing buffer thresholds based on difficulty). The user's task is to manipulate the simulated laparoscope to achieve the corresponding cursor position, orientation, and observation distance in order to "frame" the assigned target.

[0250] In various embodiments, when the cursor (1700) and target (1720) perfectly coincide or overlap, the scope view generator may start a timer to track the time required to maintain the cursor position and / or the positioning of the simulated laparoscope. The camera navigation system may show or display the timer as a bar that goes from empty to full. When the timer bar is full, it may indicate that the current target (1720) has been successfully "framed". In various embodiments, if there is movement that causes a misalignment between the cursor (1700) and target (1720), the timer bar may stop filling and slowly become empty or completely empty. When the cursor (1700) and target (1720) perfectly coincide or overlap again, the timer bar may continue to fill.

[0251] In various embodiments, the "framing" training ends after a predetermined number of targets (1720) have been completed (i.e., matched / overlapped with a predetermined number of targets (1720)). The number of targets (1720) can vary based on the selected difficulty level. In various embodiments, feedback provided by the camera navigation system is displayed in a similar manner to that described above for previous training provided by the camera navigation system. For example, feedback for "framing" training provided by the camera navigation system may include a bar graph showing the accuracy of the alignment of the targets (1720) with the cursor (1700), the time taken to complete the training, and a leaderboard showing previous scores of users and / or other users who have completed the same training. In various embodiments, the difficulty level of the "framing" training, set or predetermined by the camera navigation system, depends on a predetermined tolerance indicating whether the cards match, the length of time required to focus on each target (1720), and / or the number of targets (1720) that need to be found.

[0252] In various embodiments, a "Home" button (1730) is provided if the user wishes to end the training before completion. In various embodiments, the camera navigation system determines that the overlay (1700) has interacted with the "Home" button (1730), allowing the user to return to the main menu screen or select a different level or difficulty level.

[0253] While this specification has described some camera navigation training in the context of zero-degree laparoscopy, in various embodiments, angled laparoscopy is also applicable, although it involves some computation to convert the image data of the angled laparoscopy into the corresponding zero-degree representation generated by the camera navigation system. In various embodiments, the camera navigation system can provide the different training described below in relation to "angled" laparoscopy. In some embodiments, these applications may not be suitable for zero-degree laparoscopy due to the physical limitations of zero-degree laparoscopy.

[0254] The difference between zero-degree laparoscopes and angled (i.e., 30-degree) laparoscopes lies in the placement of the image sensor / camera at the distal end of the simulated laparoscope. In angled scopes, the image sensor / camera is not aligned with the longitudinal axis of the simulated laparoscope. Angled configurations also present further complexity in determining when and how to rotate or otherwise manipulate the camera relative to the training environment.

[0255] Figures 18A–18E illustrate exemplary camera navigation training for simulated angled laparoscopy provided by a camera navigation system. Specifically, these figures show at least two different camera navigation training exercises: tube targeting and star tracking.

[0256] As shown in Figure 18A, a menu (1800) is provided that includes a tutorial option (1802) and training tailored to the “angled” laparoscope (1804, 1806). The tutorial option (1802) allows the user to practice how to operate the “angled” laparoscope in a training environment and see how these movements are translated into a cursor (405) in the digital environment (410). Because the angled laparoscope is complex, the tutorial option (1802) provided by the camera navigation system offers a scenario in which the user uses only one operating point to see how the view of the point where image data is acquired changes in the digital environment as the angled laparoscope changes.

[0257] In various embodiments, the position tracking of a simulated angled laparoscope relative to the training environment performed by the camera navigation system differs from that of a zero-degree laparoscope. This is because the user needs to not only rotate the camera that captures the image data, but also introduce further rotations to correct the view, which affects where the image is being captured. The following describes embodiments of a simulated angled laparoscope. This description can be done via a tutorial option (1802) to allow the user to become familiar with the operation of the simulated angled scope.

[0258] In various embodiments, the simulated angled laparoscope has two operating points that need to be considered in order to correctly identify its position within the training environment. Firstly, the simulated angled laparoscope operates in different ways by allowing the user to "look around" for objects or obstacles that would otherwise be difficult to see. To facilitate the operation of the simulated angled laparoscope within the training environment, the first operating point concerns the rotation of the camera. In various embodiments, the camera rotation is performed at the distal end of the simulated angled laparoscope (i.e., the part of the simulated angled laparoscope closest to the handle). The user operating the simulated angled laparoscope utilizes the camera rotation to ensure that the image being acquired is properly oriented (e.g., in the correct posture).

[0259] The difference from a zero-degree laparoscope lies in tracking the physical rotation of the simulated laparoscope (i.e., the entire instrument). In various embodiments, the simulated angled laparoscope is equipped with a rotation sensor to monitor and acquire the physical rotation angle of the simulated angled laparoscope. In various embodiments, the camera also rotates by a corresponding amount along with the physical rotation of the angled portion of the simulated angled laparoscope. To maintain a stable image while the angled portion of the simulated angled laparoscope is rotating, the camera needs to be rotated in the opposite direction by a corresponding amount to compensate for the introduced rotation. The rotation of the angled portion of the simulated angled laparoscope affects how the horizontal plane is presented in the image data. In various embodiments, it is desirable that the horizontal plane is arranged in the left-right direction at the center of the image data. In various embodiments, the horizontal plane is defined at the bottom of the camera or the image sensor.

[0260] Unlike the embodiment of the simulated zero-degree laparoscope, the embodiment of the simulated angled laparoscope utilizes not only the image data of markers captured by the camera but also the rotation angle measured by the rotation sensor. The camera navigation system uses both pieces of information to determine the 6-degree-of-freedom position information of the simulated angled laparoscope relative to the training environment, and how to display the cursor in the digital environment. Specifically, the angle value acquired by the rotation sensor is added to the pitch value acquired for the simulated laparoscope via the PnP process. Furthermore, the digital environment is generated to rotate the digital environment by a corresponding amount based on, for example, the angle measured by the rotation sensor, so as to provide a specific viewpoint of the digital environment as viewed from the viewpoint of the cursor.

[0261] In various embodiments, the camera navigation system is configured to be usable with any number of different simulated laparoscopes. The camera navigation system can include specific simulated laparoscopes (e.g., zero-degree and angled) that can be used to participate in one or more camera navigation trainings, while other (e.g., third-party) simulated laparoscopes are also compatible with or usable with the camera navigation systems described herein.

[0262] In various embodiments, a camera navigation system is configured to identify the type of a connected simulated laparoscope. In various embodiments, identification information of different simulated laparoscopes that can be used with the camera navigation system is stored in a memory. In various embodiments, the camera navigation system is configured to compare identification information from the connected simulated laparoscope with the identification information stored in the memory, and determine whether the connected simulated laparoscope is compatible. When compatibility is confirmed, in various embodiments, associated laparoscope calibration information used to calibrate or otherwise transform information being processed by camera navigation to account for various features specifically associated with the connected simulated laparoscope is retrieved from the memory. For example, when the camera navigation system determines that the "field of view" of the connected simulated laparoscope is less than a predetermined threshold, it may perform a modification to provide an output similar to that when the "field of view" is at the predetermined threshold during calculation of the position of the simulated laparoscope. In various embodiments, the camera navigation system retrieves a corresponding camera navigation training set determined to be compatible with the connected simulated laparoscope from the memory. The camera navigation system is configured to reflect this compatible camera navigation training in a digital environment.

[0263] In embodiments where the connected dummy laparoscope is not recognized or identified as compatible, such dummy laparoscopes may not be usable with the camera navigation system. However, calibration can be performed to make the dummy laparoscope compatible with the camera navigation system. Specifically, an unrecognized dummy laparoscope can be calibrated with the camera navigation system to ensure that information captured by any unrecognized dummy laparoscope is processed to match that of other recognized or compatible dummy laparoscopes, based on the characteristics of the dummy laparoscope (e.g., field of view, zero-degree / angled, image quality). In various embodiments, such calibration may involve the camera navigation system changing or updating variables or adding different factors to align the image information of the unrecognized dummy laparoscope with that of a compatible dummy laparoscope. In various embodiments, the corresponding calibration data is stored in memory together with, or in relation to, the identification information of the newly calibrated or recognized dummy laparoscope. Thus, in future applications, the camera navigation system may be able to recognize different dummy laparoscopes and enable them to be used with various menus and camera navigation training.

[0264] Figure 18A shows a menu that includes camera navigation training for angled laparoscopy. As shown in Figure 18A, exemplary training for a simulated “angled” laparoscopy includes tube target training (1804) and star tracking training (1806). In various embodiments, further camera navigation training (e.g., the zero-degree related camera navigation training described above) may also be included in the menu (1800).

[0265] Each of the exercises described here (1804, 1806) relates to a simulated “angled” laparoscope, and the camera navigation training provided by the camera navigation system enables users to practice, train, and / or evaluate the ability to operate the simulated laparoscope and acquire image data using the “angled” laparoscope, and enables users to enhance their surgical or surgical support skills using the angled laparoscope.

[0266] In various embodiments, acquiring image data from an insert or grid using an angled laparoscope can be more complex than when using a zero-degree laparoscope, as it requires adjusting and considering the orientation of the image sensor (housed within the "angled" laparoscope) used to acquire the image, and the orientation of the plane from which the image is being acquired. In various embodiments, control of the image sensor may be related to a part of the "angled" laparoscope (e.g., a handle or light cable), while control of the orientation of the image plane may be related to a different part (e.g., a camera head). In various embodiments, in order to correctly acquire images using the simulated "angled" laparoscope, the user needs to be familiar with operating two parts of the simulated "angled" laparoscope, not only to control the rotation of the image sensor but also to maintain its orientation relative to the horizontal plane.

[0267] In a "tube targeting" training example, there is a predetermined number of targets that can be placed within a digital environment provided by a camera navigation system. In various embodiments, the placement and / or positioning of targets provided by the camera navigation system are the same regardless of the difficulty level in order to maintain a consistent baseline that can be used to quantify (e.g., rank) subsequent performance by the user and / or other users. However, at each difficulty level, the number of targets to be captured, as well as the complexity associated with manipulating from one target to subsequent targets, can increase. Figure 18B shows an exemplary menu (1810) for tube target training. The title (1816) informs the user of the specific training currently selected.

[0268] As shown in the exemplary menu (1810), the camera navigation system provides several different difficulty levels (1812) within the digital environment (405), which the user can select using a cursor (410). Each difficulty level (1812) may have an associated level image (1814) that provides hints about what the training requires, such as the number of targets that need to be acquired. To select one of the difficulty levels (1812), the user must manipulate the cursor (410) to overlap at least a portion of a particular difficulty level (1812) for a given period of time. Once selected, the camera navigation system retrieves relevant data from memory and updates the digital environment (405) with the various targets and objects required for training.

[0269] In various embodiments, the camera navigation system may initially restrict or lock different difficulty levels (1812) to prevent user access. Generally, access is granted after meeting several prerequisites determined and / or confirmed by the camera navigation system. For example, access to level 1 requires completion of a tutorial, access to level 2 requires completion of level 1 with a predetermined skill level, and access to level 3 requires completion of level 2 with a predetermined skill level.

[0270] Each difficulty level (1865) can indicate to the user how many targets (1867) need to be acquired. The ability to select higher difficulty levels in tube targeting can be based on completion of previous difficulty levels and / or achievement of specific scores / qualifications.

[0271] Furthermore, the menu for tube targeting training (1810) may also include a "Home" button (1818). The "Home" button (1818) allows the user to exit the tube targeting menu (1810) and return to the main menu to select a tutorial or other available training.

[0272] Figures 18C-1 to 18C-3 illustrate an exemplary progression of "tube targeting" training. These three figures illustrate the progression of "tube targeting" training. Specifically, the trainee manipulates a simulated "angled" laparoscope while referring to the training environment to align a cursor (410) displayed in the digital environment (405) with one of the targets (1820) displayed in the digital environment (405). Alignment between the cursor (410) and the target (1820) is achieved when it is determined that the cursor (410) is interacting with / overlapping with one of the targets (1820) displayed in the digital environment (including a designator such as a gray area (e.g., a circle) with markings (1830) that highlight the edge or perimeter). Although one type of cursor (410) is shown in the figures (and described herein), other types of cursors (410) with different shapes are also compatible with the training and can be used.

[0273] For a user to freely manipulate the cursor (410) within the digital environment (405) using an angled laparoscope, they must be proficient in operating the angled laparoscope using both control points of the simulated angled laparoscope. Using only one of the control points may cause the image being acquired to rotate (e.g., upside down), so the user must use both control points to maintain the correct horizontal plane. The user is not simply positioning the simulated laparoscope over the target (1820), but is tested on their expertise in controlling the angled laparoscope through the use of its first and second control points. Specifically, to change the cursor's viewpoint within the digital environment, the user must view (or acquire) the target (1825) in a way that avoids the viewpoint interfering with tubes (1825) or other nearby obstacles within the digital environment (405). Furthermore, both control points must be used; otherwise, the image data acquired by the angled laparoscope will appear to rotate by a certain amount and not maintain the correct orientation.

[0274] The camera navigation system can instruct the user to highlight multiple targets (1820) in the digital environment (405) using a cursor (410) while camera navigation is being performed. One or more targets (1820) may not have any obstacles (or parts of a tube (1825)) that obscure the alignment of the cursor (410) with respect to the target (1820), but there may be tubes (1825) that otherwise obscure at least a portion of the target (1820) from any viewing direction except directly above (i.e., the user's viewpoint). As shown in Figure 18C-1, an exemplary view provided by the camera navigation system shows a view in which the target (1820) is positioned within a tube (1825) as seen through an “angled” laparoscope. The walls of the tube (1825) may be partially transparent to allow visibility and / or display of the target (1820) within the tube (1825). However, correct acquisition of target (1820) using cursor (410) is achieved, or determined to be achieved, by the camera navigation system only if the camera navigation system identifies that cursor (410) surrounds the entire target (1820) without interference from the walls of tube (1825). As shown in Figure 18C-2, even if target (1820) is surrounded within cursor (410), if part of the walls of tube (1825) is also visible within cursor (410), it is not a desirable configuration (for example, based on the difficulty level of training predetermined by camera navigation training), and therefore does not correspond to proper acquisition of target (1820) as determined by the camera navigation system. Figure 18C-3 shows cursor (410) surrounding the entire target (1880) without any part of tube (1825). Specifically, although part of tube (1825) is still visible, it is located outside cursor (410).In various embodiments, proper acquisition is achieved, or the camera navigation system can determine, when only the entire target (1820) is located within the boundary defined by the cursor (410) marking (1830).

[0275] The determination of whether the cursor (410) has correctly surrounded the target (1820) can be made by training and memorizing the target (1820) and specific details about where the cursor (410) should be positioned (e.g., position, depth, viewpoint) to correctly capture the target (1820). As referred to herein, the cursor position corresponds to a set of x, y, and z coordinates that highlight where the cursor should be located in three-dimensional space. The observation distance represents how far the simulated laparoscope is from the training environment. The observation distance can be indicated by appropriately changing the size of the cursor. For example, the cursor can be made larger when the simulated laparoscope is close to the training environment. In another embodiment, the observation distance can represent the point in time when the simulated laparoscope is held away from the training environment. Accordingly, the size of the cursor can be made smaller to reflect the observation distance. The cursor viewpoint relates to the simulated direction and resulting rotation; specifically, the field of view is a simulated viewpoint in the digital environment corresponding to the positional information from the simulated laparoscope. In various embodiments, the observation distance can also be reflected by increasing or decreasing the size of the digital environment (i.e., the viewpoint of the digital environment from the cursor point) based on how close or far the simulated laparoscope is from the training environment (i.e., the observation distance).

[0276] The camera navigation system can conclude that the target (1820) is properly surrounded by the cursor (410) if it detects and determines that the cursor (410) has a position, depth, and / or viewpoint close to the values ​​assigned to a particular target (1820). In various embodiments, in order to ensure that no collisions indicating that the target is at least partially obscured are detected, collision calculations can also be performed between the positions of the cursor (410) and the tube (1825), in addition to comparing with stored details regarding the target (1820) and the cursor (410).

[0277] The camera navigation system can generate and provide a notification to the user once it identifies that the cursor (410) is properly positioned over the target (1820). As shown in Figure 18C-3, the camera navigation system can begin filling the cursor (410) in a clockwise direction (e.g., changing from a first color or shade to a second color or shade) to identify and notify the user that the cursor is in the correct position relative to the target. In various embodiments, the camera navigation system provides the cursor (410) to be used as a meter or gauge (1835) that highlights not only that the target (1820) has been correctly acquired, but also how long the target (1820) should be held within the cursor (410). The rate at which the meter (1835) changes, e.g., the rate at which it fills and / or changes color, can correspond to the time required to hold the simulated “angled” laparoscope in a specified position. Other methods exist to notify the user via other visual or audible notifications (e.g., a beep) when the cursor (410) correctly interacts with the target (1820) (e.g., the color of the cursor (410) outline changes).

[0278] The camera navigation system can display the next target (1820) once it determines that a target (1820) has been acquired for a predetermined period of time. The camera navigation system can remove previously acquired targets (1820) and tubes (1825) from the digital environment (405) to reduce or eliminate confusion regarding which target (1820) should be acquired next or thereafter during training. The camera navigation system can terminate training after all predetermined targets (1820) have been acquired. In various embodiments, feedback on the user's performance regarding training can then be provided, such as metrics of the user's performance (e.g., completion time, number of collisions, accuracy, stability).

[0279] In various embodiments, another training exercise used in conjunction with an "angled" scope provided by a camera navigation system is called "star tracking." In this training, the user is tasked with tracking and locating an object (e.g., a star-shaped target) that moves around in a digital environment and stops at a predetermined spot. As with other training exercises, there may be a menu (1860) for star tracking training in which the user can select from different difficulty levels (1855), as shown in Figure 18D. Each difficulty level (1855) has an image (1860) that provides information about that level, for example, identifying how many targets need to be acquired. Access to different levels may be restricted initially, but can be permitted after the user has met relevant requirements, such as completing a tutorial and / or completing lower difficulty levels.

[0280] Level selection is performed by using a cursor (410) within the digital environment (405), similar to the method described above for tube targeting. The star tracking menu (1850) also includes its own "home" button (1865) that allows the user to return to the main menu.

[0281] Figures 18E-1 to 18E-3 show examples of the progress of "star tracking" training. During training, one or more objects (e.g., tubes / pillars) (1870) can be placed within the digital environment (405) that are designed to obstruct the direct line of sight from the cursor (410) to the target (1880). In this embodiment, the target (1880) is star-shaped, but other different shapes are also possible. The camera navigation system provides the placement and number of objects (1870) according to the difficulty level to make it more difficult to acquire the moving target (1880). The camera navigation system provides a cursor (e.g., a circle with an outline) (410) that determines how the moving target (1880) should be correctly acquired. In various embodiments, the user is commanded to move the cursor (410) toward the target (1880) and ensure that the target (1880) is surrounded by the cursor (410).

[0282] As described above, movement in the three-dimensional space of the digital environment requires the user to be able to operate two control points of the simulated "angled" laparoscope to change the viewpoint and the object being viewed within the digital environment (405), as well as to maintain the horizontal plane (i.e., the image in the correct orientation). If only one of the two control points is used, the image captured by the camera / image sensor will be slightly rotated. When the camera navigation system determines that a target (1880) is within the cursor (410), it can provide a notification indicating whether the target (1880) is correctly positioned within the cursor (410). For example, the cursor (410) can change color (as shown in Figure 18E-3). If the cursor (410) is not correctly positioned and needs to be adjusted (e.g., moved further away or closer, rotated to the correct plane), the camera navigation system can provide an indicator or meter (1882) as shown in Figure 18E-2.

[0283] The camera navigation system can use this meter (1882) to measure the user's performance using a plurality of different zones. The meter (1882) has a marker (1886) indicating the user's performance based on different zones. Further, the meter (1882) can also have hints or observations (1888) that allow the camera navigation system to provide the user with information on how the simulated "angled" laparoscope needs to be moved to better capture the moving target (1880).

[0284] In some embodiments, when the camera navigation system determines that the target (1880) has been correctly captured, the moving target (1880) can move to the next position within the digital environment (405) over a predetermined period. The movement of the target (1880) presents the user with the task of manipulating the simulated "angled" laparoscope to a new position while navigating around one or more objects (1870). In various embodiments, hints can be included to help the user identify where the target (1880) will move. For example, an arrow can be used to highlight the destination of the target. In some embodiments (as shown in Figure 18E-1), the target (1880) can leave a line or trajectory (1875) as it moves to the next location. The line or trajectory (1875) can be visible for a predetermined period of time or until the target (1880) is captured.

[0285] The star tracking training can end after the camera navigation system determines that the moving target (1880) has been captured a predetermined number of times. In various embodiments, feedback regarding the user's performance can be provided, which quantifies the user's performance in the training (e.g., completion time, number of collisions with objects, accuracy and / or stability).

[0286] In both training exercises used with the simulated “angled” laparoscope (“tube targeting” and “star tracking”), an implementation of collision detection by the camera navigation system is used to quantify whether the target has been correctly acquired. For example, in “tube targeting”, the camera navigation system performs a collision detection check to verify that the user’s viewpoint of the target (corresponding to the cursor’s viewpoint) is directly above the target and is not otherwise obscured by various tubes in the digital environment. If the cursor is at an angle to the target, or if the tube is transparent, it is possible to see the target within the tube. However, in various embodiments, the tube is not physically present, so the camera navigation system limits or prevents such instances of target acquisition and instead determines acquisition only when the cursor is directly above. In various embodiments, collision detection is configured to identify the moment when at least a portion of the tube exists between the location of the target and the cursor’s viewpoint. In various embodiments, collision detection simulates target targeting where a physical object obstructs the user’s view, prompting the user to operate in a more appropriate position (i.e., directly above). In various embodiments, a predetermined amount of interference (calculated using collision detection) through an object (i.e., a tube) can be tolerated, which may vary depending on the difficulty level of the selected training.

[0287] In various embodiments, since the positions of the target and object are known (i.e., stored in memory associated with the selected training), the camera navigation system performs collision detection by calculating a path from the cursor's current position to the target and identifying whether there are any points along that path that intersect with known locations of tubes or other obstacles. If at least one point on the path intersects with the location of a tube or obstacle, the camera navigation system detects a collision corresponding to at least a portion of the view between the cursor and the target being obstructed by the tube. In various embodiments, the camera navigation system may tolerate a number of collisions below a threshold based on the difficulty level of the training being performed.

[0288] In “star tracking” training, collision detection is used to determine whether the user’s view of the star-shaped target is obstructed (within a specified tolerance) by various objects in the digital environment (e.g., tubes). Similar to tube targeting, the camera navigation system checks the known positions of various objects between the cursor’s path and the star-shaped object. If at least one object is located on the path, it can indicate that the view is at least partially obstructed. In various embodiments, the user can be prompted to manipulate a simulated “angled” laparoscope to different angles or positions to correctly “see” around an object, or to track different positions and / or orientations to obtain different views of the star-shaped target that are not obstructed by at least one object (e.g., tube).

[0289] As mentioned above, after completing training related to the simulated angled laparoscope, the camera navigation system provides feedback. For example, it provides an evaluation of the performance of one and / or multiple users through a score calculated based on various criteria, such as the time taken to complete the training. Other criteria are also possible and are assumed to be suitable for characterizing and highlighting areas of improvement for the users.

[0290] In each of the different training sessions provided by the camera navigation system, the way the system obtains feedback from performance data determined by the system to characterize performance during training differs and can be customized as appropriate. For example, in various embodiments, feedback on the rotation and observation distance of the simulated laparoscope may be emphasized or given greater weight because these skills have a significant impact on the quality of camera navigation during actual surgical procedures. In other embodiments, speed may be more important, or it may not be given much importance. While completing training quickly is a sign of proficiency, completion should not be achieved at the expense of scores in other skills such as roll, observation distance, and accuracy. In various embodiments, these criteria may be predetermined by the camera navigation system and / or adjusted based on other criteria such as the user's past experience and / or criteria predetermined or set by the instructor and / or evaluator. For example, criteria for practicing surgeons may differ from those for students.

[0291] In various embodiments, feedback for the different training described above can be provided on a leaderboard (see Figure 9). The leaderboard displays feedback (i.e., scores) from a specific user, as well as feedback (i.e., scores) from various different users. How different user performances, or possibly the performance of a group of users, are ranked for the same training can be based on a specific order or weighting of elements. For example, in one embodiment, the leaderboard may give more weight to camera rotation performance than to observation distance scores. Other elements such as accuracy and time can then be used (with lower importance) in the ranking. Different scores can be stored locally, for example, in a database, to track user data even when the instructor is absent, and to retrieve and provide feedback on progress over a period of time. In various embodiments, performance data and feedback can be stored remotely (e.g., in the cloud) so that users in physically different locations can compare their performance with other users in different locations.

[0292] Figure 29 shows an exemplary embodiment of the camera navigation system. Specifically, the figure shows an example flowchart outlining the steps or actions that the camera navigation system can perform in connection with any one of the training exercises used in conjunction with training related to simulated laparoscopy. Note that this figure shows a general overview of the steps or actions, and more or fewer steps or actions may be used, the order may be changed, and / or they may be performed sequentially and / or in parallel as needed.

[0293] Once the user connects and inserts the simulated laparoscope (e.g., zero-degree or angled) they wish to practice with, the camera navigation system can identify the corresponding practice set and display it via a menu (e.g., Figure 4A or Figure 18A) (2910). The user can then select one of the training exercises and the corresponding difficulty level. The camera navigation system retrieves relevant data for the digital environment and features (e.g., targets, obstacles, paths) that need to be rendered for the training.

[0294] When training begins, data is acquired from a simulated laparoscope in the training environment. In various embodiments, image data is acquired from the simulated laparoscope in the training environment. Next, the computer vision portion of the camera navigation system (e.g., a scope view generator) processes the markers incorporated into the image data (via the computer vision process) to determine information about the position of the simulated laparoscope while referring to the training environment (2920).

[0295] The camera navigation system generates a digital environment corresponding to the selected training (2930). Depending on the training being performed, corresponding features such as markers, obstacles, and paths are retrieved from memory and generated within the digital environment. Using previously acquired positional information of the simulated laparoscope, a viewpoint of the corresponding digital environment is provided. This corresponding viewpoint (e.g., the viewpoint of the digital environment from the laparoscope viewpoint) is generated by the scope view generator and displayed on the monitor for the user to view. In various embodiments, the size / dimensions of the digital environment have a one-to-one correspondence with the training environment.

[0296] Once the position of the simulated laparoscope from the training environment is transmitted as a cursor in the digital environment, in various embodiments, a camera navigation system (e.g., a scope view generator) processes data related to the selected training and the cursor's position in the digital environment. In various embodiments, the training of the camera navigation system utilizes information from the digital environment and stored information related to the selected training. These systems can determine whether a particular path is being followed, whether various obstacles are being encountered, or whether the target is being correctly acquired. The system uses stored information about the features of the digital environment (e.g., paths, markers, obstacles) and the position information of the simulated laparoscope in the digital environment. The camera navigation system is configured to provide different training and difficulty levels to match different sets of information to be used. For example, if the cursor's position has the same position information (e.g., coordinates) as the target, the camera navigation system can use this position information to determine and / or indicate that the target has been acquired. In another example, if the cursor's position is within a predetermined position range, the camera navigation system can use this position to determine and / or indicate that the target is within a predetermined path.

[0297] The cursor position can be updated at predetermined intervals by monitoring / tracking updates to the position of the simulated laparoscope relative to the training environment (2950). For example, in various embodiments, the position can be updated every second (e.g., 60 times per second). As a result, the updated position of the simulated laparoscope can be processed and transferred to the digital environment (2940). Subsequently, any further subsequent processing related to the training in progress can be performed using stored information related to the training (e.g., targets, object positions). For example, updating the user's progress along a path or determining whether a target has been captured uses a comparison of stored information about the cursor position in the digital environment and related features of the training in progress (e.g., paths, markers, obstacles). In various embodiments, a predetermined set of coordinates for the cursor position can be stored for each training session corresponding to a position where the cursor needs to be located in order to correctly capture the target (considering framing aspects such as the target being in the correct position, orientation, and depth). Thus, a comparison of the cursor's current position with the stored position can be used to determine whether the target is properly framed or captured by the cursor.

[0298] The camera navigation system can continue updating and processing the position of the simulated laparoscope according to the various embodiments described above as long as training is being performed. When the user completes the training, the system can automatically terminate the training (2960). In various embodiments, the camera navigation system can provide the user with the option to repeat the training or select a different training.

[0299] Upon completion, the camera navigation system can provide the user with relevant feedback based on the user's performance during training (2970). The feedback is based on data acquired within the digital environment during training. For example, in various embodiments, the accuracy of tracking the user's performance may be based on the number of times the position of the simulated laparoscope was determined to be within a predetermined location related to the path. For example, in various embodiments, collisions may be tracked based on the frequency with which the position of the simulated laparoscope corresponds to the position of one or more objects in the digital environment.

[0300] In various embodiments, dedicated applications provided to and / or associated with a camera navigation system that provides camera navigation training can be written in any number of computing languages ​​(e.g., C++) so that they can run natively on any desired combination of a processor, computer device, and / or operating system. Furthermore, the applications can also be written using web-based languages ​​(e.g., WebAssembly) so that they can run from a browser or the cloud / web.

[0301] In web-based applications, access to and control of relevant hardware (such as cameras, rotary encoders, and GPUs) can be performed through a web browser. A specific application / program can be written to access the relevant hardware using APIs or other routines and provided along with the camera navigation system. The data is then communicated to the web-based application, along with access to the hardware, to generate and render a corresponding user interface and images displayed on a monitor for viewing and interaction. According to various embodiments, the camera navigation system provides the aforementioned training, user interface, feedback, elements, images, and / or metrics via one or more local or remote processors and / or computer devices (and, in various embodiments, via applications, programs, and / or libraries) configured to monitor and / or track a simulated laparoscope, evaluate data, interactions, and / or generate inserts or grids, surgical environments, and interactions with the simulated laparoscope, as well as providing, generating, displaying, updating, operating, and / or evaluating the training, user interface, feedback, elements, images, and / or metrics as described above.

[0302] As described above, the camera navigation system may include a surgical trainer and / or body form related to the training environment. In various embodiments, the body form may be a surgical trainer configured to simulate a patient's torso. In various embodiments, the body form or surgical trainer may be configured to receive inserts or grids (as described above) to simulate conditions related to laparoscopic surgery performed in a patient. An exemplary surgical trainer is shown in Figure 30. This figure shows surgical trainer 3010 as an exemplary embodiment having a top perspective view.

[0303] As shown in the figure, the surgical trainer 3010 is configured to simulate conditions associated with laparoscopic procedures, such as the patient's torso, abdomen, pelvis, and / or other body parts. One feature that facilitates the simulation of conditions associated with laparoscopic procedures is that the surgical trainer 3010 can be positioned to obstruct the direct view of the insert or grid being practiced, which is housed within the surgical trainer 3010.

[0304] Continuing to refer to Figure 30, the surgical trainer 3010 provides a body cavity 3012 that is substantially obstructed from direct view. The body cavity is configured to receive inserts or grids, etc., as described in the present invention. In some embodiments, the body cavity 3012 can be accessed through a tissue simulation area 3014 through which surgical instruments (e.g., laparoscopes) placed within the body cavity 3012 penetrate, for the purpose of practicing surgical techniques (i.e., interaction with inserts or grids). In various embodiments, the body cavity 3012 can also be accessed via a hand-assisted access device or a single-site port device, which are employed instead to access the body cavity 3012. In various embodiments, the body cavity 3012 can be accessed via both the tissue simulation area 3014 and the hand-assisted access device or single-site port device. In various embodiments, the body cavity 3012 can be accessed via adapters or openings, etc., attached to or integrated with the surgical trainer. An exemplary surgical training device is described in U.S. Patent Application No. 13 / 248,449, filed on September 29, 2011, entitled “Portable Laparoscopic Trainer,” which is incorporated herein by reference in its entirety.

[0305] To obstruct the view of the body cavity 3012 from direct vision, the surgical trainer 3010 is designed to have a top cover 3016 spaced apart from the base 3020 and connected to the base 3020 via at least one leg 3020. In various embodiments, the surgical trainer 3010 may have one or more legs 3020. The surgical trainer 3010 is configured to simulate laparoscopic conditions in which the body cavity 3012 is obstructed from direct vision, using the top cover 16, the base 3020, and at least one leg 3020. Such laparoscopic conditions can correspond to procedures in which a surgeon operates on tissues or organs located inside a patient (e.g., within a body cavity), such as in the abdominal region. Thus, the surgical trainer 3010 is a useful tool for instructing, practicing, and demonstrating surgical procedures using relevant surgical instruments by simulating a patient undergoing a surgical procedure.

[0306] As described above, surgical instruments are inserted into the body cavity 3012 through one or more tissue simulation areas 3014, and also through a pre-established opening 3022 via a hand-assisted access device or single-site port device located on the upper cover 3016 of the surgical trainer 3010. The opening can be pre-formed in the top cover 3016, but further simulation of surgical procedures can also be enabled by penetrating the top cover 3016 using various surgical instruments and techniques to access the body cavity 3012. Within the body cavity 3012, a simulated laparoscope / camera can be used to interact with inserts or grids located within the body cavity 3012 between the top cover 3016 and the base 3018.

[0307] Continuing to refer to Figure 30, in various embodiments, the insert or grid may be a separate component, but to ensure that the insert or grid does not move during use of the surgical trainer 3010, the insert or grid is fixed below one or both of the tissue simulation area 3014 or the aperture 3022 located within the top cover. To secure the insert or grid housed in the body cavity 3012, the base 3018 may be designed to have a receiving area 3024 or tray configured to step-by-step or fix the insert or grid in place within the surgical trainer 3010. In various embodiments, the receiving area 3024 of the base 3018 may include mounting elements that hold the insert or grid in place. The mounting elements work in conjunction with at least a portion of the insert or grid to prevent movement or displacement of the insert or grid during use of the surgical trainer 3010. In various embodiments, the mounting elements are configured to accept multiple different inserts or grids so that the insert or grid is removable and interchangeable with other inserts or grids.

[0308] Other means for securing the insert or grid within the body cavity 3012 are also conceivable. For example, the insert or grid can be secured to the base 3018 via the use of a patch of hook-and-loop type fastening material, such as VELCRO®, which allows the insert or grid to be removably connected to the base 3018. In other embodiments, other mounting methods that do not provide a removable connection between the base 3018 and the insert or grid can also be utilized. For example, an adhesive can be used to provide an easily removable, non-removable further connection between the base 3018 and the insert or grid.

[0309] In various embodiments, a video display monitor 3028 is provided with the surgical trainer 3010. For example, the video display monitor 3028 is hinged to the top cover 3016 and can have at least two different states: a closed state in which the video display monitor 3028 is hidden and an open state in which the video display monitor 3028 is visible. In various embodiments, the video display monitor 3028 can be separated from the top cover 3016 but can still be communicated with the surgical trainer 3010.

[0310] In various embodiments, the video display monitor 3028 is communicably connected to various visual systems that deliver images to the video display monitor 3028. For example, a laparoscope inserted through one of the pre-established apertures 3022, or an image acquisition device (e.g., a webcam) placed in the body cavity 3012, can be configured to capture images of a simulated surgery in progress and transfer the captured images to the video display monitor 3028 and / or other computer devices (e.g., desktop, mobile devices) so that images of areas within the surgical trainer 3010 can be viewed. In various embodiments, other devices (e.g., microphones, sensors) can also be used with the surgical trainer 3010 to capture other types of data, such as acoustic data, which can be displayed on the video display monitor 3028 in combination with the visual data.

[0311] The surgical trainer 3010 can be configured to receive portable memory storage devices such as a flash drive, a smartphone, a digital audio or video player, or other digital mobile devices that further facilitate the recording of simulated surgical procedures for demonstration purposes and / or playback of data acquired from the surgical trainer 3010 onto a monitor. In various embodiments, the surgical trainer 3010 can also be connected to an additional or another (e.g., larger) audiovisual device that can be used to display audiovisual data acquired from the surgical trainer 3010. In various embodiments, the surgical trainer 3010 can be communicated (e.g., wired or wirelessly) to a different computer device (e.g., a desktop, laptop, or mobile device) configured to receive data acquired from the surgical trainer 3010 and display the data so that others can view it. Such embodiments can be useful in variations of the surgical trainer 3010 that do not include the video display monitor 3028.

[0312] As shown in Figure 30, the top cover 3016 is generally positioned directly over the base 3018, with one or more legs 3020 positioned substantially along the periphery. The legs 3020 interconnect the top cover 3016 and the base 3018. In embodiments where two or more legs 3020 are present, each leg is equidistant from one another and can function as a structural support, holding the top cover 3016 in place above the base 3018. In various embodiments, the top cover 3016 and the base 3018 are substantially the same shape and size and have substantially the same peripheral contour. In various embodiments, this shape can correspond to human body structures such as the torso / abdominal region of a patient.

[0313] In various embodiments, depending on the arrangement of the top cover 3016, base 3018, and one or more legs 3020, the body cavity 3012 can be partially or entirely obstructed from direct view. In some variations, the legs 3020 may include openings that allow the body cavity 3012 to be illuminated by ambient light and contribute to the overall weight reduction of the surgical trainer 3010. Apertures associated with the legs 3020 can also allow the surgical trainer device 3010 to view and / or access the body cavity 3012.

[0314] In various embodiments, the upper cover 3016 is detachable from one or more legs 3020. Also, in various embodiments, each leg is detachable or foldable relative to the base 3018. These features allow the surgical trainer 3010 to be converted into a low-profile, portable form.

[0315] Various embodiments provide a camera navigation system. The camera navigation system includes a training environment, a simulated laparoscope configured to capture one or more images from the training environment, a scope view generator that determines the position information of the simulated laparoscope relative to the training environment from the captured one or more images to generate a digital environment, and a monitor configured to receive and display the digital environment from the scope view generator.

[0316] In various embodiments, the camera navigation system is also configured to generate computer-generated elements to be incorporated with the digital environment. These computer-generated elements are configured for menus and / or camera navigation training.

[0317] In various embodiments, the camera navigation system is also configured to generate supplemental graphic elements, and the augmented image data generated by the scope view generator is superimposed on at least a portion of the image data acquired from the simulated laparoscope using these generated supplemental graphic elements.

[0318] In various embodiments, the camera navigation system also includes a surgical trainer configured to simulate the patient's torso, the surgical trainer having a top cover spaced apart from a base that defines an internal cavity.

[0319] In various embodiments, the internal cavity of the training environment also includes an insert or grid, which includes a plurality of markers arranged on a flat or planar sheet.

[0320] In various embodiments, the insert or grid is positioned on the base of the surgical trainer. In various embodiments, the insert or grid is positioned on one or more of the side walls, including the ceiling and / or anterior and / or posterior walls, of the surgical trainer. In various embodiments, the insert or grid is positioned on one or more objects or obstacles located within the internal cavity of the surgical trainer.

[0321] In various embodiments, the digital environment corresponds to or includes one or more simulated surgical training sessions. In various embodiments, one or more simulated surgical training sessions include follow training in which a simulated laparoscope is instructed to follow a path, track training in which a simulated laparoscope is instructed to follow a moving target, and / or framing training in which a simulated laparoscope is instructed to overlap one or more targets.

[0322] In various embodiments, the scope view generator further monitors the movement of the simulated laparoscope within the training environment and updates the digital environment to include a cursor, where the cursor's position on the monitor corresponds to the position of the simulated laparoscope relative to the training environment.

[0323] In various embodiments, the scope view generator further monitors the position of the simulated laparoscope relative to the training environment, determines that the cursor position overlaps with at least one computer-generated element related to the digital environment, and updates the digital environment with different computer-generated elements based on the at least one computer-generated element over which the cursor overlaps.

[0324] In various embodiments, determining the position and orientation of the simulated laparoscope within the training environment includes identifying two or three or more adjacent markers found in one or more captured images.

[0325] In various embodiments, the scope view generator further evaluates the performance of one or more simulated surgical training sessions and generates feedback based on the evaluated performance. In various embodiments, the generated feedback includes generating a leaderboard containing the evaluated performance of multiple different users. In various embodiments, the generated feedback includes visual indicators that provide instructions to complete one or more simulated surgical training sessions.

[0326] In various embodiments, some or all of the scope view generator is implemented remotely from the training environment, and remote implementations include the scope view generator running on a cloud-based server or a remote server.

[0327] In various embodiments, the simulated laparoscope simulates a zero-degree laparoscope or an angled laparoscope. In various embodiments, the camera navigation system includes a plurality of markers and a scope view generator configured to determine the position of the simulated laparoscope using a subset of the plurality of markers. In various embodiments, the system, such as the camera navigation system, includes a plurality of markers and a view generator, such as a scope view generator, configured to generate a digital environment using the plurality of markers. In various embodiments, the camera navigation system includes a scope view generator configured to determine the position of the simulated laparoscope and / or generate a digital environment using image data captured by the simulated laparoscope. In various embodiments, the system, such as the camera navigation system, includes a view generator, such as a scope view generator, configured to receive and / or retrieve image data, determine the position of a device such as a simulated laparoscope, and / or generate a digital environment using the image data.

[0328] Various embodiments provide a camera navigation system. The camera navigation system includes a plurality of markers arranged on one or more planes, a simulated laparoscope having a camera and configured to capture images including two or more of the plurality of markers, a monitor, and a scope view generator. The scope view generator is configured to determine the scope view of the simulated laparoscope based on the captured images, generate elements based on the scope view of the simulated laparoscope, and / or provide the monitor with a digital environment that replaces and / or overlaps at least a portion of the images captured from the simulated laparoscope.

[0329] The above description is intended to enable those skilled in the art to fabricate and use surgical apparatus and to perform the methods described herein, and describes the best mode considered by inventors practicing the present invention. However, various modifications will remain obvious to those skilled in the art. These modifications are intended to be included within the scope of this disclosure. Furthermore, different embodiments or aspects of such embodiments can be illustrated and described in various figures throughout this specification. However, each embodiment and aspect thereof illustrated or described individually may be combined with one or more other embodiments and aspects thereof unless otherwise explicitly stated. The lack of explicit mention of each combination is solely for the sake of readability of this specification. Furthermore, the embodiments of the present invention should be considered in all respects as illustrative and not limiting.

Claims

1. It is a camera navigation system, Simulated laparoscopy and, A training environment including a plurality of markers configured to track the position of the simulated laparoscope, A computer device, A digital environment is generated that includes multiple computer-generated elements corresponding to the aforementioned training environment. By processing image data from the simulated laparoscope including one or more markers from the plurality of markers, the current position of the simulated laparoscope relative to the training environment is tracked, and the tracking is repeated after a predetermined period of time. The positional information of the simulated laparoscope is calculated from the image data, The digital environment is updated using the calculated position information of the simulated laparoscope, which is represented at least partially using a cursor. The user performance of camera navigation training is monitored by comparing the current position information of the simulated laparoscope with the stored information of one or more of the multiple computer-generated elements in the digital environment. Based on the monitored user performance of the camera navigation training, feedback is generated. A computer device configured as follows, A monitor configured to display the digital environment and the generated feedback, A camera navigation system equipped with a camera.

2. Each of the plurality of markers in the training environment is unique to one another, the plurality of markers are arranged in a checkerboard pattern, and / or the plurality of markers are QR codes. The system according to claim 2.

3. Tracking the current position of the simulated laparoscope includes identifying one or more of the plurality of markers of the training environment from the image data acquired by the simulated laparoscope, the identification being performed using computer vision, and / or the tracking characterizing the current position of the simulated laparoscope using six degrees of freedom. The system according to claim 1.

4. Information relating to each of the plurality of markers is stored in memory, and the information relating to each of the plurality of markers includes the coordinates of each of the plurality of markers, configured to identify the position of each of the markers relative to the training environment. The system according to claim 3.

5. The aforementioned digital environment is updated 60 times per second. The system according to claim 1.

6. The update of the digital environment includes skipping the current calculation and using the next image data if the current position of the simulated laparoscope is not updated within a predetermined period. The system according to claim 1.

7. The aforementioned digital environment update is Identifying that the position of the cursor in the digital environment overlaps with the position of one of the multiple computer-generated elements in the digital environment, one of which is a virtual button corresponding to the camera navigation training. Retrieving one or more computer-generated elements related to the camera navigation training stored in memory, Updating the digital environment configured to perform the aforementioned camera navigation training with the one or more computer-generated elements, The system according to claim 6, including the system described in claim 6.

8. Information relating to each of the plurality of computer-generated elements related to the camera navigation training is stored in memory, and / or the information relating to one or more of the plurality of elements related to the camera navigation training is retrieved when the camera navigation training is selected by the user. The system according to claim 1.

9. The method further includes generating a single computer-generated element which is a meter based on a comparison of the current position of the simulated laparoscope with the position of one or more computer-generated elements among a plurality of computer-generated elements related to the camera navigation training, and / or the meter is configured to quantify the user's current performance in the camera navigation training. The system according to claim 1.

10. Further includes generating findings or hints configured to give the user instructions on how to improve the current performance, The system according to claim 9.

11. A device for tracking the position of a simulated laparoscope, Simulated laparoscopy and, A training environment including an insert or grid containing multiple unique markers, A scope view generator configured to determine the position of the simulated laparoscope, The scope view generator is equipped with, A memory storage device configured to store the location of each of the aforementioned multiple unique markers, An executable application, From the image data acquired by the simulated laparoscope, the plurality of unique markers are identified. Based on a subset of the plurality of unique markers identified from the image data, the positional information of the simulated laparoscope relative to the training environment is determined. An executable application configured in such a way, This includes and / or the scope view generator, A digital environment corresponding to the aforementioned training environment is generated, To generate computer-generated elements for integration into the aforementioned digital environment, The apparatus is configured such that one of the computer-generated elements is a cursor having an arrangement in the digital environment corresponding to the position information of the simulated laparoscope relative to the training environment.

12. Each of the aforementioned multiple unique markers is a QR code. The apparatus according to claim 11.

13. The memory storage device is related to remote server or cloud-based storage. The apparatus according to claim 11.

14. The memory storage device is configured to store a set of coordinates for each of the plurality of unique markers, which is used to specifically identify the location of each of the plurality of unique markers, and the set of coordinates is based on a predetermined reference point. The apparatus according to claim 12.

15. It is necessary to extract and identify at least four unique markers from the aforementioned multiple markers from the image data. The apparatus according to claim 11.

16. The executable application converts the image data into a grayscale binary image, determines which part of the grayscale binary image is "on" for each pixel, and identifies the plurality of unique markers by filtering the grayscale binary image to include only the parts having a predetermined shape corresponding to the shapes of the plurality of unique markers stored in memory. The apparatus according to claim 11.

17. Identifying the plurality of unique markers includes removing distortion from the image data by calculating the corresponding translation, rotation, scaling, or skewing of the image data, wherein the removal of distortion is configured to transform the images of the plurality of unique markers in the image data to correspond to a predetermined shape associated with each of the plurality of unique markers stored in memory. The apparatus according to claim 11.

18. The computer-generated elements further include elements specific to camera navigation training, the characteristics of each of the camera navigation training-specific elements are stored in memory and retrieved when the user selects the camera navigation training. The apparatus according to claim 11.

19. The computer-generated element includes a meter near the cursor, the meter is configured to quantify the current performance of the camera navigation training and to provide instructions on how to improve the current performance of the camera navigation training. The apparatus according to claim 18.

20. The position of the cursor in the aforementioned digital environment is updated 60 times per second. The apparatus according to claim 11.

21. An insert or grid comprising a plurality of markers, each of the plurality of markers being unique from all other markers of the plurality of markers, each of the plurality of markers being arranged in a predetermined pattern over a predetermined space, the plurality of markers being configured to be imaged by a camera, the image data being configured to be processed to determine the position of a simulated laparoscope within the predetermined space, the insert or grid being removable, and information regarding the position of the markers on the insert or grid being stored in memory and retrieveable to determine the position of the simulated laparoscope.

22. It is a camera navigation system, A simulated angled laparoscope including a camera and rotation sensor, A training environment comprising multiple markers, wherein the multiple markers are configured to track the position of the simulated angled laparoscope relative to the training environment, Scope view generator and, The scope view generator is equipped with, A digital environment corresponding to the aforementioned training environment is generated, which includes multiple elements related to camera navigation training selected by the user. Image data of one or more of the aforementioned plurality of markers, The rotation angle of the simulated laparoscope relative to the horizontal plane, measured via the rotation sensor, Based on the combination, the current position of the simulated laparoscope is tracked by referring to the training environment, Using the acquired image data, the position information of the simulated laparoscope is calculated. The digital environment is updated using the position information of the simulated laparoscope, and the tracked position is indicated by a cursor within the digital environment. The user performance of the camera navigation training is monitored by comparing the current position of the cursor with one or more of the multiple elements related to the camera navigation training. Determine whether a collision has been detected between the cursor and a predetermined computer-generated element from the plurality of elements. The system is configured such that the predetermined element is a target, and the plurality of computer-generated elements include at least one tube configured to obstruct the predetermined element from the cursor. Camera navigation system.

23. The tracked position is characterized by the x, y, and z coordinate set and roll, pitch, and yaw. The camera navigation system according to claim 22.

24. The tracked position then rotates further by an amount corresponding to the measured rotation angle. The camera navigation system according to claim 23.

25. The cursor is represented in the digital environment as a gray circle with markings that highlight its outer edge. The camera navigation system according to claim 22.

26. The cursor is configured to change color or fill up in order to track the time required to capture the target. The camera navigation system according to claim 25.

27. The determined collision between the cursor and the predetermined element is configured to inform the user that a view of the predetermined element has been detected that is obstructed from the cursor's viewpoint. The camera navigation system according to claim 22.

28. The collision is determined by calculating the path from the current position of the cursor to the predetermined element and identifying whether there is a point on the path where it intersects with one or more of the multiple elements. The camera navigation system according to claim 22.

29. The calculation of the path and the identification of points along the path are performed by comparing the information with the current position of the cursor using information about the predetermined element and the plurality of elements stored in memory. The camera navigation system according to claim 28.

30. Collisions of predetermined threshold amounts along the aforementioned path are permitted. The camera navigation system according to claim 29.

31. A device for tracking the position of a simulated angled laparoscope, A simulated angled laparoscope, comprising a camera configured to acquire image data and a sensor configured to detect the rotation of the simulated angled laparoscope, A training environment including an insert or grid containing multiple unique markers, A scope view generator configured to identify the position of the simulated angled laparoscope, The scope view generator is equipped with, A memory storage device configured to store each of the plurality of unique markers along with their corresponding locations within the training environment, From the image data acquired by the simulated angled laparoscope, the plurality of unique markers are identified. Based on the image data captured by the camera and the rotation captured by the sensor, the position of the simulated angled laparoscope within the training environment is determined. An executable application configured in such a way, The scope view generator includes, A digital environment corresponding to the aforementioned training environment is generated, To generate computer-generated elements for integration into the aforementioned digital environment, The apparatus is configured such that one of the computer-generated elements is a cursor having a position in the digital environment corresponding to the position of the simulated angled laparoscope in the training environment based on the captured image data from the simulated angled laparoscope, the digital environment being further oriented by an amount corresponding to the rotation captured by the sensor.

32. The orientation of the digital environment indicates that the simulated angled laparoscope is aligned with the horizontal plane corresponding to the insert or grid by having a preferred rotation of zero degrees. The apparatus according to claim 31.

33. The computer device is configured to quantify the user performance related to the rotation of the simulated angled laparoscope, and the user performance is characterized as "good" when the rotation is between a preferred 0 degrees and 10 degrees. The apparatus according to claim 32.

34. The threshold configured to characterize the user performance related to the rotation of the simulated angled laparoscope is based on the difficulty level of the selected training or the training being selected. The apparatus according to claim 33.

35. One of the computer-generated elements is a target configured to move to different locations within the digital environment, and the target provides the user with hints about its current location by further including a visibly configured trajectory. The apparatus according to claim 31.

36. The aforementioned trajectory is visible over a predetermined period of time. The apparatus according to claim 35.

37. The aforementioned trajectory is visible until the image of the target is captured by the simulated angled laparoscope. The apparatus according to claim 35.

38. An insert or grid containing multiple markers, wherein each of the multiple markers is not identical to or different from all the other markers in the multiple. Insert or grid.

39. Multiple markers, A view generator configured to determine positional information from image data incorporating a subset of the aforementioned multiple markers, A tracking system equipped with [the following features].

Images