System and method for real-time intraprocedural endoscope shaft movement tracking

Abstract

To provide an endoscope tracker for measuring the movement of an endoscope by using a pair of trackballs and two video cameras.SOLUTION: The trackball measures changes in intraprocedural position and rotation of the scope in real time. The camera detects the scale markings on a surface of the scope and corrects accumulated trackball measurement errors. A dual-modality tracking method measures the movement of the insertion tube of the endoscope in real time, including insertion length, rotation angle, and speed. An optical trackball measures the motion of the endoscope insertion tube and a camera corrects for accumulated errors. Purely optical endoscope trackers are also provided that measure the movement of the endoscope using a video camera and software that processes images acquired from the video camera to determine how the endoscope moves over time.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Various embodiments are described herein generally relating to systems and methods for medical imaging diagnostics, and more particularly, to real-time in-procedure endoscope shaft motion tracking, such as insertion length and orientation tracking. [Background technology]

[0002] The following paragraphs are provided as background to the present disclosure, but they are not intended to be an admission that anything discussed herein is prior art or part of the knowledge of those skilled in the art.

[0003] Gastrointestinal endoscopy is the gold standard for detecting and characterizing abnormalities in the digestive tract. For example, upper endoscopy (esophagogastroduodenoscopy: EGD) is used to evaluate precancerous lesions such as Barrett's esophagus and esophageal cancer, while colonoscopy is used to detect precancerous polyps and cancer in the large intestine. In the United States, Barrett's esophagus is present in 1-2% of the adult population, and colorectal cancer is the third most commonly diagnosed cancer in both men and women, with an estimated 150,000 new cases and 53,000 deaths in 2021 [1], [2]. Current endoscopic practices are excellent at detecting cancer and precancerous lesions, but are unreliable at localizing lesions [3]-[7]. This poses problems for surveillance of Barrett's esophagus and colonic polyps, and for resection of esophageal and colonic cancers. Barrett's esophagus lesions and colonic polyps must be localized during endoscopic resection so that the same site can be checked for recurrent lesions during a subsequent endoscopic procedure. Esophageal and colorectal tumors must be precisely located by preoperative endoscopy to facilitate planning of surgical removal.

[0004] One of the challenges in endoscopy is to improve the accuracy of lesion localization during preoperative endoscopy using endoscopic motion trackers. A retrospective study (2013–2016) found that preoperative colonoscopy was inaccurate in 16.7% of patients [3]. In 15 high-quality studies, colonoscopy localized tumors inaccurately in 13.7 ± 3.6% (95% CI) of patients, and colonoscopic tattooing was inaccurate in 6.5 ± 3.4% [4]. Moreover, in other studies, incorrect localization required changes in the surgical plan in 5.3–11% of tumors [5][7].

[0005] Magnetic endoscopic imaging (MEI) currently exists that can detect loops that form along the endoscope insertion tube or shaft during colonoscopy. MEI may help improve localization accuracy, but this requires new endoscopes or accessories with magnetic coils that support motion tracking and pose estimation [4], [8], [9]. Furthermore, MEI cannot localize lesions during upper endoscopy and is not precise enough to characterize the extent or location of lesions in Barrett's esophagus.

[0006] What is needed is a system and method that addresses the above challenges and / or shortcomings through real-time, in-procedure endoscope shaft motion tracking. Summary of the Invention

[0007] In accordance with the teachings herein, various embodiments of systems and methods for real-time, in-procedure endoscope shaft motion tracking, and computer products for use therewith, are provided.

[0008] According to one aspect of the invention, a real-time endoscope tracker system is disclosed that includes a sensor cuff having a housing, a first trackball disposed within a first side of the housing and a second trackball disposed within a second side of the housing and spaced apart from the first trackball forming a gap between the first and second trackballs for receiving an endoscope such that the endoscope contacts the first and second trackballs, a leader camera disposed adjacent a third side of the housing and pointed toward the endoscope gap from a first direction, a follower camera disposed adjacent a fourth side of the housing and pointed toward the endoscope gap from a second direction different from the first direction, and at least one computing device including a non-transitory computer readable medium storing program instructions that, when executed by the computing device, cause the computing device to track a position and orientation of the endoscope in the sensor cuff in real time.

[0009] In at least one embodiment, at least one computing device comprises a leader computer and a follower computer, with the first trackball and the second trackball connected to the leader computer.

[0010] In at least one embodiment, the system further comprises a first light source for the leader camera and a second light source for the follower camera.

[0011] In at least one embodiment, a first trackball and a second trackball measure changes in the longitudinal insertion position of the endoscope over time and changes in the rotation of the endoscope over time.

[0012] In at least one embodiment, the insertion position of the endoscope is measured using a first axis of a first trackball and a second trackball, and the rotation of the endoscope is measured using a second axis perpendicular to the first axis.

[0013] In at least one embodiment, cumulative uncertainty in the measurements made by the first and second trackballs is mitigated by using leader and follower cameras to detect white scale marks that appear at regular intervals on the insertion tube portion of the endoscope.

[0014] In at least one embodiment, a leader camera and a follower camera provide image data that is used to correct for rotation by a predetermined number of degrees based on detection of markings from opposite sides of the endoscope.

[0015] In at least one embodiment, a leader computer is connected to a leader camera and a follower computer is connected to a follower camera.

[0016] In at least one embodiment, the reader computer is configured to combine all the sensor data to calculate endoscope position, endoscope rotation, and endoscope motion.

[0017] In at least one embodiment, at least one computing device, when executing the program instructions, is configured to use the white scale lines to correct for accumulated error in the insertion length of the endoscope and to use the black line gap to correct the rotation angle of the endoscope.

[0018] In at least one embodiment, the at least one computing device, when executing the program instructions, is further configured to combine the data regarding the lateral and rotational movement with live video from at least one of the leader camera or follower camera to determine whether a loop is being formed by the insertion tube of the endoscope.

[0019] In at least one embodiment, the at least one computing device, when executing the program instructions, is further configured to combine the data regarding the lateral and rotational movement with live video from at least one of the leader camera or follower camera to generate an image mosaic that maps the inner surface of the colon being imaged to identify the location of tumors and polyps.

[0020] In at least one embodiment, the at least one computing device, when executing the program instructions, is further configured to compensate for a delay in recording frames from one of the leader camera or the follower camera.

[0021] In at least one embodiment, the at least one computing device, when executing the program instructions, is further configured to detect movement of the endoscope from images obtained from the leader camera and the follower camera using at least one of computer vision, machine learning, or artificial intelligence.

[0022] In at least one embodiment, the at least one computing device is further configured to integrate at least one of a position or an orientation of the endoscope into the endoscopy training simulator when executing the program instructions.

[0023] In accordance with another aspect of the invention, a surgical instrument is provided comprising a holder having a casing with a spaced apart distal end and a central hole in the casing for receiving an endoscope having an insertion tube, at least two cameras, each of the cameras facing the center of the casing and each of the cameras mounted on one of the distal ends such that each of the cameras has an unobstructed view of the endoscope when received in the central hole of the casing, and a non-invasive surgical instrument storing program instructions that, when executed by a processor, cause the processor to track in real time the position and orientation of the distal end of the endoscope based on images acquired from a portion of the endoscope that is within the field of view (FOV) of the at least two cameras. A real-time endoscope tracker system is disclosed that includes a processor in communication with a temporary computer-readable medium, the processor being configured, when executing program instructions, to: acquire real-time images from at least two cameras of a portion of the endoscope within the FOV of the at least two cameras; measure lateral and rotational movement of the endoscope based on markings printed on the endoscope and captured in the real-time images; and correlate the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movement to determine a position and orientation of a distal end of the endoscope.

[0024] In at least one embodiment, at least two cameras cover a full 360 degree field of view of the insertion tube of the endoscope about the lateral axis of the insertion tube to measure lateral and rotational movement of the endoscope.

[0025] In at least one embodiment, the at least two cameras comprises three cameras, each of which is fixed at a point such that two consecutive cameras are located 120° to the axis of the insertion tube to provide an unobstructed view of the insertion tube and the markings thereon.

[0026] In at least one embodiment, at least two cameras are configured to capture video of the endoscope as it moves relative to a fixed FOV of the at least two cameras.

[0027] In at least one embodiment, the processor, when executing the program instructions, is further configured to identify white line markings along the length of the insertion tube to correct for cumulative errors accumulated in the measurements.

[0028] In at least one embodiment, the processor, when executing the program instructions, is further configured to identify a dark gap at a point on the circumference of the insertion tube to correct for rotational errors.

[0029] In at least one embodiment, when the processor executes the program instructions, it is configured to combine data regarding lateral and rotational movement with live video from a camera at the tip of the endoscope to determine whether a loop is being formed by the insertion tube.

[0030] In at least one embodiment, the processor, when executing the program instructions, is further configured to combine the data regarding the lateral and rotational movements with live video from a camera at the tip of the endoscope to generate an image mosaic that maps the inner surface of the colon that is imaged to identify the location of tumors and polyps.

[0031] In at least one embodiment, the at least two cameras comprise two cameras, and the processor, when executing the program instructions, is further configured to compensate for a delay in recording of frames from one of the two cameras.

[0032] In at least one embodiment, the processor, when executing the program instructions, is further configured to use computer vision to detect movement of the endoscope from images acquired from the at least two cameras.

[0033] In at least one embodiment, the processor, when executing the program instructions, is further configured to provide at least one of a position or an orientation of the distal end of the endoscope to the endoscopy training simulator.

[0034] According to another aspect of the present invention, a method of tracking an endoscope using an endoscope tracker is disclosed, the method including fixing a sensor cuff of the endoscope tracker to a position outside a patient or training phantom, manually inserting an endoscope through entrance and exit holes in the sensor cuff, positioning an insertion tube of the endoscope between the trackballs and between the cameras of the endoscope tracker, measuring a position of the endoscope using y-axis data from each trackball and measuring a rotation of the endoscope using x-axis data from each trackball, correcting for uncertainties in the position and rotation of the endoscope using image data from the cameras, and providing an output of movement of the endoscope outside the patient or training phantom.

[0035] In accordance with another aspect of the present invention, a method of tracking an endoscope is disclosed that includes connecting a trackball to one of a plurality of cameras, placing or positioning the camera to view the endoscope from an opposing end of the sensor cuff, measuring the rotation of the trackball, capturing images with the camera indicative of markings along the insertion tube of the endoscope, using the images to correct for errors in the position and rotation of the endoscope, and providing an output of the movement of the endoscope outside of a patient or training phantom.

[0036] In accordance with another aspect of the present invention, a method for tracking an endoscope is disclosed that includes polling a trackball at regular intervals to update endoscope motion, acquiring images with a camera and detecting image features of the endoscope in real time, combining the trackball data with the image data to update a real-time estimate of endoscope motion, correcting positional and rotational errors using the image features, and providing an output of endoscope motion outside a patient or training phantom.

[0037] In accordance with another aspect of the present invention, a method of tracking an endoscope is disclosed that includes capturing video of the movement of the endoscope with a camera as the endoscope moves relative to a fixed field of view of the camera, tracking the apparent movement of the endoscope using markings displayed on the insertion tube of the endoscope, correcting cumulative errors accumulated in the measurements of the apparent movement, and providing an output of the movement of the endoscope outside of a patient or training phantom.

[0038] Other features and advantages of the present application will become apparent from the following detailed description taken in conjunction with the accompanying drawings. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the present application, are given by way of illustration only, since various changes and modifications within the spirit and scope of the present application will become apparent to those skilled in the art from this detailed description.

[0039] For a better understanding of the various embodiments described herein, and to show more clearly how these may be put into practice, reference is made by way of example to the accompanying drawings, in which at least one exemplary embodiment is shown and which are now described, and which are not intended to limit the scope of the teachings described herein. [Brief description of the drawings]

[0040] [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary embodiment of a system for real-time, intraprocedural endoscope shaft motion tracking. [Diagram 2] FIG. 1 illustrates an example embodiment of an endoscope tracker with an endoscope inserted. [Diagram 3] FIG. 1 illustrates an exemplary embodiment of a sensor cuff showing its two-piece housing and its sensor. [Figure 4] 1 is an example of still images captured almost simultaneously by a leader camera and a follower camera. [Diagram 5] 13 is a graph of the endoscope tracker's position tracking versus manually recorded ground truth for a single position calibration trial. [Figure 6] 6 is a graph of error versus ground truth for FIG. 5. [Figure 7] 1 is a graph of position displacement ground truth versus position error from ground truth. [Figure 8] 1 is a graph of rotational displacement ground truth versus rotation error from ground truth. [Figure 9] 1 is a graph of trackball update count versus rotation speed. [Figure 10] 1 is a graph of position error versus average velocity results. [Figure 11] 13 is a graph in which the position error in each movement of groups 1 to 3 is plotted against the average speed. [Figure 12] 1 is a graph plotting rotation error and average rotation speed versus rotation groundruce. [Figure 13] 13A-13D illustrate an exemplary embodiment of an endoscope holder for an optical endoscope tracker. [Figure 14] 1 is a flow chart of an exemplary embodiment of a method for tracking an endoscope from the perspective of a sensor cuff. [Figure 15] 1 is a flow chart of an exemplary embodiment of a method for tracking an endoscope in terms of a trackball and a camera. [Figure 16] 1 is a flow chart of an exemplary embodiment of a method for tracking an endoscope from a data acquisition perspective. [Figure 17]1 is a flowchart of an example embodiment of a method for tracking an endoscope using a non-contact tracker. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Further aspects and features of the exemplary embodiments described herein will become apparent from the following description taken in conjunction with the accompanying drawings.

[0042] Various embodiments according to the teachings of the present specification are described below to provide an example of at least one embodiment of the claimed subject matter. Any embodiment described herein does not limit the claimed subject matter. The claimed subject matter is not limited to a device, system, or method having all of the features of any one of the devices, systems, or methods described below, or features common to more than one or all of the devices, systems, or methods described herein. There may be a device, system, or method described herein that is not an embodiment of any claimed subject matter. Any subject matter described herein that is not claimed herein may be the subject of another means of protection, for example, a continuing patent application, and the applicant, inventor, or owner does not intend to abandon, disclaim, or offer to the public any such subject matter by its disclosure herein.

[0043] It should be understood that for simplicity and clarity of description, where deemed appropriate, reference numerals may be repeated among the figures to indicate corresponding or similar elements. Additionally, numerous specific details have been described to provide a thorough understanding of the embodiments described herein. However, it will be understood by those skilled in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the embodiments described herein. Additionally, the description should not be considered as limiting the scope of the embodiments described herein.

[0044] It should also be noted that the terms "coupled" or "couple" as used herein can have several different meanings depending on the context in which the terms are used. For example, the terms coupled or couple can have a mechanical or electrical meaning. For example, as used herein, the terms coupled or couple can indicate that two elements or devices can be directly connected to each other, or can be connected to each other via one or more intermediate elements or devices, via electrical signals, electrical connections, or mechanical elements, depending on the particular context.

[0045] It should also be noted that, as used herein, the term "and / or" is intended to represent an inclusive "or." That is, "X and / or Y" is intended to mean, for example, X or Y or both. As a further example, "X, Y, and / or Z" is intended to mean X or Y or Z, or any operable combination thereof (this is meant to cover any combination of elements that results in a workable embodiment).

[0046] It should be noted that terms of degree, such as "substantially," "about," and "approximately," as used herein, refer to a reasonable amount of deviation from the modified term such that the end result is not materially altered. These terms of degree may also be construed to include deviations from the modified term, such as 1%, 2%, 5%, or 10%, if this deviation does not negate the meaning of the term it modifies.

[0047] Additionally, the recitation of numerical ranges herein by endpoints includes all numbers and fractions subsumed within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It is also understood that all numbers and fractions thereof are presumed to be modified by the term "about," which refers to a variation by up to the referenced quantity number, for example, 1%, 2%, 5%, or 10%, if the end result would not be significantly altered.

[0048] Also, it should be noted that the use of the term "window" in describing the operation of any system or method described herein is intended to be understood as describing a user interface for performing initialization, configuration, or other user operations.

[0049] Exemplary embodiments of the devices, systems, or methods described by the teachings herein may be implemented as a combination of hardware and software. For example, the embodiments described herein may be implemented, at least in part, by using one or more computer programs to run on one or more programmable devices including at least one processing element and at least one storage element (i.e., at least one volatile memory element and at least one non-volatile memory element). The hardware may include input devices including at least one of a touch screen, a keyboard, a mouse, a button, a key, a slider, etc., as well as one or more of a display, a printer, etc., depending on the hardware implementation.

[0050] It should also be noted that some elements used to implement at least some of the embodiments described herein may be implemented via software written in a high-level procedural language, such as object-oriented programming. ++, C#, JavaScript, Python, or any other suitable programming language and may include modules or classes as known to those skilled in the art of object-oriented programming. Alternatively, or in addition, some of these elements that are implemented via software may be written in assembly language, machine language, or firmware, as appropriate. In either case, the language may be a compiled or interpreted language.

[0051] At least some of these software programs may be stored on a computer readable medium, such as, but not limited to, a ROM, a magnetic disk, an optical disk, a USB key, etc., that is readable by a device having a processor, an operating system, and associated hardware and software necessary to implement the functionality of at least one of the embodiments described herein. The software program code, when read by the device, configures the device to operate in a new specific predefined manner (e.g., an application specific computer) to perform at least one of the methods described herein.

[0052] At least some of the programs associated with the device, system, and method embodiments described herein may be dispersible in a computer program product that includes a computer-readable medium carrying computer usable instructions, such as program code, for one or more processing units. The medium may be provided in a variety of forms, including, but not limited to, one or more diskettes, compact discs, tapes, chips, and non-transitory forms, such as magnetic and electronic storage devices. In alternative embodiments, the medium may be transitory in nature, such as, but not limited to, wired transmissions, satellite transmissions, Internet transmissions (e.g., downloads), media, digital signals, and analog signals. The computer usable instructions may also be in a variety of formats, including compiled and non-compiled code.

[0053] In accordance with the teachings herein, various embodiments are provided for real-time in-procedure endoscope shaft motion tracking, and computer products for use therewith. At least one of these embodiments includes tracking of the insertion length and orientation of the endoscope. These embodiments also have application to EGD, small intestine endoscopy, and large intestine endoscopy. More generally, these embodiments may be applied to any potential gastrointestinal (GI) endoscopic procedure, as well as the inspection of the airways, for example, by bronchoscopy, with or without (unless necessary) appropriate modifications known to those skilled in the art.

[0054] One advantage of at least one embodiment of the invention described herein is that it provides a real-time endoscope shaft motion tracker that works with most unmodified endoscopes, lowering the barrier to adaptation to endoscope motion trackers and enabling improved localization with less equipment and training costs.

[0055] Another advantage of at least one of the embodiments of the invention described herein is that lateral and rotational movement can be measured using built-in markings (patterns) printed on the insertion tube.

[0056] Yet another advantage of at least one of the embodiments of the invention described herein is that they can be used with any existing endoscope currently used by healthcare providers, reducing initial setup costs. This is in contrast to 3D localization solutions that require the purchase of a new dedicated endoscope with built-in coils or sensors and large detection units; these systems do not measure rotation, but instead display a 3D position model.

[0057] Referring first to FIG. 1, a block diagram of an exemplary embodiment of a system 100 for real-time endoscope insertion length and orientation tracking is shown. The system 100 includes at least one server 120. The at least one server 120 may be, for example, one or more server computers in a client / server architecture. Alternatively, or in addition, the at least one server 120 may be two or more computers operating independently or interdependently from each other, such as leader-follower or peer-to-peer. Alternatively, or in addition, the at least one server 120 may be a single computer configured to send instructions to and receive data from two or more cameras and two trackball electronic signal interfaces. For ease of reference, the at least one server 120 is referred to herein as a "server 120" even if it is multiple computers. The server 120 may communicate with one or more user devices (not shown), for example, wirelessly or via the Internet. The system 100, when so used, may also be referred to as a machine learning system.

[0058] A user device may be a computing device operated by a user. A user device may be, for example, a smartphone, a smartwatch, a tablet computer, a laptop, a virtual reality (VR) device, or an augmented reality (AR) device. A user device may be a combination of computing devices operating together, such as, for example, a smartphone and a sensor. A user device may also be a device operated by a user, such as, for example, a drone, a robot, a remotely operated device, and in such cases, the user device may be operated by a user via, for example, a personal computing device (such as a smartphone). A user device may be configured to run applications (e.g., mobile apps) that communicate with other parts of the system 100, such as the server 120.

[0059] The server 120 may operate on a single computer including a processor unit 124, a display 126, a user interface 128, an interface unit 130, input / output (I / O) hardware 132, a network unit 134, a power supply unit 136, and a memory unit (also called a "data store") 138. In other embodiments, the server 120 may have more or fewer components, but generally function in a similar manner. For example, the server 120 may be implemented using multiple computing devices.

[0060] The processor unit 124 may include a standard processor, such as an Intel Xeon processor. Alternatively, there may be multiple processors used by the processor unit 124, which may work in parallel to perform certain functions. The display 126 may be, but is not limited to, a computer monitor or an LCD display, such as for a tablet device. The user interface 128 may be an application programming interface (API) or web-based application accessible via the network unit 134.

[0061] The processor unit 124 may execute a prediction engine 152 that functions to provide predictions by using one or more machine learning models 146 stored in the memory unit 138. The prediction engine 152 may build predictive algorithms by machine learning. The training data may include, for example, image data, video data, audio data, and / or text data.

[0062] The processor unit 124 may also execute a graphical user interface (GUI) engine 154 that is used to generate the various GUIs. The GUI engine 154 provides data according to a fixed layout for each user interface and receives data or control input from a user. The GUI then uses the input from the user to change the data displayed in the current user interface or to change the operation of the server 120, which may include displaying a different user interface. For example, the GUI may allow a user to connect, reconnect, or disconnect to other parts of the system 100, as well as perform debugging. The GUI may also allow a user to display images or video from an external source (e.g., a camera) and overlay a box (or other shape) around a portion of the image or video. The GUI may also provide controls for a user to test some or all of the system 100. The GUI may also provide a visual representation of an object (e.g., an endoscope) connected to the system 100.

[0063] The interface unit 130 may be any interface that allows the processor unit 124 to communicate with other devices in the system 100. In some embodiments, the interface unit 130 may include at least one of a serial or parallel bus and corresponding ports, such as a parallel port, a serial port, a USB port, and / or a network port. For example, a network port may be used to allow the processor unit 144 to communicate over the Internet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a wireless local area network (WLAN), a virtual private network (VPN), or a peer-to-peer network, either directly or through a modem, router, switch, hub, or other routing or conversion device.

[0064] The I / O hardware 132 may include, but is not limited to, for example, at least one of a microphone, a speaker, a keyboard, a mouse, a touchpad, a display device, a printer, a camera, and a medical device (e.g., an endoscope).

[0065] The network unit 134 includes various communications hardware to enable the processor unit 124 to communicate with other devices. For example, the network unit 134 includes at least one of a network adapter, such as an Ethernet or 802.11x adapter, a Bluetooth radio or other short-range communications device, or a wireless transceiver for wireless communications according to the CDMA, GSM, or GPRS protocols using standards such as IEEE 802.11a, 802.11b, 802.11g, or 802.11n.

[0066] Power supply unit 136 may include one or more power supplies (not shown) connected to provide power to the various components of system 100, as is commonly known to those skilled in the art.

[0067] The memory unit 138 may comprise one or more non-transitory machine-readable memory units or storage devices accessible by other modules, such as the image signal processing module 126, the image / video encoder module 130, and the communication module 132, to read and execute machine-executable instructions (e.g., as firmware) stored therein, and to read and / or store data, such as data of captured images and data generated during operation of the other modules. The memory module 128 may be a volatile and / or non-volatile memory, non-removable memory or removable memory, such as RAM, ROM, EEPROM, solid state memory, hard disk, CD, DVD, flash memory, etc.

[0068] The memory unit 138 may comprise one or more non-transitory machine-readable memory units or storage devices accessible by other modules for storing program instructions of the operating system 140, program code 142 of other applications, an input module 144, one or more machine learning models 146, an output module 148, and a database 150. A process may access the memory unit 138 to read and execute the machine-executable instructions stored therein, and to read and / or store data during operation of the other modules. The memory unit 138 may be a volatile and / or non-volatile memory, non-removable memory or removable memory, such as RAM, ROM, EEPROM, solid state memory, hard disk, CD, DVD, flash memory, etc. The machine learning models 146 may include, but are not limited to, image recognition and classification algorithms based on deep learning models and other approaches. Although the database 150 is shown as being stored on a local memory, it should be understood that the database 150 may be, for example, an external database, a database on the cloud, multiple databases, or a combination thereof.

[0069] In at least one embodiment, the machine learning model 146 may include one or more convolutional neural networks, one or more recurrent neural networks, or a combination of one or more convolutional neural networks and one or more recurrent neural networks. A convolutional neural network (CNN) may be designed to recognize images or patterns. A CNN may perform convolutional operations and may be used, for example, to classify regions of an image and determine edges of objects recognized within an image region. A recurrent neural network (RNN) may be used to recognize sequences, such as text, speech, time evolution, etc., so that an RNN may be applied to a sequence of data to predict what will happen next. Thus, a CNN may be used to read what is happening on a given image at a given time, and an RNN may be used to recognize one or more sequences and, optionally, make one or more predictions and provide an informational message. Alternatively or additionally, the machine learning model may use computer vision, for example, alone or in combination with a CNN and / or an RNN.

[0070] Program 142 includes program code that, when executed, configures processor unit 124 to operate in a particular manner to implement various functions and tools for system 100, such as the functions discussed in connection with methods 1400, 1500, 1600, and 1700.

[0071] The input module 144 may include program code for obtaining input data and / or operating parameters from a variety of sources including, for example, a user and / or memory.

[0072] The output module 148 may include program code for displaying various data and / or storing the calibration data, image data, patient data, and / or operating parameters in a file, such as in a database 150 .

[0073] 2, there is shown an exemplary embodiment of an endoscope tracker 250. In the implementations of endoscope tracker 250 described herein, some or all of system 100 may be used to perform any one or more functions requiring computer functionality (e.g., digital input, digital output, software processing, image processing, etc.).

[0074] FIG. 2 shows an exemplary setup in which an endoscope tracker 250 is inserting an endoscope 255 into an 85 cm line marking. The endoscope controls are shown on the left, and the insertion tube passes through the sensor cuff on the right. A leader Raspberry Pi (sometimes called "RasPi", "RPi") computer 260 is in the foreground, and a follower RasPi computer 270 is in the background. Two trackballs (see, e.g., FIG. 3) are connected to the leader RasPi 260 via USB. Each RasPi computer is connected to a camera via a camera serial interface (CSI) and to a light source via an integrated circuit (I2C). The endoscope 255, leader RasPi 260, and / or follower RasPi 270 may be considered part of the I / O hardware 132 of the system 100. The leader RasPi 260 may alternatively be referred to as a leader computer (or "master computer"), and the follower RasPi 270 may alternatively be referred to as a follower computer (or "server computer"). Each RasPi computer may have some or all of the functionality of the server 120. It should be understood that the embodiments of the invention described herein should not be limited to a particular make or model of Raspberry Pi computer, for example, other single board computers (SBCs) may be used instead. Some SBCs can connect to two cameras, and the program instructions (or code) for the leader computer may be combined with the program instructions (or code) for the follower computer such that all calculations are performed in one processor. Examples of SBCs that can connect to more than one camera include various NVIDIA® Jetson™ SBCs, Raspberry Pi Compute Module 4 (two cameras), and Raspberry Pi 5 (two cameras).

[0075] To maximize the use of endoscope trackers, the position and orientation of the distal tip of the endoscope must be accurately measured in order to pinpoint the location of any lesions discovered during an endoscopy. In current practice, the position of the endoscope's distal tip camera is estimated using graduated lines on the endoscope's insertion tube. Each line indicates a distance from the endoscope tip. This allows the insertion depth of the endoscope into the patient's body to be manually recorded at critical points during an endoscopy, such as when a lesion is discovered. However, traditionally, the rotation of the endoscope's camera is not measured, and therefore its orientation cannot be determined.

[0076] According to the teachings herein, a real-time endoscope motion tracker continuously records the position and rotation of the endoscope, as well as the insertion and rotational speed (and optionally the acceleration) of the endoscope insertion tube at the orifice. The motion tracker uses a sensor cuff that surrounds the endoscope insertion tube at a point outside the patient. Because the endoscope does not change length and is rotationally rigid and unbending, this movement of the endoscope insertion tube at the orifice (i.e., the movement of the endoscope sheath at the location where the endoscope enters) correlates with the movement of the distal tip inside the patient. The motion tracker mechanically tracks the movement of the endoscope using a trackball and detects the scale marks using a camera to correct for accumulated errors due to slippage of the endoscope movement relative to one or both of the trackballs. The motion tracker can determine the position of the endoscope tip based, for example, on the insertion depth of the endoscope. Alternatively or additionally, the position of the tip can be inferred from the movement of the endoscope shaft and analysis (e.g., machine learning) of the simultaneous movement or change in the image features of the endoscope.

[0077] The goal is to improve polyp and cancer localization by simultaneously combining motion tracking data from the endoscope tip with video captured by an endoscope camera at the tip of the endoscope. The tracker data is used to stitch together frames to create a 3D reconstruction, or unwrapped map, of the colonic mucosa.

[0078] 2, the real-time endoscope tracker 250 includes a sensor cuff 265 and a real-time motion tracking program. The sensor cuff 265 may be considered part of the I / O hardware 132 of the system 100. The real-time motion tracking program may be considered part of the program 142 of the system 100. The program may be executed simultaneously, for example, on two Raspberry Pi (RasPi) computers 260, 270 that cooperate in a leader / follower model (leader RasPi 260, follower RasPi 270). The endoscope tracker 250 may be more generally referred to as a real-time endoscope tracker system when it includes computer hardware and / or software that cooperate to track a portion of an endoscope, such as, for example, the endoscope tip.

[0079] FIG. 3 shows several views of an exemplary embodiment of a sensor cuff 300 showing its housing (in this case, two mounting parts) and its sensor. The top view shows the upper housing part 380 with the upper trackball 310 mounted, the middle view shows the sensor schematic in side view, and the bottom view shows the lower housing part 382 with the lower trackball 320 and the two cameras 330, 340 housed. The coordinate system of the endoscope 305 is annotated next to the endoscope 305 in the side view. The positive x-axis (axis of rotation) and y-axis (axis of insertion) of the trackball are marked on the trackball using arrows at right angles to each other. The upper housing part 380 has two openings 380a, 380b for the passage of the endoscope 305. The bottom housing part 382 has two openings 390a, 390b for the passage of the endoscope 305. The sensor cuff 300 may be part of a real-time endoscope tracker 250. Sensor cuff 300 may be the same as, for example, sensor cuff 265 of Figure 2. Sensor cuff 300 may be considered to be part of I / O hardware 132 of system 100.

[0080] Inside the sensor cuff 300 are two trackballs 310, 320, two cameras 330, 340, and two light sources 360, 370, which may be mounted as shown in FIG. 3. The trackballs 310, 320 measure the insertion and rotation of the endoscope over time. Accumulative uncertainty (i.e., cumulative error) in the measurements of the trackballs 310, 320 may be mitigated, for example, by using the cameras 330, 340 to detect white scale marks that appear at regular intervals (e.g., every 5 cm) on the insertion tube of the endoscope 305. The trackballs 310, 320 and / or the cameras 330, 340 may be considered part of the I / O hardware 132 of the system 100. method

[0081] A method of using an endoscopic tracker includes steps involving one or more of a sensor cuff, a trackball, a camera, data acquisition, and tracker performance evaluation. A. Sensor cuff

[0082] The sensor cuff 300 may be fixed in position outside the patient or training phantom. The sensor cuff 300 may be prism-shaped through whose center the endoscope 305 passes horizontally, or another suitable shape. The endoscope 305 may be considered part of the I / O hardware 132 of the system 100. The endoscope 305 may be manually inserted through the entrance holes (collectively openings 380b, 390a) and exit holes (collectively openings 390a, 390b) of the sensor cuff 300, such that the insertion tube of the endoscope 305 is positioned vertically between the upper trackball 310 and the lower trackball 320, and horizontally between the cameras 330, 340 at either end of the sensor cuff 300. The endoscope 305 is sandwiched vertically between the two trackballs 310, 320 and is visible by the cameras 330, 340 at the left and right ends of the sensor cuff 300. In at least one implementation, the bottom trackball 320 may support the endoscope 305 and prevent the endoscope 305 from moving downward inside the sensor cuff 300 due to gravity, making it easier to process images. In alternative embodiments, the endoscope tracker may be calibrated for different orientations of the sensor cuff 300, in which case the trackballs 310, 320 may be placed in different locations, such as on either side of the endoscope along different planes (e.g., horizontal), but this may not work accurately / robustly unless the weight of the endoscope 305 is supported differently by the housing of the sensor cuff 300.

[0083] More generally, the orientations "horizontal" and "vertical" can be used to refer to relative positions. In practice, the cameras 330, 340 can be positioned 180° from each other to cover a full view of the tube portion of the endoscope 305. The two trackballs 310, 320 can be positioned opposite each other to provide pressure / fit to the tube (e.g., to maintain alignment and reduce slippage). The relative positions of the trackballs 310, 320 and the cameras 330, 340 do not need to be perfectly orthogonal to each other, as long as the trackballs 310, 320 do not obstruct the view of the cameras 330, 340. The lateral positions of the trackballs 310, 320 and / or the cameras 330, 340 can be shifted along the length of the endoscope 305 to achieve full visibility of the tube portion of the endoscope 305.

[0084] The housing of the sensor cuff 300 can be 3D printed in two parts 380, 390. The top part 380 holds one trackball 310 in place and has slits on both ends for the cables. The bottom part 390 aligns the cameras 330, 340 and another trackball 320 to the endoscope 305. Four screws, or other suitable fasteners, can connect the two halves, one at each corner of the housing. To ensure accurate tracking, the screws can be adjusted so that the trackball has the necessary tracking force on the endoscope. Figure 3 is a schematic diagram showing the top and bottom parts 380, 390 of the tracker sensor cuff 300, as well as how the sensors are arranged.

[0085] The endoscope 305 is in direct contact with both trackballs 310, 320, so that both trackballs 310, 320 rotate in position as the endoscope 305 moves. The position (e.g., insertion length) of the endoscope 305 is measured using the y-axis of the trackballs 310, 320, and the rotation is measured using the x-axis of the trackballs 310, 320. The measurement uncertainty of each trackball is proportional to the rotation of each axis. The resulting uncertainty (or accumulated measurement error) in the position and rotation of a portion of the endoscope 305 can be corrected by using the cameras 330, 340 to record images that can be used to detect scale markings that occur at regular intervals (e.g., 5 cm intervals) along the insertion tube of the endoscope 305. The two cameras 330, 340 of the sensor cuff 300 view the markings from opposite sides of the endoscope 305, allowing the rotation of the endoscope 305 to be corrected (e.g., at a predetermined number of degrees, such as every 180°, at different intervals, continuously, etc.) during operation. In another embodiment using three cameras, the measurement uncertainty (or accumulated measurement error) is corrected for example every 120°.

[0086] The entrance and exit diameters and the distance between the two trackballs 310, 320 can be customized for a particular type of endoscope based on its shaft diameter, allowing for good contact without significant slippage as the endoscope 305 moves within the sensor cuff 300.

[0087] All endoscope tracker calibrations and results presented herein were performed with a prototype device according to the teachings herein using a 1 m long gastroscope (GIF-XQ10, Olympus Canada), however the prototype device can be adjusted to work with different endoscope models that indicate insertion length with scale markings. The prototype device was tested with three common commercially available endoscope insertion tube shaft diameters with similar results to those reported here.

[0088] In one exemplary setup, two RasPis (Model 3BC, RasPi Trading; Cambridge, England, UK) read the trackball movements, capture and process images, and calculate information about the scope's movements in real time. The tracker output can be displayed on a separate monitor (current setup) and / or integrated with the endoscope controller. Each RasPi is connected to a RasPi camera (Version 1.3, RasPi Trading) and a white light-emitting diode (LED) ring light for illumination (Pi-Light, Mindsensors.com; Henrico, VA, USA). We designate one of the RasPis as the "leader" and the other as the "follower." The leader RasPi 260 is connected to one camera and two USB trackballs. The follower RasPi 270 controls only the second camera. The leader RasPi 260 combines all the sensor data and calculates the position, rotation, and movement of the endoscope.

[0089] 14 shows a flowchart of an example embodiment of a method 1400 for tracking an endoscope from the perspective of a sensor cuff. Method 1400 may be implemented using some or all of system 100 and / or some or all of endoscope tracker 250.

[0090] At 1410, the sensor cuff is secured in position outside the patient or training phantom.

[0091] At 1420, an endoscope is manually inserted through the entrance and exit holes in the sensor cuff.

[0092] At 1430, an insertion tube of an endoscope is placed between the trackballs (eg, two trackballs) and between the cameras (eg, two cameras).

[0093] At 1440, the y-axis of the trackball is used to measure the position of the endoscope and the x-axis of the trackball is used to measure the rotation of the endoscope.

[0094] At 1450, image data acquired from the camera is used to correct for position and rotational uncertainties in the endoscope. Correcting the uncertainties may be accomplished in one or more of the ways described herein for correcting positional or rotational errors.

[0095] At 1460, an output of the movement of the endoscope outside the patient or training phantom is provided (eg, displayed).

[0096] FIG. 4 shows the inside of the sensor cuff 410 and where the trackball and camera are mounted 420. B. Trackball Setup

[0097] Two trackballs (e.g., X13 trackballs, manufactured by Cursor Controls Ltd.) directly measure the movement of the endoscope. The side view in FIG. 3 shows that the positive insertion of the endoscope 305 is oriented into the page and the positive rotation is clockwise. The positive y-axis of the trackballs 310, 320 is aligned with the positive insertion axis of the endoscope. Similarly, the positive x-axis of the trackballs 310, 320 is aligned with the positive rotation direction of the endoscope. Two laser trackballs (e.g., X13, Cursor Controls; Newark, Nottinghamshire, UK) are connected to the reader RasPi260 and convert the movement of the endoscope into trackball rotation. Trackball rotation can also be measured optically without direct contact between the trackballs and the sensors 350 and 360 via USB.

[0098] In this exemplary prototype, each trackball has a diameter of 12.7 mm and is configured to output 300 ± 10% counts per revolution in linear mode

[10] . Thus, a single trackball has a linear resolution (l spec ).

number

[0099] The angular resolution achievable by using a single trackball to track an endoscope is proportional to the insertion tube (d it ) depends on the diameter of the gastroscope. An Olympus GIF-XQ10 gastroscope with a diameter of 9.8 mm was used for the tests. The angular resolution (a spec ) is given by the following formula:

number

[0100] Each trackball is configured to produce approximately 231 counts per 360 degrees. Depending on the diameter of the endoscope shaft, other numbers of counts per revolution are possible (e.g., 100 counts per 360°, 360 counts per 360°, 480 counts per 360°, etc.).

[0101] The trackballs 310, 320 generate more counts than expected during tracking, and the counts from both trackballs 310, 320 are combined to generate a more robust estimate of the endoscope motion, as described in the Calibration and Calculations section. C. Camera Setup

[0102] Two Raspberry Pi cameras 330, 340 view the endoscope 305 from opposite ends of the sensor cuff 265. The cameras 330, 340 are set up to acquire grayscale images as detailed in Table 1. The cameras have rolling shutters with a shutter speed of 11,111 μs at 90 Hz. Both cameras 330, 340 are placed at a working distance of 3.5 cm and their focuses are adjusted. Illumination for the cameras is provided by two Mindsensors Pi-Lights, each consisting of four RGB LEDs

[11] .

[0103] The leader camera on the leader RasPi260 and the follower camera on the follower RasPi270 are used to detect markings that appear every 5 cm along the insertion tube of the endoscope. The leader camera is alternatively called the "master camera" and the follower camera is alternatively called the "server camera". Images of these markings are used to correct position and rotation errors accumulated from the trackball measurements. The cameras have a fixed field of view (FOV) that includes a region of interest (ROI) where the insertion tube appears. A 2.7 cm long insertion tube is visible in each image. The leader and follower cameras look at opposite sides of the endoscope's insertion tube, as shown in Figure 4. The software for the cameras includes different program code paths to execute in the leader RasPi260 and follower RasPi270, the programs are selected, for example, using flags on the command line when the software is started, and the software also includes different calibration / cropping constants for the two cameras that can be empirically determined during calibration.

[0104] More specifically, Figure 4 shows an example of still images taken by the leader and follower cameras at approximately the same time. The follower (or "server") image has been rotated 180 degrees and flipped in orientation. The image shows a 40 cm line with no gaps 410, and a gap 420 in the 40 cm line. These image features can be used to correct for insertion and rotational position errors of the endoscope 305. Also, two "40"s are displayed on the opposite side of the endoscope.

[0105] The open source AVA RaspiCam code

[18] was modified to configure the RasPi cameras and capture images at 90 frames per second. OpenCV (CCC) was used for high-speed image processing. The settings used for each camera 260, 270 are summarized in Table 1. In addition, the original focal length of each camera lens was shortened to match a working distance of 3.5 cm, and a ring light was attached to each camera to provide illumination. [Table 1]

[0106] For each white line on the gastroscope 255, there is one black line gap. The line gaps are all aligned along one side of the insertion tube. The real-time motion tracking program uses the white scale lines to correct the cumulative error of the insertion length and the black line gaps to correct the rotation angle. By using leader and follower cameras, the length position (aka insertion length) can be corrected every 5 cm and the rotation angle every 180 degrees. FIG. 4 shows an example image 400 of the white scale lines 410 and the black line gaps 420.

[0107] The real-time motion tracking program detects two types of image features: white graticule lines and black line gaps. The origin is at the top left corner of the image. A 180° rotation setting is used for the leader camera, as this causes the x-axis of the positive image to coincide with the positive endoscope insertion direction. The follower camera does not require rotation. In this setup, when the endoscope is inserted into the sensor cuff, the graticule image feature moves to the right and the line gap image moves downward during clockwise (CW) endoscope rotation. Then, both images are cropped to display the endoscope ROI before the image features are detected. For example, the ROI for white line detection is the insertion tube of the endoscope. Figure 4 shows the cropped images 400 captured by the leader and follower cameras and the detected image features.

[0108] The GUI, which may be provided by, for example, GUI engine 154, may provide the user with the ability to connect / reconnect and / or disconnect from system 100. The GUI allows the user to view video from both cameras 330, 340. The GUI may draw a box around image features of white and black line gaps. The GUI may also provide the user with controls to test the cameras 330, 340, LED lights, or some or all of the system 100. The GUI may show the insertion length (and possibly rotation and / or speed) of endoscope 305 in real time (e.g., 90 Hz).

[0109] Figure 4 shows a pair of images 400 recorded simultaneously by both cameras. The upright image from the leader camera is used to define the coordinate system of both cameras. The follower camera image is rotated 180° before feature detection. Rotating the image means that the direction of observed movement appears the same on both cameras, since the cameras are placed on opposite sides of the endoscope. When the endoscope is inserted, the image feature moves to the right. This increases the x pixel coordinate of the feature. When the endoscope rotates clockwise, the image feature moves down and the y pixel coordinate increases.

[0110] The camera images two types of features that appear on the endoscope: white scale lines 410 and black line gaps 420. Each white scale line 410 has one small black gap 420. The gaps are aligned with each other and appear on one side of the insertion tube of the scope. The camera uses the white scale lines to correct for insertion length and the black line gaps to correct for rotation. This allows, for example, to correct insertion length (e.g., at a predetermined distance, such as every 1 cm, every 5 cm, continuously, etc.). Both the white scale line features 410 and the black line gap features 420 can be seen in FIG. 4.

[0111] In at least one embodiment, the motion tracking program is one of the programs 142 and is used to retrieve all features from all cameras (e.g., two, three, or more cameras). Then, based on the location of the feature and which camera, the motion tracking program calculates the position (insertion and rotation). In at least some cases, a particular feature (e.g., the number "10") may appear in the images (or FOVs) of multiple cameras. The motion tracking program may only need one image from one camera to calculate the position, but it may be desirable to use all images. The motion tracking program may employ an algorithm (or subroutine) that averages the position calculations to obtain better results.

[0112] In embodiments with a two-camera setup (e.g., as in Figs. 13A0-13D, either with or without a trackball), a certain "feature" (e.g., a gap) may only appear in one camera's FOV. On the other hand, a white tick mark will generally be in images acquired from both cameras. A motion tracking program can use both camera images to detect the white tick mark while searching for the gap in both images (as it is not predictable which image this feature will appear in). In a three-camera (or multiple camera) setup, the cameras' FOVs may overlap.

[0113] In a situation where both cameras have images with the same timestamp where the endoscope has been rotated such that only white tick mark features or black gap features are detected, the motion tracking program can correct (e.g., if both cameras and a trackball are used) or determine (e.g., if a camera is used but not a trackball) the position and rotation measurements. The motion tracking program can use the relative position of each camera and its FOV to calculate the position with the help of a prior calibration.

[0114] In situations where both cameras have images with the same timestamp where neither the white tick mark feature nor the black gap feature was detected, the program can correct (e.g., if both cameras and a trackball are used) or determine (e.g., if a camera is used but not a trackball). As noted above, in many images, a particular feature (e.g., the number "10") may appear in the images (or FOV) of multiple cameras. A motion tracking program may use data from only one image from one camera to calculate position, although it may be desirable to use all images. The program may employ an algorithm (or subroutine) that averages position calculations to obtain better results.

[0115] 15 is a flow chart of an exemplary embodiment of a method 1500 for tracking an endoscope from the perspective of a trackball and a camera. Method 1500 may be implemented using some or all of system 100 and / or some or all of endoscope tracker 250.

[0116] At 1510, a trackball is connected to one of a number of cameras.

[0117] At 1520, a camera is placed or positioned to view the endoscope from the opposite end of the sensor cuff.

[0118] At 1530, the rotation of the trackball is measured.

[0119] At 1540, an image is captured with a camera showing markings along the insertion tube of the endoscope.

[0120] At 1550, the images are used to correct position and rotation errors of the endoscope. For example, errors can be corrected laterally every 5 cm and rotationally every 180° by resetting the trackball measurements to start from a position (rotational / lateral) determined by the imaging (camera) data. Alternatively, these corrections can be made at various intervals, i.e., not limited to 5 cm / 180°. Error correction can be accomplished by one or more of the position error correction or rotation error correction methods described herein.

[0121] At 1560, an output of the movement of the endoscope outside the patient or training phantom is provided (eg, displayed). D. Data Acquisition Setup

[0122] The real-time endoscope motion tracker 250 can acquire data using a computer program running on two RasPi 260, 270. The leader RasPi 260 is connected to both trackballs 310, 320 and polls the trackballs 310, 320 every 8 ms to update the measurement data of the movement of the trackballs 310, 320, which is then used to monitor the movement of the endoscope. In addition, both the leader RasPi 260 and the follower RasPi 270 use cameras to acquire images and detect image features in real time.

[0123] The follower RasPi 270 runs a remote procedure call (RPC) server that passes the position determined by the detected image data to the leader RasPi 260. The RPC server is initiated by the endoscope tracking program of the follower RasPi 270, and the RPC server is part of the endoscope tracker program (i.e., the motion tracking program) that runs on the follower RasPi 260. In particular, pixel coordinates (i.e., the x pixel coordinate of the center of the white line and the y pixel coordinate of the black line gap) are transmitted. Alternatively, in at least one case, it may be possible for the calculation of the position to be based on the geometric relationship of the camera to the endoscope insertion tube surface, ascertained by empirical calibration of an endoscope with the same tube diameter. The leader RasPi 260 polls the follower RasPi 270 for updates at regular intervals. To improve the reliability of the connection, the pair of RasPi 260, 270 is connected using wired Ethernet.

[0124] In at least some cases, the RPC server is started as a subprogram in the follower program. The RPC server is provided by the gRPC library, and the follower program code defines how it starts up, responds to remote procedure call queries (with image feature coordinate data of the latest frame), and shuts down. The leader program acts as a client and sends remote procedure call queries to the RPC server running on the follower. The follower program provides a queue to the RPC server with up-to-date information about the location of white and black line gaps found in the most recent image frame, if any, when they are seen by the follower's camera. The RPC server subprogram running on the follower replies to queries from the leader program using formatted messages provided by the follower program in the queue. The information flows as follows: camera → follower program → image features found and stored as messages → message queue (for valid data) → RPC server subprogram → response message to leader's query → leader program receives and decodes the message → leader program uses the information for its tracking logic.

[0125] The reader RasPi260 combines the lateral and rotational positions determined from both cameras to update a real-time estimate of the endoscope's motion. Alternatively, or in addition, position errors may be corrected every time the center of the white line (image feature) crosses the center of the x-axis in the image acquired by the reader camera. As described above, rotation errors are corrected every time the center of the black line gap (image feature) crosses the center of the cropped endoscope ROI in the y-axis for the image acquired from either camera. Thus, for example, position and rotation can be corrected at a predetermined distance, such as every 5 cm (or other intervals, such as 1 cm or 10 cm, or continuously), and at a predetermined number of degrees, such as 180° (or other intervals, such as 90° or π radians, or continuously), respectively.

[0126] The endoscope tracker 250 can estimate the endoscope's motion in real time using two Raspberry Pi 3 B+ (RasPi) computers. The program can be written in C++ and can be multi-threaded to optimally use the limited resources of the RasPi platform. In an exemplary setup, each RasPi is connected to a Pi camera and an associated Pi-Light. Since it is not possible to connect both cameras to one RasPi, a client-server model is used instead. The first RasPi is called the leader (or "master" or "client") RasPi, and the second RasPi is called the follower (or "server") RasPi. The cameras are similarly designated as leader (or "master") and follower (or "server").

[0127] The leader RasPi combines the information from all tracker sensors. It reads updates from both trackballs and uses its camera to detect image features of the endoscope (e.g., an image of the endoscope shaft). At the same time, it acts as a leader to the follower RasPi and receives endoscope features from the follower camera. Finally, the leader RasPi measures the movement of the endoscope outside the patient. This allows real-time tracking of the endoscope tip movement as it correlates the movement of the endoscope inside and outside the patient.

[0128] The follower RasPi specializes in detecting the image features of the endoscope using its camera and provides the features to the leader RasPi, as described below. The follower RasPi communicates with the leader RasPi over wired Ethernet by performing remote procedure calls using gRPC. During normal operation, the leader RasPi periodically sends update queries to the follower RasPi. The follower RasPi responds by sending image features from the latest processed image.

[0129] In at least one embodiment, each set of endoscope features can be time-stamped. Statistics can be recorded to measure the time lag in the process of getting features from the follower cameras. This can be measured by subtracting the most recent timestamp of the leader camera from the timestamp of the most recent follower camera feature. The time lag (delay) during use can be small. However, in at least one embodiment, the leader camera has less lag and can be used for some decisions. Alternatively, the leader and follower cameras (or multiple cameras if there are more than two cameras) can be connected to one single board computer (SBC) to eliminate network lag.

[0130] In at least one embodiment, the endoscope tracker tracks the movement of a sensor cuff external to the patient, which is related and correlated to the movement of the distal end of the endoscope because the endoscope is rotationally stiff and does not bend.

[0131] 16 is a flow chart of an exemplary embodiment of a method 1600 for tracking an endoscope from a data acquisition perspective. Method 1600 may be implemented using some or all of system 100 and / or some or all of endoscope tracker 250.

[0132] At 1610, the trackball is polled at regular intervals to update the movement of the endoscope.

[0133] At 1620, images are captured by the camera and image features of the endoscope are detected in real time.

[0134] At 1630, the trackball data and image data are combined to update a real-time estimate of the endoscope's motion, and the position measured by the trackballs is periodically corrected by the position determined by the image data. The trackball data provides measurement information, and the image data is used for the correction. For example, if the endoscope is in direct contact with both trackballs, the trackballs will rotate in place as the endoscope moves. The position (insertion length) of the endoscope may be measured using the y-axis of the trackballs, and the rotation may be measured using the x-axis of the trackballs. The measurement uncertainty of each trackball is proportional to the rotation of each axis. The resulting uncertainty in the position and rotation of the endoscope may be corrected using a camera. For example, the camera may detect scale markings that appear at regular intervals (e.g., 5 cm) along the insertion tube. Also, for example, the camera may capture images of the markings from the opposite side of the endoscope so that the rotation of the endoscope can be corrected at regular (e.g., every 180°) intervals (or partial rotations).

[0135] At 1640, the image features are used to correct for position and rotation errors of the endoscope. Examples of these corrections are described in more detail below.

[0136] At 1650, an output of the movement of the endoscope outside the patient or training phantom is provided (eg, displayed). E. Tracker Performance Evaluation

[0137] The real-time endoscopic motion tracker was tested using an Olympus GIF-XQ10 gastroscope with a maximum insertion length of 102.5 cm and an insertion tube diameter of 9.8 mm.

[0138] The positional accuracy of the system was evaluated. Each trial was started by positioning the sensor cuff on the tracker so that the 20 cm line was aligned with the outer edge of the sensor cuff. When the trial started, the endoscope was inserted 80 cm. The camera first captured the 20 cm line and continued to move until the 100 cm line reached the outer edge of the sensor cuff. Finally, the endoscope was withdrawn until it again reached the starting position of 20 cm. Calibration and Calculations

[0139] The method of endoscope tracker calibration and computation includes steps related to one or more of trackball calibration, motion estimation, camera calibration, and camera feature detection. The tracker's trackball and camera are calibrated before they can be used to accurately record the endoscope's motion. Information from all sensors is then combined to estimate the endoscope's motion in real time. A. Trackball Calibration and Motion Estimation

[0140] The X13 trackball is a USB device designed to allow humans to interact with a computer. The user's hand can easily apply the required trackball tracking force. The situation is different inside the sensor cuff of an endoscopic motion tracker. There, the endoscope is sandwiched between two trackballs, and as the operator moves the endoscope through the sensor cuff, the tracking force applied to each trackball changes. If the tracking force is too low, the endoscope may slip through the trackballs. If it is too high, the trackball cannot rotate smoothly. Thus, when the tracking force of a trackball is outside its expected range, the trackball will usually record fewer counts. This is mitigated by periodically adjusting the input from the two trackballs and retaining the results from the trackball that recorded more counts in the meantime. The absolute values ​​of the new position and rotation counts are added in quadrature to determine the trackball that recorded the most counts. In this case, the maximum of the two values ​​is less susceptible to error than the average.

[0141] The real-time endoscopic motion tracking program reads input from the two trackballs by polling them every 8 ms (this depends on the frame rate of the camera, but can be reduced to e.g. 4 ms, 2 ms, 1 ms, etc. for high speed cameras). In at least one embodiment, the 8 ms time value may be the low level USB polling rate used by Linux, for example, if the program is running on an ARM CPU and Linux, and the USB polling rate is negotiated with the trackball by the operating system to be 8 ms. For each trackball, the program receives a time-stamped response if the trackball has recorded movement. If both trackballs measure position and rotation changes, as many as four changes may be recorded during a single polling loop. Each loop that recorded one or more changes is counted and this value is called the Sample Count (n sample ) for a sample count of n rec = 50 (may be changed to a higher count, such as 20, 100, 400, etc., in devices with such capabilities), and then the results from the two trackballs are compared. A sample count of 50 results in approximately 8 corrections being made for a 5 cm displacement. In practice, the sample count should be large enough to allow a statistically better choice between the top and bottom difference between the upper and lower trackballs. The sample count should preferably be small enough that a few corrections are made for each 5 cm segment to reduce the percentage of uncertainty. The sample count typically varies proportionally to the number of counts per 5 cm (approximately 400 to 420 counts in Table 2). This results in a dynamic sampling rate that adjusts the trackball results more frequently when the trackball is moving continuously and does not adjust the trackball when the endoscope is stationary. In use, the count does not increase unless the trackball records movement, and if there is no movement, no corrections are made until movement begins again. [Table 2]

[0142] The best estimate of the endoscope's motion is updated each time the trackball is adjusted, using the result from the more reliable trackball for each time interval. rec =50 can be set to change the frequency of trackball adjustment. During trackball rotation calibration, n rec With a setting of =50, the trackball was adjusted 11 to 13 times per full 360° rotation of the endoscope. The speed of the movement can be calculated by dividing the length / rotation value by time. Similarly, the acceleration of the movement can be calculated by dividing the velocity value by time. And because the data obtained from the trackball measurements is time-stamped, the tracker can also measure the average linear and angular velocity of the endoscope each time the trackball is adjusted. [Table 3]

[0143] The trackball was calibrated by recording the output during operation, including insertion, removal, CW rotation, and reverse CW rotation. This calibration process was performed several times during the tracker's development.

[0144] Position calibration was accomplished using four insertion movements and four removal movements. All insertion movements started at an insertion length of 20 cm and ended at 100 cm. The GIF-XQ10 endoscope has a maximum insertion length of 102.5 cm. The removal movements were the same but reversed. Ground truth points were manually recorded every ±5 cm during each ±80 cm movement for a total of 16 points. Referring to Figure 5, the start, end, and ground truth points are recorded when the corresponding white line is aligned with the left edge of the sensor cuff.

[0145] The position calibrations shown in Table 2 were combined for both insertion and removal as there were no significant differences between them. Four of the ±5 cm movements were outliers that did not accurately track the endoscope movement, so the results include 124 data points (8 movements × 16 movements - 4).

[0146] Table 2 contains the distance conversion factors for the lower trackball, upper trackball, and adjustment position. The upper trackball recorded less movement than the lower trackball for most of the ±5 cm movement. This occurs because of different forces on the two trackballs as the endoscope moves through the sensor cuff. If the friction of the endoscope against the trackball becomes too small, some of the endoscope movement may not be recorded.

[0147] It was experimentally determined that the most accurate tracking was achieved by periodically comparing the motion measured by the two trackballs, taking the maximum of the two measurements, and using that to update the real-time estimate of the endoscope's motion. This allows the endoscope tracker to be calibrated based on the adjusted trackball counts generated during a known calibration run. This approach is justified because the adjusted positions in Table 2 have a smaller relative uncertainty (approximately 3%) than the >5% for a single trackball.

[0148] Rotational calibration was accomplished by completing one movement of +2880° (8 clockwise (CW) rotations) and one movement of -2880° (8 counterclockwise (CCW) rotations). After each rotation (±360°), a ground truth point was recorded by aligning the temporary arrows on the insertion tube and the sensor cuff. There was no significant difference between CW and CCW rotations. The results are shown in Table 3. In at least one embodiment, the calibration coefficients used during actual use are the adjusted trackball values. These may be hard-coded into the motion tracking program for the model of endoscope being used. These calibration values ​​are used to convert counts from the USB trackball (internal units of movement are counts) into actual position change (in cm) and rotation change (in degrees).

[0149] In at least one embodiment, a calibration performed using, for example, a trackball can be used to derive a misalignment factor per 5 cm of insertion and a rotation adjustment factor per 180° of rotation based on how far the actual position and rotation measurements are from the adjusted position and rotation measurements. Thus, after determining these coefficients, they are used to multiply the actual measurements to obtain calibration-corrected position and rotation values. A trackball calibration can then convert the trackball counts for both axes into real-world cm and degrees, respectively. The average amount of slippage can then be addressed by comparing (adjusting) the two trackballs and retaining the larger value during calibration. The same adjustments then allow for reliable measurements during operation. B. Camera Calibration

[0150] The leader and follower cameras were calibrated by taking several images while using the camera settings summarized in Table 1. To remove noise, the camera calibration program saves an average image of a series of 256 images cropped to the endoscope ROI. During the calibration of each camera, three primary images were acquired: a reference image of the background, an image showing the white line feature, and an image showing the white line feature with the black line gap feature facing the camera. Camera calibration may be performed for at least one purpose, such as determining thresholds (lines, gaps, glare, etc.), checking that constants used to crop the camera image to view the endoscope are correct, and / or acquiring images needed for image subtraction.

[0151] Background reference image I ref shows the black part of the endoscope insertion tube and the glare of the LED light reflected on its surface. This glare is similar to the glare shown in Figure 4. ref are captured by inserting the endoscope to a point where no white lines or numbers are visible in the FOV of the camera in the sensor cuff. These images are then loaded by the endoscope tracker 250, which performs background subtraction during real-time feature detection. This enhances the signal from the white line features and helps reduce glare.

[0152] Images showing white line and black line gap features were manually inspected. The white line feature was selected (e.g., as a region of interest in GIMP or ImageJ) and a pixel histogram (constructed from the intensity values ​​of all pixels in the cropped image) was used to select a value of 50 as the threshold for distinguishing the white line from the black background (T line ). The same threshold value of 50 is also used for the black line gap (T gap ) was chosen to detect edges of the image. A glare threshold (T glare) was chosen. In at least one embodiment, there is a background subtraction step in the code that removes glare that occurs in the reference background image in the black parts of the endoscope. However, as the endoscope flexes and moves the sensor cuff, the glare appears to move a few pixels. To attempt to correct for this, a glare threshold is chosen. The values ​​50 and 180 are values ​​that are convenient thresholds for 8-bit values ​​(i.e., values ​​between 0 and 255), where the pixel values ​​from the camera are 8-bit integers. For the camera, 0 represents black and 255 represents 8-bit white. Table 1 shows that the camera's grayscale intensity values ​​are 8-bit values ​​ranging from 0 to 255 for each image pixel. The value 50 is the threshold used to find the white lines (averaged over all pixels in the image column). If the value after background subtraction is greater than 180 (or less than -180), it is considered glare. This means that these pixels are not useful and are set to 0 (no difference) in the post-background subtraction image.

[0153] Finally, the process of imaging the feature is repeated with the line positioned at various locations within the camera's FOV. This step measures how the apparent width of the white line and the height of the black line gap change as you move within the FOV. This value is used to check if the image feature is a valid size close to the expected value, reducing false positives.

[0154] In at least one embodiment, for example, a calibration performed for imaging may then be performed to determine background image pixel values ​​that may be used to perform image subtraction on images acquired during actual use of the endoscope tracker during a procedure. This may include using a background image for background selection, as well as selecting an 8-bit (range 0-255) image processing constant. This may also include setting parameters for cropping, and the correct size range (in pixels) of image features (white lines and black line gaps in white lines). C. Camera feature detection

[0155] The endoscope tracker software is a multi-threaded C++ code, with feature detection using its own dedicated thread. This means that to achieve 90Hz real-time performance, the software must process each image within 11.1ms. The algorithm can be designed to detect valid white line futures and black line gap features when they appear within the camera's FOV. Black line gap features are always found within larger white line features. At this point, image rotation and endoscope ROI cropping have already been completed. The feature detection algorithm has five phases: background subtraction, glare correction, white line detection, black line gap detection, and image feature verification.

[0156] Background subtraction is used to enhance the signal from the white graticule of the endoscope and to compensate for most of the illumination non-uniformity across the camera's FOV. To perform background subtraction, the image pixel (I0) of the acquired image to be processed is converted to a signed 16-bit integer and the result is subtracted from the reference image (I ref ) is subtracted from the intensity value of that pixel, as shown in equation (4). glare Less than or +T glare If the grayscale level is greater than 0, most of the remaining glare is corrected by setting pixels with extreme grayscale values ​​to a value of 0. I1 = int16(I0) - I ref (3)

number

[0157] The algorithm then attempts to detect white line features, if any, that appear in the image. First, the average intensity (μ col ) is calculated as follows:

number

[0158] Then, the values ​​of each column are compared to a threshold T line= 50 to detect white image features. Finally, a one-dimensional (1D) distance transform is used to find the center of the white grid lines in the frame, if any, and determine whether the line width is close to the expected value to determine whether it is valid.

[0159] A similar approach is used to detect possible black line gap features when white line features are found. First, the image is cropped to the ROI of the white line feature. Then the average of the pixel rows is calculated, T gap = 50 to detect black regions. Finally, a 1D distance transform is used to locate the black line gaps and verify their height (e.g., the determined height is close to the actual or expected height). Testing Procedure

[0160] Throughout the development of the endoscope tracker, many trials were completed to test its performance. This section describes the trials that were performed on the prototype endoscope tracker. These trials and their results are an accurate representation of the overall performance of the endoscope tracker. Other trials that were omitted showed similar results, with the white line feature and black line gap feature being largely missed.

[0161] Three position tracking trials were completed to test the position error of the endoscope tracker and how the error relates to the average speed during the movement. Position error is defined as the difference between the measured position and the ground truth position. Each trial includes an 80 cm insertion movement, and an 80 cm removal movement. These movements are very similar to the calibration movements, except that the speed of the movement was changed between trials (slow, medium, fast) and 15 movements per movement (each ±5 cm) were recorded by the camera. The camera data compensates for errors at the center of the box 19 mm from the left edge of the sensor cuff, so one less movement is recorded per movement.

[0162] The data from the position tracking trial can be organized into three groups by considering each movement individually and categorizing them based on average velocity: Group 1 consists of 31 slow-speed movements with velocities between -10mm / s and +10mm / s, Group 2 includes fast-removal movements with velocities between -40mm / s and -10mm / s, and Group 3 includes fast-insertion movements with velocities greater than +10mm / s.

[0163] Two trials were completed to test the device's ability to measure rotational error and rotational speed. Rotational error is defined as the difference between the measured rotation angle and the ground truth. Both trials started with a movement of +1440° (4 CW rotations) followed by a movement of -1440° (4 reverse CW rotations). The movements were half the size compared to during calibration, and the RasPi connected to the two cameras could run software that attempts to correct the accumulated error after every ±180° rotation. result

[0164] Figure 5 plots measured position versus ground truth 500 for a single calibration trial. The inset shows the area where the original displacement before repair 520 has the largest positive and negative error compared to the ground truth. Line 510 shows the ideal case displacement, and line 530 shows the corrected displacement.

[0165] 6 plots the position displacement ground truth versus the position error from the ground truth 600. Line 610 shows the original displacement error and line 620 shows the corrected displacement.

[0166] Figure 7 plots the ground truth of position displacement versus position error (relative to ground truth) 700. Line 710 shows the lower trackball error, line 720 shows the upper trackball error, line 730 shows the best error before correction, and line 740 shows the ideal case of zero displacement error. Figure 7 shows a graph of the output from two trackballs adjusted by 5 cm to create a best combined estimate before correction.

[0167] The rotation counts measured by the two trackballs are adjusted every 50 trackball updates. A single update occurs whenever one or both of the trackballs record a count. The adjusted output is denoted as the Uncorrected Rotation Error. The Corrected Rotation Error uses ground truth data recorded every 360° to correct the accumulated uncertainty to zero. The rotation error is corrected in a manner similar to how lateral insertion errors are corrected.

[0168] Figure 8 plots the ground truth of the rotational displacement versus the rotation error from the ground truth 800. Line 810 shows the original rotation error and line 820 shows the corrected rotation error. Figure 8 shows a graph of the error in tip rotation that can be reduced by correcting on the fly to avoid cumulative error.

[0169] The position outputs from the two trackballs are aligned and a timestamp is recorded after every 25 trackball updates. This allows the average speed to be calculated each time. The best estimate selects the fastest trackball. The tracker is unique because it can measure insertion speed in real time during an endoscopic procedure. This can be done in real time as the speed and acceleration along the insertion axis and the rotation axis are measured, in contrast to less accurate and useful traditional methods that measure tip position and speed / acceleration in random 3D space. The graph on the left shows that the 5cm insertion was done in two separate bursts of movement rather than one smooth movement.

[0170] Figure 9 plots the number of trackball updates versus rotation speed 900. Line 910 shows the lower trackball insertion speed, line 920 shows the upper trackball insertion speed, and line 930 shows the best combined estimate. Figure 9 shows a graph of endoscope insertion speed measured in real time.

[0171] The position tracking results are shown in Figures 10-11. Figure 10 is a graph of the position error vs. average velocity results 1000. Figure 11 shows a graph 1100 in which the position error for each trip in groups 1-3 is plotted against the average velocity.

[0172] In FIG. 10, the arrows indicate the direction of insertion or removal in each plot. No white line features were missed during these trials. FIG. 10 shows the results 1000 of (a) slow insertion 1010, (b) slow removal 1020, (c) fast insertion 1030, and (d) fast removal 1040. In FIG. 10, the white line image feature is defined as the ground truth and is successfully detected by the camera every 5 cm to reset the position error to zero. In the graph, each 5 cm movement is displayed as a sloping line and the error correction as a vertical line. Line 1050 shows the average speed and line 1060 shows the position error.

[0173] The relationship between the position error 1060 and the average velocity 1050 is shown in Figure 10. For both (a) insertion and (b) removal at speeds below 10 mm / s, the position error 1060 is small and distributed around zero. When the speed exceeds 10 mm / s, the position error is usually in the same direction as the direction of advancement, i.e., (c) positive for insertion and (d) negative for removal. This shows that the position error is positively correlated with the average velocity at speeds above 10 mm / s.

[0174] The same trend can be observed in FIG. 11, which plots the position error versus velocity for all three trials 1100. In FIG. 11, the three groups are clearly separated along the x-axis. For group 1 1110, the slope of the line of best t passing through the origin was -0.05±0.04s (p-value=18%), so the slope of the segmental trend line was set to 0. The position error of the movement of group 1 1110 is not correlated with its velocity. This indicates that the device is well calibrated for speeds below 10 mm / s. The trend line for group 2 1120 passes through the point (-10,0) and has a slope of 0.46±0.05s. The trend line for group 3 1130 passes through (10,0) and has a slope of 0.61±0.06s. The quoted uncertainties for the slope of the trend lines 1140 for each group are standard errors.

[0175] The movements of Group 1 1110 were then used to calculate the absolute position error statistics of the endoscope tracker, since the endoscope tracker was well calibrated for speeds ≤ 10 mm / s. The median absolute error (AE) of position tracking was 0.88 mm, or 1.8% of the 50 mm of movement. The 10th percentile AE was 0.19 mm, and the 90th percentile AE was 2.2 mm. The position error of faster movements in Group 2 1120 and Group 3 1130 correlates positively with speed. The maximum position error measured was approximately 10 mm (20%) for speeds < 40 mm / s. During the rotational tracking trials, the device did not perform perfectly. It failed to detect the black line gap image feature three times.

[0176] FIG. 12 is a graph 1200 showing (a) CW rotation tracking data from a first trial 1210, and (b) CCW rotation data from a second trial 1220. The arrows indicate the direction of CW or CCW rotation in each plot. In (a), no black line gap features were missed, while in (b), three black line gap features were not correctly detected as they crossed the center (vertical direction) of the leader or follower image.

[0177] In Fig. 12, we define the image features of the black line gap as the ground truth. In (a), every line gap is detected by the camera (every 180°) to reset the rotation error to zero. Line 1230 shows the rotation error and line 1240 shows the average speed. This means that every time a line gap appears in front of the leader or follower camera, it is tracked correctly. The corrected error is usually around ±10°, and occasionally it can reach up to ±30°. In (b), the camera cannot track the line gap accurately and cannot correct the rotation error three times: 1420°, 1240°, and 700°. In (b), the device is not perfect, but this situation shows that it can self-correct when the next line gap is detected after rotating 360°. The rotation error in this case is still less than 20° even after rotating 360°.

[0178] Device AE statistics for rotational tracking were calculated based on 18 segments of 180° magnitude each collected during two trials. The median rotational AE was 11° (6.8%). The 10th and 90th percentile AEs were 1.8° and 21°, respectively. The device was tested at angular velocities <40° / s. Consideration

[0179] The endoscope tracker embodiments described herein can measure the insertion length (position) and orientation (rotation) of the tubes, as well as their motion, in real time. Such information can be continuously recorded and correlated with video, potentially improving procedure documentation, biopsy site and lesion relocation, and navigation. Such technology can also be used for physician training and quality assurance.

[0180] The endoscope tracker uses a dual-modality approach, where a trackball measures the movement of the endoscope's axis and a camera images scale markings to correct for cumulative errors. The prototype device achieved a median AE of 0.88 mm in position and 11° in rotation, with a 90th percentile AE of 2.2 mm and 21°. This is encouraging, as a paper using MEI technology and computed tomography (CT) to localize points in a colon phantom found that 8 out of 12 points had a location error of more than 10 mm

[16] .

[0181] Localization of the distal tip of an endoscope is a common problem in endoscopy, but the conventional methods used vary widely depending on the application

[16] ,

[20] ,

[21] ,

[22] ,

[23] . Several conventional solutions exist for measuring the location of the distal tip of an endoscope. For example, external imaging modalities including ultrasound

[21] , fluoroscopy

[22] , and CT

[16] ,

[20] ,

[23] have been reported. However, these modalities generally add significant costs to the procedure, and fluoroscopy and CT may add radiation risks.

[0182] A large study of colonoscopy outcomes found that the prevalence of large (>9 mm) tumors / polyps in average-risk screening was 6.6%.

[24] Thus, although bronchoscopic resection of lung cancer may warrant a CT scan, CT is unlikely to benefit patients undergoing screening or surveillance colonoscopy.

[0183] There are also MEI methods

[16] , and hybrid methods that combine MEI with data from endoscopic video or inertial measurement units (IMUs)

[20] ,

[25] . Pentax, Olympus, and Fujifilm offer MEI products that detect and localize in 3D space coils embedded at regular intervals along the insertion tube of the endoscope. The coils provide the basis for showing a 3D image of the insertion tube and any loops or bends that may have formed in the colon. Information about loops and bends is useful for navigating the endoscope in the colon, especially in training. However, MEI usually requires modification of the endoscope itself (i.e., purchase of a new scope or accessories), and each manufacturer uses proprietary technology

[12] ,

[26] ,

[27] .

[0184] Moreover, these techniques mainly focus on measuring the absolute 3D position of the endoscope tip. Although such information is important, the ability of these techniques to take into account the insertion length from the orifice and the orientation of the lesion is limited. During endoscopy, especially colonoscopy, the absolute 3D position of the colon (and its segments) changes within the abdominal cavity

[16] . For example, even during the same procedure, if the patient's position changes, the 3D position of the scope tip will change with the movement of the colon, but the tip may not move inside the colon itself. Similarly, such 3D positions are not as useful as compared to follow-up procedures.

[0185] Finally, wireless capsule endoscopes can be tracked by combining IMU data with a camera that uses visual odometry to estimate distance traveled.

[28] However, the cumulative error is large and requires an external device to correct for it.

[0186] In contrast to conventional equipment and techniques, the dual modality approach described herein for locating the tip of an endoscope has been found to have less error than existing solutions and is suitable for endoscopy. At least one embodiment of the endoscope tracker described directly herein measures the movement of the insertion tube of the endoscope outside of the patient, which correlates with the insertion and rotational movement of its distal tip. This correlation should occur because the majority of commercially available gastrointestinal endoscopes have insertion tubes that are rotationally stiff and of a fixed length.

[0187] Existing solutions focus on localizing the tip of the endoscope in absolute 3D space, with little consideration of its relative position within the colon. For example, a hybrid bronchoscope tracking method using CT, MEI, and endoscopic video achieved an error of 2.5 mm and 4.7°

[20] , which is sufficient for clinical practice. However, the prototype of the endoscope tracking technique described herein had a 90th percentile error of 2.2 mm and 21°. The positional accuracy is comparable, and the rotational error is sufficient for colonoscopy. For example, when the location of a polyp in an image is described using a 12-hour clock face, the 20° accuracy of the prototype of the endoscope tracking technique described herein is lower than the typical 1-hour / 30° error used in clinical practice.

[0188] The endoscope tracking techniques described herein can be used alone, but can also be combined with other tracking solutions to create more accurate hybrid solutions that measure the movement of the endoscope both outside the patient and at its distal tip camera.

[0189] We found that the rotation (or orientation) of the endoscope tip provides additional location information of the biopsy / lesion site, useful for loop detection and / or avoidance

[29] , as well as improved navigation. Current commercially available 3D positioning techniques cannot measure rotation. Bernhardt et al. reported tracking of rotational motion using intraoperative CT images

[23] . In most endoscopic procedures, radiation from CT should be avoided whenever possible. Other previously reported techniques used endoscopic video

[16] ,

[23] and IMU motion sensors

[16] ,

[25] ,

[29] . These approaches are limited to applications in bronchoscopy, where endoscopic images have significantly more landmark features

[16] or have significantly more accumulated error than colonoscopy

[25] ,

[29] .

[0190] Second, the results described above (e.g., Figures 5 and 6) also demonstrated that insertion speed can be calculated and recorded from real-time insertion length measurements. Insertion speed is not typically measured in clinical practice, but is an important parameter for training new endoscopists and for quality assurance. Such applications do not typically require regulatory approval, opening a more rapid translation path. Thus, in at least one embodiment, feedback that may be provided to the user may include insertion speed. This may be displayed in real-time during the procedure and / or provided as a post-procedure report after the procedure is completed.

[0191] Third, gastrointestinal endoscopy screening and surveillance, especially colonoscopy, is a population-wide program with high volume. When new technologies are developed, both the cost of equipment / facilities and access to qualified endoscopists should be considered. The endoscope tracker technology described herein is designed to be an add-on adjunct device compatible with most commercially available endoscopes. It can be used with existing endoscopes after going through a separate regulatory approval process. This add-on approach may significantly reduce the barriers and costs of technology conversion when using the new endoscope tracker technology described herein. Notes and motivation

[0192] Colonoscopy is the gold standard for detecting CRC. However, tumor location (colon section) reported by preoperative colonoscopy is still inaccurate in 10–17% of patients

[12] . It is generally difficult for endoscopists to accurately record the location of tumors and polyps found during colonoscopy. Colon polyp surveillance involves monitoring patients at high risk for polyp development by regular colonoscopy. Improved polyp localization would allow the presence of recurrent polyps at the site of polyp removal during subsequent colonoscopies. It should be appreciated that other endoscopic procedures (e.g., EGD, bronchoscopy) face similar challenges.

[0193] In clinical practice, insertion and rotation speeds vary widely depending on the stage of the endoscopic procedure and even more so among endoscopists, however, there are no existing techniques that can objectively measure the speed of such movements, so these observations are anecdotal and qualitative.

[0194] The real-time endoscopic motion tracker described herein is an improvement over previous approaches. An externally mounted device records the movement of the endoscope in real time, with the goal of improving the accuracy of lesion localization, for example, during preoperative colonoscopy. The device uses a trackball to measure the position and rotation of the endoscope and uses camera-based feature detection to periodically correct the uncertainty accumulated in the measurements. The associated velocity is calculated each time the camera-based feature detection corrects the position or rotation of the endoscope. A prototype device based on the teachings herein was found to accurately measure the movement of the shaft of the endoscope, with a median position AE of 0.88 mm, and a median rotation AE of 11°. The real-time motion recording generated by the endoscope tracker can also be a valuable tool both for clinical endoscopy processing and to answer several research questions. For example, it can be synchronized with recorded video frames, resulting in a more accurate recording of the location of lesions during colonoscopy. The recorded position and tip orientation (rotation) can be combined with the video to create a 3D reconstruction of the colon or an unwrapped map of the colonic luminal surface

[30] ,

[31] . This technology can also provide an objective and quantitative measure of endoscopic procedures to new residents during their training.

[0195] Alternatives to the motion trackers mentioned above could offer additional advantages: Because the trackball comes into direct contact with the endoscope, the tracker must be thoroughly cleaned and disinfected after use; a non-contact, fully camera-based tracker could eliminate the need for disinfection.

[0196] 13A, a perspective view of one embodiment of an endoscope tracker 1310 that can measure the movement of an endoscope in real time via a fixed sensor device is shown. The endoscope passes through the center of the endoscope tracker 1310, and three cameras 1310a, 1310b, 1310c are attached to three separate distal ends of a star-shaped (e.g., Y-shaped) device. In the implementations of the endoscope tracker 1310 described herein, some or all of the system 100 can be used to perform any one or more functions that require computer functions (e.g., digital input, digital output, software processing, image processing, etc.). Thus, the tracker 1310 includes at least one processor as in the various embodiments described for the system 100, except that there are three cameras and therefore there can be two followers.

[0197] Figure 13B shows a photograph of one embodiment of a holder 1320 for an endoscope tracker 1310 (or "endoscopic tracker system"). Figure 13C shows a perspective view of one embodiment of a holder 1330 for an endoscope tracker 1310. Figure 13D shows a close-up photograph of one embodiment of an inner casing 1340 of the endoscope tracker 1310. The holder 1330 can have a casing 1340 with spaced apart distal ends and a central hole in the casing 1340 for receiving an endoscope having an insertion tube.

[0198] The sensor device measures position and rotational changes over time using at least two cameras (e.g., two, three, four, or more cameras) that cover a full 360-degree field of view around the transverse axis of the endoscope insertion tube 305. For example, three cameras 1310a, 1310b, 1310c can be used, with each camera 1310a, 1310b, 1310c fixed at a position 120° relative to the longitudinal axis of the endoscope insertion tube and having an unobstructed view of the markings on the insertion tube and its sheath. Also, for example, the at least two cameras may be mounted on one of the distal ends such that each of the cameras faces the center of the casing 1340 and each of the cameras has an unobstructed view of the endoscope when received in the central hole of the casing 1340.

[0199] This design of this embodiment contrasts with other designs that use contact points (e.g., trackballs) that physically contact the insertion tube and directly measure its changes in position and rotation. In this design, movement can be measured optically, non-contact, by cameras 1310a, 1310b, 1310c. However, in alternative embodiments, two of the three cameras 1310a, 1310b, 1310c can be used, as described below.

[0200] In at least one implementation of this embodiment, the cameras are configured to capture video of the endoscope (or a portion thereof, such as the endoscope's sheath (e.g., endoscope insertion tube)) moving relative to a fixed field of view (FOV) of the cameras 1310a, 1310b, 1310c. Each frame of the camera video is processed in real time after capture. The endoscope tracker 1310 software tracks the apparent movement of white markings that appear on the black surface of the endoscope's insertion tube. Specifically, most commercially available endoscopes include white line markings that appear at regular intervals (e.g., every 5 cm) along the length of the endoscope's insertion tube. The software uses these to correct for any cumulative errors that accumulate in the measurements when large position changes occur. The line markings also have a black gap at one point on their circumference to allow for correction of rotational errors. The software and corresponding functions performed may be similar to that described with reference to endoscope tracker 250, for example, but may be modified for use without a trackball, in which case position and / or rotation measurements may be refined to a scale corresponding to one or more pixels (e.g., 1 pixel, 2 pixels, 5 pixels, 8 pixels, etc.).

[0201] In at least one implementation of this embodiment, the endoscope tracker 1310 executes program instructions for tracking the position and orientation of the distal end of the endoscope in real time based on images acquired from a portion of the endoscope within the FOV of the at least two cameras. The program instructions may include, for example, (a) acquiring real-time images from the at least two cameras of a portion of the endoscope within the FOV of the at least two cameras, (b) measuring lateral and rotational movement of the endoscope based on markings printed on the endoscope and captured in the real-time images, and (c) correlating the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movement to determine the position and orientation of the distal end of the endoscope.

[0202] 17 is a flow chart of an example embodiment of a method 1700 for tracking an endoscope using a non-contact tracker. Method 1700 may be implemented using some or all of system 100 and / or some or all of endoscope tracker 1310.

[0203] At 1710, video of the endoscope's movement is captured by the cameras as the endoscope moves relative to the fixed field of view of the cameras, thereby generating image data (or video data) of the endoscope. Image features may be obtained as pixel positions (or pixel values) for each camera individually.

[0204] At 1720, the apparent movement of the endoscope is tracked using markings (e.g., scale markings) displayed on the insertion tube of the endoscope. Alternatively, or additionally, pixel values ​​may be transformed to calibrate position and rotation values ​​based on geometry and distance from the endoscope to the cameras. For example, a three camera Y-configuration provides more overlap, making geometry-based methods more reliable.

[0205] At 1730, the cumulative error accumulated in the apparent motion measurements is corrected. Correcting the cumulative error (or uncertainty) may be accomplished by one or more of the position error correction or rotation error correction methods described herein. For example, image data of the endoscope taken from cameras (e.g., two cameras facing each other, three cameras in a Y-configuration, four cameras in a diamond configuration, etc.) and image data of scale markings along the insertion tube may be used to correct for position and rotation errors.

[0206] At 1740, an output of the movement of the endoscope outside the patient or training phantom is provided (eg, displayed).

[0207] The endoscope tracker 1310 functions similarly to a two-camera setup with two trackballs, but may use some or all of the methods, functions, and / or programs described above as they relate to images acquired from a camera that has been modified to function without the trackballs.

[0208] In a non-contact (or camera only) endoscope tracker, the motion tracking program can measure position using images that contain "white line" and "black line gap" features. The motion tracking program can estimate the current position (e.g., measured changes in position and rotation angle) using the "history" of the endoscopic procedure, e.g., the distal tip enters the orifice, then the remainder of the endoscope is inserted, and upon removal, the proximal end exits first, then the distal end. In practice, the "black line gap" image feature is found within the "white line" image feature, and the "black line gap" can only exist if it is part of a larger "white line" image feature. Position and rotation calculation and calibration

[0209] The endoscope tracker 1310 may employ a non-contact (or camera-only) endoscope motion tracking program that calculates and calibrates position and rotation by executing some or all of the following program instructions. (1) Use a single useful viewpoint camera model to determine the distortion parameters for each camera. These parameters are likely to be similar for a given make and model of camera unit, assuming they have not been modified since leaving the factory. (2) The images from each camera are rectified using the camera model parameters. The camera scenes can be reprojected onto a screen at a fixed distance from the camera, equal to the distance between the endoscope and the camera. (3) Using trigonometry and the known distance from the endoscope to the camera and / or the known diameter of the endoscope, convert pixel distances in the undistorted image to distance units (e.g., cm or mm) for position tracking using the white line. Similar triangles (with the same angles) can be created using trigonometry and real-world distances to convert from pixels to real-world units. (4) Using the known diameter of the endoscope and the known distance from the endoscope to the cameras, find the center of the black line gap in pixel coordinates of at least one camera and use trigonometry to find the corresponding angle in degrees for rotational tracking, approximating the surface of the endoscope as a cylinder within the FOV of the cameras. (5) Record the starting position (e.g., cm or mm) and update the position in real time using the position tracking information from the multiple cameras. If the position tracking information from different cameras usually matches, average the results from the cameras using an appropriate algorithm. The position tracking information from different cameras "match" if they appear to reflect the same actual measurements without significant measurement errors or (time) delays between the measurements. In at least some cases, the measurements (from the cameras) are kept close compared to the measurement resolution of the device. As an example, assume that the linear distance in mm per count of the endoscope tracker is about 0.12 mm. In this case, a difference in measurements significantly greater than 0.12 mm indicates a mismatch. In this case, the angular distance in degrees per count may be about 1.3°. These values ​​represent the limits of resolution in mm and degrees (°) for a particular case. The resolution in a particular implementation may depend, for example, on the number of pixels in each camera, the distance to the endoscope, and whether the image is sharp enough to measure the position to an accuracy of one pixel (or a small number of pixels). If the position tracking information is noisy and / or the cameras are inconsistent about the position of the endoscope, a Kalman filter (or similar algorithm) may be included. For example, an average may be used, in which case the filter is a mean filter. Another option that works well with noise is a median filter, which takes the median of the last N measurements rather than the average, i.e., the median value of three cameras, for example. A Kalman filter takes a series of measurements taken over time from one or more noisy signals and models the change in the actual measured values ​​over time. Kalman filters are commonly used for noisy data when it is desirable to process a smoother output signal. (6) Record the starting rotation angle in degrees and update the rotation in real time using rotation tracking information from multiple cameras. That is, the recording can be done "intelligently" for the purpose of filtering outliers, erroneous data, empty data, etc., if the image processing algorithm is confused by other elements in the FOV. The vertical center (on the y-axis) of one camera can be used as the origin for the rotation angle measurement. An offset of 360 / n degrees can be applied between adjacent cameras, where n is the number of cameras used. For camera-only tracking, the number of cameras can be a minimum of two, but three or more may give better results. If black line gaps appear in the FOV of multiple cameras, average the rotation results of those cameras. A Kalman filter may be required to remove noise. (7) In a three-camera setup, the program uses other markings on the surface of the endoscope tube (e.g., 5 cm, 10 cm, 20 cm, etc.) in addition to the white lines and gaps. In at least some cases, the markings (e.g., 20 cm) clearly indicate the position to which the lines next to it correspond. This can be used to start recording from any insertion length (position) in the colon, to restart recording when errors occur, to ensure that there are no errors in the position tracking, and / or to correct errors in the position tracking that may have occurred due to an obstacle between the camera and the endoscope or for other reasons. Image recognition algorithms can be used to recognize these and compare them with prior knowledge (e.g., procedure history, medical device data, video data, image data) about the markings on a particular type of endoscope and their positions. The image recognition algorithm can be, for example, a text recognition algorithm such as optical character recognition (OCR). There are several image recognition techniques that can be used to detect image features, and as known to those skilled in the art, the open source computer vision library OpenCV is one example of an image processing technique that can be used. For example, an image recognition algorithm can crop out the necessary portions of the image, determining the average intensity of columns (for white lines) or rows (for black line gaps) of the image as needed, and then the algorithm can take the row or column average of the image or portion thereof. The parameters used for cropping can be determined empirically.

[0210] In at least one implementation of endoscope tracker 130 (e.g., when used with system 100), the ROI in the images acquired by the camera may be detected using, for example, an edge detector, a rectangle detector, or a blob detector. Once the endoscope is found in one frame, techniques such as active contouring may be employed. Starting from the ROI in the last frame, active contouring (e.g., edge detection) may find a new ROI in the new frame.

[0211] One advantage of the endoscope tracker 1310 (e.g., when used with the system 100) is to improve patient outcomes in medical endoscopy procedures. Background work has focused on improving colonoscopy, particularly for colon cancer treatment. The endoscope tracker 1310 allows for precise localization of abnormal or cancerous colon tissue and polyps by improving localization to within 1 mm, for example, compared to rough estimation of location using 5 cm markers. This allows for measurement of disease progression and treatment efficacy at specific sites during a series of colonoscopy procedures. Improved localization also allows for better treatment options for larger tumors and better planning of minimally invasive surgery to remove the tumors.

[0212] Another benefit of the endoscope tracker 1310 (e.g., when used with the system 100) is that it improves the training process for new endoscope operators. It can be used to monitor how smoothly a trainee inserts and moves the endoscope. It can also record how an expert operator completes a colonoscopy procedure, providing a useful benchmark for trainees.

[0213] In at least one implementation of the endoscope tracker 1310 (e.g., when used with the system 100), position (or lateral) and rotational information (or data) from the endoscope tracker 1310 can be combined with live video from a camera at the tip of the endoscope to determine if a loop is being formed by the endoscope's insertion tube, or to generate an image mosaic that maps the entire inner surface of the colon to identify the location of tumors and polyps.

[0214] In at least one implementation of the endoscope tracker 1310, the system 100 measures position and rotation and corrects for position errors by imaging white lines that are present at regular intervals (e.g., every 5 cm) along the endoscope. Rotational correction can be omitted because the black gaps in each white line appear only on one side of the endoscope. Thus, the gaps are often hidden from the view of one camera. Modifications to the implementation (e.g., changing the color of the lines, changing the spacing, providing rotational correction) can be made as desired or required to achieve the same results.

[0215] One advantage of the endoscope tracker 1310 (e.g., when used with the system 100) is that it can be used to provide training to endoscope operators. It can monitor how smoothly a trainee inserts and moves the endoscope into a training phantom or patient. This can quantify how their technique differs from that of their peers, or even from expert operators who have completed many endoscopic procedures.

[0216] Another advantage of the endoscope tracker 1310 (e.g., when used with the system 100) is that it may be beneficial in more complex applications, such as clinical use. The endoscope tracker may be able to more accurately locate abnormal or cancerous colon tissue and polyps during a colonoscopy compared to traditional methods. Better localization may allow for more or different treatment options. It may also improve planning of minimally invasive surgery to remove large tumors in the colon by providing better definition of the tumor's location.

[0217] In at least one implementation of the endoscope tracker 1310, software for the endoscope tracker determines how the endoscope moves over time. The software uses an image processing pipeline, and each captured camera frame includes a timestamp. The time required for each piece of the tracker's logic can be measured and compared to the time (e.g., 11 ms) used to acquire each frame (e.g., 90 fps).

[0218] At least one implementation of the endoscope tracker 1310 uses a two-camera setup. With two cameras, a delay in recording frames from the second camera can cause discrepancies between the cameras. Therefore, the system is programmed to compensate for the delay.

[0219] At least one implementation of the endoscope tracker 1310 uses a free-standing design without a "cuff." For example, this configuration may allow for a completely free-standing camera. In such a case, a calibration procedure can register the relative position and orientation between the camera and the endoscope to make measurements.

[0220] In at least one implementation of the endoscope tracker 1310, the endoscope tracker 1310 records the movement of the endoscope outside the patient and infers how the tip of the endoscope may have moved over the same time. This information can be combined with live video from a camera at the endoscope tip to determine if loops are being made by the endoscope or to generate an image mosaic that maps the entire inner surface of the colon to identify the location of tumors and polyps. The map can be unwrapped to display a single image summary of the colon, reducing the number of times the full video from the endoscope camera needs to be viewed. Clicking on a feature of interest in the unwrapped map can jump to the timecode of the video in which the feature was displayed.

[0221] In at least one implementation of the endoscope tracker 1310, there may be additional motion sensors on the endoscope, additional motion sensors on the operator, and / or ambient sensors attached to the endoscope to improve motion tracking.

[0222] In at least one implementation of the endoscope tracker 1310 (e.g., when used with system 100), the optical system design (i.e., two or more cameras) has an unobstructed view of the endoscope insertion tube and has a lighting design to minimize reflections.

[0223] Another advantage of the endoscope tracker 1310 (e.g., when used with system 100) is that when used with system 100, it provides algorithms that correlate real-time images from multiple cameras, calculate pattern changes, and determine position.

[0224] Another advantage of the endoscopic tracker 1310 (e.g., when used with system 100) is that it may provide better coverage using three cameras (or more cameras) of a 360° view, as compared to a two-camera setup that may also provide a 360° view, but where the two-camera setup may have lower resolution (or image processing) at the ends of the FOV.

[0225] At least one implementation of the endoscope tracker 1310 (e.g., when used with system 100) uses one or more cameras (e.g., three cameras) external to the endoscope that do not touch the endoscope and capture images of the endoscope that are used to detect movement of the scope.

[0226] Another advantage of the endoscope tracker 1310 (e.g., when used with system 100) is that it does not require a contact point such as a trackball. This eliminates issues with physical contact between the trackball and the endoscope, which would require additional cleaning of both the endoscope and the tracker device. The non-contact design also allows for future modifications that may use a freestanding design without a "cuff."

[0227] Yet another advantage of the endoscope tracker 1310 (e.g., when used with system 100) is reduced error compared to contact-based sensors. Technically, mechanical sensors (trackballs) create cumulative error. In the absence of contact, any error is corrected in real time by the video-based sensor.

[0228] While applicants' teachings have been combined with various embodiments for illustrative purposes, it is not intended that applicants' teachings described herein be limited to such embodiments, as the embodiments described herein are intended to be examples. Rather, applicants' teachings as described and illustrated herein encompass various alternatives, modifications, and equivalents without departing from the embodiments described herein, the general scope of which is defined in the appended claims. References [Table 4] TIFF2025072849000010.tif219159

Claims

1. a sensor cuff having a housing; a first trackball disposed within a first side of the housing, and a second trackball disposed within a second side of the housing and spaced apart from the first trackball to form a gap between the first and second trackballs for receiving the endoscope such that the endoscope contacts the first and second trackballs; a reader camera disposed adjacent a third side of the housing and directed toward the gap of the endoscope from a first direction; a follower camera disposed adjacent a fourth side of the housing and directed toward the gap of the endoscope from a second direction different from the first direction; at least one computing device including a non-transitory computer readable medium storing program instructions that, when executed by the computing device, cause the computing device to track the position and orientation of the endoscope in the sensor cuff in real time; A real-time endoscope tracker system comprising:

2. The system of claim 1 , wherein the at least one computing device comprises a leader computer and a follower computer, and the first trackball and the second trackball are connected to the leader computer.

3. a first light source for the reader camera; a second light source for the follower camera; The system of claim 1 or 2, further comprising:

4. The system of claim 1 , wherein the first trackball and the second trackball measure changes in the longitudinal insertion position of the endoscope over time and changes in the rotation of the endoscope over time.

5. 5. The system of claim 1, wherein the insertion position of the endoscope is measured using a first axis of the first trackball and the second trackball, and rotation of the endoscope is measured using a second axis perpendicular to the first axis.

6. 6. The system of claim 1, wherein cumulative uncertainty in measurements made by the first and second trackballs is mitigated by using the leader and follower cameras to detect white scale marks that appear at regular intervals on an insertion tube portion of the endoscope.

7. 7. The system of claim 1, wherein the leader camera and the follower camera provide image data used to correct rotation by a predetermined number of degrees based on detection of markings from opposite sides of the endoscope.

8. 8. The system of claim 5, wherein the leader computer is connected to the leader camera and the follower computer is connected to the follower camera.

9. The system of claim 8 , wherein the reader computer is configured to combine all sensor data to calculate the position of the endoscope, the rotation of the endoscope, and the movement of the endoscope.

10. 10. The system of claim 1, wherein the at least one computing device, when executing the program instructions, is configured to use white scale lines to correct accumulated error in the insertion length of the endoscope and to use black line gaps to correct the rotation angle of the endoscope.

11. 11. The system of claim 1, wherein the at least one computing device, when executing the program instructions, is further configured to combine data regarding the lateral and rotational movement with live video from at least one of the leader camera or the follower camera to determine whether a loop is formed by an insertion tube of the endoscope.

12. 12. The system of claim 1, wherein the at least one computing device, when executing the program instructions, is further configured to combine data regarding lateral and rotational movement with live video from at least one of the leader camera or the follower camera to generate an image mosaic that maps the inner surface of the colon being imaged to identify the location of tumors and polyps.

13. 13. The system of claim 1, wherein the at least one computing device, when executing the program instructions, is further configured to compensate for delays in recording frames from one of the leader camera or the follower camera.

14. 14. The system of claim 1, wherein the at least one computing device, when executing the program instructions, is further configured to detect movement of the endoscope from images acquired from the leader camera and the follower camera using at least one of computer vision, machine learning, or artificial intelligence.

15. The system of any one of claims 1 to 14, wherein the at least one computing device, when executing the program instructions, is further configured to integrate at least one of the position or the orientation of the endoscope into an endoscopy training simulator.

16. a holder having a casing having spaced apart distal ends and a central bore in said casing for receiving an endoscope having an insertion tube; at least two cameras, each of the cameras facing the center of the casing and each of the cameras mounted on one of the distal ends such that each of the cameras has an unobstructed view of the endoscope when received in the central bore of the casing; A processor in communication with a non-transitory computer readable medium storing program instructions that, when executed by the processor, cause the processor to track a position and orientation of a distal tip of the endoscope in real time based on images acquired from a portion of the endoscope that is within a field of view (FOV) of the at least two cameras, wherein when the processor executes the program instructions: acquiring real-time images from the at least two cameras of the portion of the endoscope within the FOV of the at least two cameras; determining lateral and rotational movement of the endoscope based on markings printed on the endoscope and captured in the real-time image; correlating the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movements to determine the position and orientation of the distal end of the endoscope; a processor and A real-time endoscope tracker system comprising:

17. 17. The system of claim 16, wherein the at least two cameras cover a full 360 degree field of view of the insertion tube of the endoscope about a lateral axis of the insertion tube to measure lateral and rotational movement of the endoscope.

18. 18. The system of claim 16 or 17, wherein the at least two cameras comprise three cameras, each of the three cameras fixed at a point such that two consecutive cameras are located 120° to an axis of the insertion tube to provide an unobstructed view of the insertion tube and markings thereon.

19. 19. The system of any one of claims 16 to 18, wherein the at least two cameras are configured to capture video of the endoscope as the endoscope moves relative to a fixed FOV of the at least two cameras.

20. 20. The system of claim 16, wherein the processor, when executing the program instructions, is further configured to identify white line markings along a length of the insertion tube to correct for accumulated errors accumulated in measurements.

21. 21. The system of claim 18, wherein the processor, when executing the program instructions, is further configured to identify black gaps at points around a circumference of the insertion tube to correct for rotational errors.

22. 22. The system of any one of claims 16 to 21, wherein the processor, when executing the program instructions, is configured to combine data regarding the lateral and rotational movement with live video from a camera at the tip of the endoscope to determine whether a loop is being formed by the insertion tube.

23. 23. The system of any one of claims 16 to 22, wherein the processor, when executing the program instructions, is further configured to combine data regarding lateral and rotational movement with live video from a camera at the tip of the endoscope to generate an image mosaic that maps the inner surface of the colon being imaged to identify the location of tumors and polyps.

24. 17. The system of claim 16, wherein the at least two cameras comprise two cameras, and the processor, when executing the program instructions, is further configured to compensate for a delay in recording a frame from one of the two cameras.

25. 25. The system of any one of claims 16 to 24, wherein the processor, when executing the program instructions, is further configured to use computer vision to detect movement of the endoscope from images acquired from the at least two cameras.

26. 26. The system of any one of claims 16 to 25, wherein the processor, when executing program instructions, is further configured to provide at least one of the position or the orientation of the distal end of an endoscope to an endoscopy training simulator.

27. 1. A method for tracking an endoscope using an endoscope tracker, the method comprising: Fixing a sensor cuff of the endoscopic tracker to a position outside a patient or a training phantom; manually inserting the endoscope through an entrance hole and an exit hole of the sensor cuff; positioning an insertion tube of the endoscope between a trackball and a camera of the endoscope tracker; measuring a position of the endoscope using the y-axis data from each trackball and measuring a rotation of the endoscope using the x-axis data from each trackball; correcting for the positional and rotational uncertainties of the endoscope using image data from a camera; providing an output of the movement of the endoscope outside a patient or training phantom; The method includes:

28. connecting the trackball to one of a plurality of cameras; placing or positioning a camera to view the endoscope from an opposing end of the sensor cuff; Measuring the rotation of the trackball; capturing an image with the camera indicative of markings along an insertion tube of the endoscope; correcting position and rotation errors of the endoscope using the images; Providing an output of the movement of said endoscope outside a patient or training phantom; A method for tracking an endoscope comprising:

29. Polling the trackball at regular intervals to update the endoscope movement; acquiring images by a camera and detecting image features of the endoscope in real time; combining the trackball data with the image data to update a real-time estimate of the endoscope's motion; correcting positional and rotational errors using said image features; Providing an output of the movement of said endoscope outside a patient or training phantom; A method for tracking an endoscope comprising:

30. capturing video of the movement of the endoscope with a camera as the endoscope moves relative to a fixed field of view of the camera; tracking the apparent movement of the endoscope using markings displayed on an insertion tube of the endoscope; correcting cumulative errors accumulated in the measurements of apparent motion; Providing an output of the movement of said endoscope outside a patient or training phantom; A method for tracking an endoscope comprising:

Images