System and method for real-time, intraoperative, endoscopic shaft motion tracking

Abstract

An endoscope tracker is provided that measures the motion of an endoscope using a pair of trackballs and two video cameras. The tracking ball measures the position and rotation changes of the endoscope in real time in the endoscopic surgery process. The camera detects scale marks on the surface of the endoscope to correct accumulated trackball measurement errors. The bimodal tracking method measures the movement of the endoscope insertion tube in real time, including insertion length, rotation angle and speed thereof. The optical trackball measures the movement of the endoscope insertion tube, and the camera corrects the cumulative error. Also provided is a pure optical endoscope tracker that measures endoscope motion that uses a video camera and software that processes images obtained from the video camera to determine how the endoscope moves over time.

Description

Technical Field

[0001] Various embodiments are described herein that generally relate to systems and methods for medical diagnostic imaging. More specifically, the present invention relates to real-time, intraoperative, endoscopic axis motion tracking, such as insertion length and orientation tracking. Background Art

[0002] The following paragraphs are provided by way of background to the present disclosure. However, they are not an admission that anything discussed therein is prior art or part of the knowledge of a person skilled in the art.

[0003] Gastrointestinal endoscopy is the gold standard for detecting and characterizing gastrointestinal abnormalities. For example, upper endoscopy (esophagogastroduodenoscopy: EGD) is used to evaluate precancerous lesions such as Barrett's esophagus or esophageal cancer, and colonoscopy is used to detect precancerous polyps and cancer in the large intestine. In the United States, 1-2% of the adult population has Barrett's esophagus, and colorectal cancer is the third most commonly diagnosed cancer in men and women, with an estimated 150,000 new cases and 53,000 deaths in 2021 [1], [2]. Although current endoscopic practices are very effective in detecting cancer or precancerous lesions, they are less reliable in localizing lesions [3]-[7]. This is a problem for surveillance of Barrett's esophagus or colon polyps and for resection of esophageal or colon cancer. Barrett's esophageal lesions or colon polyps must be localized during endoscopic resection so that the same site can be examined for recurrent lesions during the next endoscopic procedure. Esophageal or large colorectal tumors must be well localized by preoperative endoscopy to facilitate planning of surgical resection.

[0004] One of the challenges in endoscopy is the use of endoscopic motion trackers to improve the accuracy of lesion localization during preoperative endoscopy. A retrospective review (2013-2016) found that preoperative colonoscopy was inaccurate in 16.7% of patients [3]. In 15 high-quality studies, colonoscopy incorrectly localized the tumor in 13.7±3.6% (95% CI) of patients, and the inaccuracy of colonoscopy tattooing was 6.5±3.4% [4]. In addition, in other studies, incorrect localization required a change in the surgical plan for 5.3-11% of tumors [5]-[7].

[0005] Magnetic endoscopic imaging (MEI) currently exists that is able to detect loops formed along the endoscope insertion tube or shaft during colonoscopy. While MEI may help improve localization accuracy, this requires a new endoscope or an accessory with a magnetic coil to support motion tracking and pose estimation [4], [8], [9]. In addition, MEI cannot localize lesions during upper gastrointestinal endoscopy and cannot accurately describe the extent or location of Barrett's esophagus lesions.

[0006] What is needed is a system and method that addresses the above-mentioned challenges and / or shortcomings through real-time, intraoperative, endoscopic axis motion tracking. Summary of the invention

[0007] Various embodiments of systems and methods for real-time, intraoperative, endoscopic axis motion tracking and computer products for use therewith are provided in accordance with the teachings herein.

[0008] According to one aspect of at least one embodiment of the present invention, a real-time endoscope tracker system is disclosed, comprising: a sensor sleeve having a housing; a first tracking ball disposed within a first side of the housing and a second tracking ball disposed within a second side of the housing and spaced apart from the first tracking ball, a gap for receiving an endoscope being formed between the first tracking ball and the second tracking ball so that the endoscope contacts the first tracking ball and the second tracking ball; a leader camera disposed adjacent to a third side of the housing and facing the gap for the endoscope from a first direction; a follower camera disposed adjacent to a fourth side of the housing and facing the gap for the endoscope from a second direction different from the first direction; and at least one computing device comprising a non-transitory computer-readable medium storing program instructions, which, when executed by the computing device, cause the computing device to track in real time the position and orientation of the endoscope at the sensor sleeve.

[0009] In at least one embodiment, the at least one computing device includes a leader computer and a follower computer, and the first trackball and the second trackball are connected to the leader computer.

[0010] In at least one embodiment, the system further includes: a first light source for the leader camera; and a second light source for the follower camera.

[0011] In at least one embodiment, the first trackball and the second trackball measure changes in the longitudinal insertion position of the endoscope over time and changes in the rotation of the endoscope over time.

[0012] In at least one embodiment, a first axis of the first and second trackballs is used to measure insertion position of the endoscope, and a second axis perpendicular to the first axis is used to measure rotation of the endoscope.

[0013] In at least one embodiment, accumulated uncertainty in the measurements of the first and second trackballs is mitigated by using the leader and follower cameras to detect white tick marks that appear at regular intervals on the insertion tube portion of the endoscope.

[0014] In at least one embodiment, the leader camera and the follower camera provide image data for correcting a predetermined degree of rotation based on detection of markers from opposite sides of the endoscope.

[0015] In at least one embodiment, the leader computer is connected to the leader camera and the follower computer is connected to the follower camera.

[0016] In at least one embodiment, the leader computer is configured to combine all sensor data to calculate the position of the endoscope, the rotation of the endoscope, and the motion of the endoscope.

[0017] In at least one embodiment, the at least one computing device is configured to use the white scale lines to correct for cumulative errors in the insertion length of the endoscope and use the black line gaps to correct for the rotation angle of the endoscope when executing program instructions.

[0018] In at least one embodiment, the at least one computing device is further configured to, when executing program instructions, combine data regarding lateral and rotational movement with real-time video from at least one of the leader camera or the follower camera to determine whether the insertion tube of the endoscope is forming a loop.

[0019] In at least one embodiment, the at least one computing device is further configured to, when executing program instructions, combine data regarding lateral and rotational motion with real-time video from at least one of the leader camera or the follower camera to generate an image mosaic that maps the inner surface of the colon being imaged to locate tumors and polyps.

[0020] In at least one embodiment, the at least one computer device is further configured to, when executing program instructions, compensate for a delay in recording a frame from one of the leader camera or the follower camera.

[0021] In at least one embodiment, the at least one computer device is further configured to, when executing program instructions, use at least one of computer vision, machine learning, or artificial intelligence to detect movement of the endoscope based on images obtained from the leader camera and the follower camera.

[0022] In at least one embodiment, the at least one computer device is further configured to, when executing the program instructions, integrate at least one of the position or orientation of the endoscope into the endoscopy training simulator.

[0023] According to another aspect of at least one embodiment of the present invention, a real-time endoscope tracker system is disclosed, comprising: a support having a housing having spaced-apart distal ends and a hole located in the center of the housing for receiving an endoscope having an insertion tube; at least two cameras, each camera mounted at one of the distal ends so that each camera points toward the center of the housing and each camera has an uninterrupted view of the endoscope when received in the hole in the center of the housing; and a processor communicating with a non-transitory computer-readable medium storing program instructions that, when executed by the processor, cause the processor to perform operations based on the instructions. The processor is configured to obtain an image of a portion of the endoscope that is in the field of view (FOV) of the at least two cameras to track the position and orientation of the distal end of the endoscope in real time, and the processor is configured to: obtain a real-time image of the portion of the endoscope in the FOV of the at least two cameras from the at least two cameras; measure lateral and rotational movements of the endoscope based on markers printed on the endoscope and captured in the real-time image; and correlate the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movements, thereby determining the position and orientation of the distal end of the endoscope.

[0024] In at least one embodiment, the at least two cameras cover a full 360 degree view of the insertion tube of the endoscope around the lateral axis of the insertion tube for measuring lateral and rotational movement of the endoscope.

[0025] In at least one embodiment, the at least two cameras include three cameras, wherein each of the three cameras is fixed at a point such that two consecutive cameras are 120° from the axis of the insertion tube to provide an uninterrupted view of the insertion tube and its markings.

[0026] In at least one embodiment, the at least two cameras are configured to capture video of the endoscope as the endoscope moves relative to a fixed FOV of the at least two cameras.

[0027] In at least one embodiment, the processor, when executing the program instructions, is further configured to identify white line markings along the length of the insertion tube to correct for accumulated errors built up in the measurements.

[0028] In at least one embodiment, the processor, when executing the program instructions, is further configured to identify a black gap at a point on the circumference of the insertion tube to correct for errors in rotation.

[0029] In at least one embodiment, the processor, when executing program instructions, is configured to combine data regarding lateral and rotational movement with real-time video from a camera on the endoscope tip to determine whether the insertion tube is forming a loop.

[0030] In at least one embodiment, the processor, when executing the program instructions, is also configured to combine data regarding lateral and rotational movement with real-time video from a camera on the tip of the endoscope to generate an image mosaic that maps the inner surface of the colon being imaged to locate tumors and polyps.

[0031] In at least one embodiment, the at least two cameras include two cameras, and the processor, when executing the program instructions, is further configured to compensate for a delay in recording a frame from one of the two cameras.

[0032] In at least one embodiment, the processor, when executing the program instructions, is further configured to use computer vision to detect movement of the endoscope based on images obtained from the at least two cameras.

[0033] In at least one embodiment, the processor, when executing the program instructions, is further configured to provide at least one of a position or an orientation of a distal end of the endoscope to the endoscope training simulator.

[0034] According to another aspect of at least one embodiment of the present invention, a method for tracking an endoscope using an endoscope tracker is disclosed, wherein the method includes: fixing a sensor sleeve of the endoscope tracker at a position outside a patient or a training phantom; manually inserting the endoscope through an entry hole and an exit hole in the sensor sleeve; positioning the insertion tube of the endoscope between tracking balls and a camera of the endoscope tracker; measuring the position of the endoscope using y-axis data from each tracking ball and measuring the rotation of the endoscope using x-axis data from each tracking ball; correcting uncertainty in the position and rotation of the endoscope using image data from the camera; and providing output of the movement of the endoscope outside the patient or training phantom.

[0035] According to another aspect of at least one embodiment of the present invention, a method of tracking an endoscope is disclosed, comprising: connecting a tracking ball to one of a plurality of cameras; placing or positioning the camera to view the endoscope from an opposite end of a sensor sleeve; measuring the rotation of the tracking ball; capturing an image with the camera, the image showing markings along an insertion tube of the endoscope; using the image to correct errors in the position and rotation of the endoscope; and providing an output of the movement of the endoscope outside of a patient or training phantom.

[0036] According to another aspect of at least one embodiment of the present invention, a method for tracking an endoscope is disclosed, comprising: polling a tracking ball at regular intervals to update endoscope motion; acquiring images via a camera and detecting image features of the endoscope in real time; combining tracking ball data and image data to update a real-time estimate of endoscope motion; correcting positional errors and rotational errors using the image features; and providing an output of the motion of the endoscope outside a patient or training phantom.

[0037] According to another aspect of at least one embodiment of the present invention, a method for tracking an endoscope is disclosed, comprising: capturing video of the movement of the endoscope with a camera as the endoscope moves relative to a fixed field of view of the camera; tracking the apparent motion of the endoscope using markers that appear on an insertion tube of the endoscope; and providing output of the movement of the endoscope outside a patient or training phantom.

[0038] In at least one embodiment, the method is performed by an endoscopic tracker having two tracking balls and two cameras, and the method further comprises correcting cumulative errors accumulated in the measurement of the apparent motion of the tracking balls by using image data obtained from the two cameras.

[0039] In at least one embodiment, the method is performed by an endoscope tracker having at least two cameras for obtaining an image of a portion of the endoscope in the field of view (FOV) of the at least two cameras, and the method also includes: obtaining a real-time image of the portion of the endoscope in the FOV of the at least two cameras from the at least two cameras; measuring lateral and rotational movements of the endoscope based on markers printed on the endoscope and captured in the real-time image; and correlating the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movements to determine the position and orientation of the distal end of the endoscope.

[0040] In at least one embodiment, capturing video includes: determining distortion parameters for each camera; based on the distortion parameters, dedistorting an image from the video captured by the camera; converting pixel distances in the dedistorted image to distance units for position tracking; and determining a corresponding angle for rotational tracking using a known diameter of the endoscope and a known distance from the endoscope to the camera.

[0041] In at least one embodiment, tracking apparent motion further comprises: recording a starting position and updating the starting position in real time using position tracking information from a camera; and recording a starting rotation angle and updating the starting rotation angle in real time using rotation tracking information from a camera.

[0042] Other features and advantages of the present application will become apparent through the following detailed description in conjunction with the accompanying drawings. However, it should be understood that while indicating the preferred embodiments of the present application, the detailed description and specific examples are given only by way of illustration, because various changes and modifications within the spirit and scope of the present application will become apparent to those skilled in the art through the detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to better understand the various embodiments described herein, and to more clearly show how these various embodiments may be implemented, reference will be made by way of example to the accompanying drawings, which illustrate at least one exemplary embodiment, which will now be described. The drawings are not intended to limit the scope of the teachings described herein.

[0044] Figure 1 A schematic diagram of an exemplary embodiment of a system for real-time, intraoperative, endoscopic axis motion tracking is shown.

[0045] Figure 2 An exemplary embodiment of an endoscope tracker is shown with an endoscope inserted therein.

[0046] Figure 3 An exemplary embodiment of a sensor sleeve is shown, illustrating its two-piece housing and its sensor.

[0047] Figure 4 An example of still images taken by a leader camera and a follower camera at almost the same time is shown.

[0048] Figure 5 Plot of the endoscope tracker position tracking versus the manually recorded ground truth for a single position calibration trial is shown.

[0049] Figure 6 Shows Figure 5 Plot of the error versus the ground truth.

[0050] Figure 7 A plot of the position displacement ground truth versus the position error of the ground truth is shown.

[0051] Figure 8 A plot of the rotational displacement ground truth versus the rotational error of the ground truth is shown.

[0052] Fig. 9 A graph showing the relationship between the number of trackball updates and the rotation speed is shown.

[0053] Fig.10 A graph of the position error and average velocity results is shown.

[0054] Fig.11A plot of position error versus average velocity for each translation in Groups 1-3 is shown.

[0055] Fig.12 Plots of the rotation ground truth versus the rotation error and average rotation speed are shown.

[0056] Figures 13A-13D An exemplary embodiment of an endoscope holder for use with an optical endoscope tracker is shown.

[0057] Fig.14 A flow chart of an exemplary embodiment of a method of tracking an endoscope from the perspective of a sensor sheath is shown.

[0058] Fig.15 A flow chart showing an exemplary embodiment of a method of tracking an endoscope from the perspective of a trackball and a camera.

[0059] Fig.16 A flow chart of an exemplary embodiment of a method of tracking an endoscope from a data acquisition perspective is shown.

[0060] Fig.17 A flow chart illustrating an exemplary embodiment of a method of tracking an endoscope using a contactless tracker.

[0061] Further aspects and features of the exemplary embodiments described herein will become apparent from the following description taken in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0062] Various embodiments according to the teachings of this article will be described below to provide examples of at least one embodiment of the subject matter claimed for protection. The embodiments described herein do not limit any of the subject matter claimed for protection. The subject matter claimed for protection is not limited to a device, system or method having all the features of any device, system or method described below, nor is it limited to the features common to multiple or all devices, systems or methods described herein. The device, system or method described herein may not be an embodiment of any of the subject matter claimed for protection. Any subject matter described herein that is not claimed for protection in this document may be the subject of another protective instrument, for example, a continuing patent application, and the applicant, inventor or owner does not intend to abandon, deny or dedicate any such subject matter to the public by disclosing any such subject matter in this document.

[0063] It should be understood that for simplicity and clarity of explanation, reference numerals may be repeated in the accompanying drawings to indicate corresponding or similar elements where deemed appropriate. In addition, many specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be appreciated by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other cases, well-known methods, processes, and components are not described in detail to avoid obscuring the embodiments described herein. Moreover, this description should not be construed as limiting the scope of the embodiments described herein.

[0064] It should also be noted that the terms "coupled to" or "coupled" as used herein can have a variety of different meanings, depending on the context in which the terms are used. For example, the terms "coupled to" or "coupled" can have mechanical or electrical meanings. For example, as used herein, the terms "coupled to" or "coupled" can mean that two elements or devices can be directly connected to each other, or connected to each other via electrical signals, electrical connections, or mechanical connections through one or more intermediate elements or devices, depending on the specific context.

[0065] It should also be noted that, as used herein, the term "and / or" is intended to mean an inclusive or. That is, "X and / or Y" is intended to mean, for example, X or Y or both. As a further example, "X, Y and / or Z" is intended to mean X or Y or Z or any operable combination thereof (which means that these expressions encompass any combination of elements that result in a working embodiment).

[0066] It should be noted that the degree terms such as "substantially", "about" and "approximately" used herein refer to a reasonable amount of deviation of the modified term so that the end result will not be significantly changed. These degree terms can also be interpreted as including deviations of the modified term, such as 1%, 2%, 5% or 10%, for example, if the deviation does not negate the meaning of the term it modifies.

[0067] In addition, numerical ranges recited herein by endpoints include all numbers and fractions contained within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.90, 4, and 5). It should also be understood that all numbers and fractions thereof are assumed to be modified by the term "about," meaning that the referenced number may vary by up to a certain amount, e.g., 1%, 2%, 5%, or 10%, if the end result is not significantly changed.

[0068] It should also be noted that use of the term "window" in connection with describing the operation of any system or method described herein is meant to be understood as describing a user interface for performing initialization, configuration, or other user operations.

[0069] Exemplary embodiments of the devices, systems, or methods described according to the teachings herein may be implemented as a combination of hardware and software. For example, the embodiments described herein may be implemented at least in part by using one or more computer programs executed on one or more programmable devices including at least one processing element and at least one storage element (i.e., at least one volatile storage element and at least one non-volatile storage element). Depending on the implementation of hardware, the hardware may include an input device including at least one of a touch screen, a keyboard, a mouse, a button, a key, a slider, etc., and one or more of a display, a printer, etc.

[0070] It should also be noted that there may be some elements for realizing at least a part of the embodiments described herein, which may be realized by software written in a high-level procedural language such as object-oriented programming. The program code may be written in C++, C#, JavaScript, Python or any other suitable programming language, and may include modules or classes, as known to those skilled in the art in the field of object-oriented programming. Alternatively or additionally, some of these elements implemented by software may be written in assembly language, machine language or firmware as required. In either case, the language may be a compiled language or an interpreted language.

[0071] At least some of these software programs may be stored on a computer-readable medium, such as but not limited to a ROM, a disk, an optical disk, a USB key, etc., which may be read by a device having a processor, an operating system, and related hardware and software necessary to implement the functions of at least one embodiment described herein. When read by the device, the software program code configures the device to operate in a new, specific, and predefined manner (e.g., as a special-purpose computer) in order to perform at least one of the methods described herein.

[0072] At least some of the programs associated with the apparatus, systems, and methods of the embodiments described herein may be capable of being distributed as a computer program product comprising a computer-readable medium carrying computer-usable instructions, such as program code, for one or more processing units. The medium may be provided in various forms, including non-transitory forms, such as, but not limited to, one or more floppy disks, optical disks, tapes, chips, and magnetic and electronic memory. In alternative embodiments, the medium may be transient in nature, such as, but not limited to, wired transmissions, satellite transmissions, Internet transmissions (e.g., downloads), media, digital and analog signals, and the like. The computer-usable instructions may also be in various formats, including compiled and non-compiled code.

[0073] According to the teachings herein, various embodiments for real-time, intraoperative, endoscopic axis motion tracking and computer products for use therewith are provided. At least one of these embodiments includes endoscope insertion length and orientation tracking. These embodiments may also be applied to EGD, enteroscopy, and colonoscopy. More generally, these embodiments may be applied to any potential gastrointestinal (GI) endoscopic procedure, as well as, for example, examination of the respiratory tract by bronchoscopy, with appropriate modifications known to those of ordinary skill in the art, or without modification (if not required).

[0074] One of the advantages of at least one embodiment of the invention described herein is that a real-time endoscopic axis motion tracker is provided that works with most unmodified endoscopes. This lowers the barrier to adaptation of endoscopic motion trackers and allows for improved positioning with less equipment and training costs.

[0075] Another advantage of at least one embodiment of the invention described herein is that it allows for the measurement of lateral and rotational motion using original built-in markings (patterns) printed on the insertion tube.

[0076] Yet another advantage of at least one embodiment of the invention described herein is that it can be used with all existing endoscopes currently used by healthcare providers, which reduces initial setup costs. This is in stark contrast to 3D positioning solutions, which require the purchase of new, specialized endoscopes with embedded coils or sensors and bulky detection units; these systems do not measure rotation, but rather display a 3D position model.

[0077] First reference Figure 1 , which shows a block diagram of an exemplary embodiment of a system 100 for real-time endoscope insertion length and orientation tracking. The system 100 includes at least one server 120. For example, at least one server 120 can be one or more server computers in a client / server architecture. Alternatively or additionally, at least one server 120 can be two or more computers that operate independently or interdependently from each other, such as a leader-follower or peer. Alternatively or additionally, at least one server 120 can be a single computer configured to send instructions to and receive data from the electronic signal interface of two or more cameras and two tracking balls. For ease of reference, at least one server 120 will be referred to herein as "server 120", even though it may be more than just one computer. The server 120 can communicate with one or more user devices (not shown), for example wirelessly or via the Internet. When used in this way, the system 100 may also be referred to as a machine learning system.

[0078] A user device may be a computing device operated by a user. A user device may be, for example, a smartphone, a smartwatch, a tablet, a laptop, a virtual reality (VR) device, or an augmented reality (AR) device. A user device may also be, for example, a combination of computing devices operating together, such as a smartphone and a sensor. A user device may also be, for example, a device operated by a user in other ways, such as a drone, a robot, or a remote control device; in this case, the user device may be operated, for example, by a user through a personal computing device such as a smartphone. A user device may be configured to run an application (e.g., a mobile application) that communicates with other parts of the system 100 (e.g., server 120).

[0079] The server 120 may be run on a single computer, including a processor unit 124, a display 126, a user interface 128, an interface unit 130, input / output (I / O) hardware 132, a network unit 134, a power supply unit 136, and a memory unit (also referred to as "data storage") 138. In other embodiments, the server 120 may have more or fewer components, but generally operate in a similar manner. For example, the server 120 may be implemented using more than one computing device.

[0080] The processor unit 124 may include a standard processor, such as an Intel Xeon processor. Alternatively, the processor unit 124 may use multiple processors, and these processors may work in parallel and perform certain functions. The display 126 may be, but is not limited to, a computer monitor or an LCD display, such as an LCD display for a flat panel device. The user interface 128 may be an application programming interface (API) or a network-based application accessible via the network unit 134.

[0081] Processor unit 124 may execute prediction engine 152 for providing predictions using one or more machine learning models 146 stored in memory unit 138. Prediction engine 152 may construct a prediction algorithm through machine learning. The training data may include, for example, image data, video data, audio data, and / or text data.

[0082] The processor unit 124 may also execute a graphical user interface (GUI) engine 154 for generating various GUIs. The GUI engine 154 provides data according to the specific layout of each user interface, and also receives data input or control input from the user. The GUI then uses the input from the user to change the data shown on the current user interface, or to change the operation of the server 120, which may include displaying a different user interface. For example, the GUI may allow the user to connect, reconnect, or disconnect to other parts of the system 100, as well as debug. The GUI may also allow the user to view images or videos from an external source (e.g., a camera) and overlay boxes (or other shapes) around portions of the image or video. The GUI may also provide controls for the user to test some or all of the system 100. The GUI may also provide a visual representation of an object (e.g., an endoscope) connected to the system 100.

[0083] The interface unit 130 may be any interface that allows the processor unit 124 to communicate with other devices within the system 100. In some embodiments, the interface unit 130 may include at least one of a serial bus or a parallel bus, and corresponding ports, such as a parallel port, a serial port, a USB port, and / or a network port. For example, a network port may be used so that the processor unit 144 can communicate via the Internet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a wireless local area network (WLAN), a virtual private network (VPN), or a peer-to-peer network directly or through a modem, router, switch, hub, or other routing or conversion device.

[0084] The I / O hardware 132 may include, but is not limited to, at least one of a microphone, a speaker, a keyboard, a mouse, a touch pad, a display device, a printer, a camera, and a medical device (eg, an endoscope).

[0085] The network unit 134 includes various communication hardware for allowing the processor unit 124 to communicate with other devices. For example, the network unit 134 includes at least one of a network adapter, such as an Ethernet or 802.11x adapter, a Bluetooth radio or other short-range communication device, or a wireless transceiver for wireless communication, such as according to CDMA, GSM, or GPRS protocols using standards such as IEEE 802.11a, 802.11b, 802.11g, or 802.11n.

[0086] The power supply unit 136 may include one or more power supplies (not shown) connected to the various components of the system 100 for providing power thereto, as is well known to those skilled in the art.

[0087] The storage unit 138 may include one or more non-transitory machine-readable storage units or storage devices that are accessible by other modules such as the image signal processing module 126, the image / video encoder module 130, and the communication module 132 for reading and executing machine-executable instructions stored therein (e.g., as firmware), and for reading and / or storing data, such as data of captured images and data generated during operation of the other modules. The memory module 128 may be a volatile and / or non-volatile, non-removable or removable memory, such as RAM, ROM, EEPROM, solid-state memory, hard disk, CD, DVD, flash memory, etc.

[0088] The memory unit 138 may include one or more non-transitory machine-readable memory units or storage devices accessible by other modules for storing program instructions for the operating system 140, program code 142 for other applications, input modules 144, one or more machine learning models 146, output modules 148, and databases 150. The process can access the memory unit 138 to read and execute the machine executable instructions stored therein, as well as read and / or store data during the operation of other modules. The memory unit 138 can be a volatile and / or non-volatile, non-removable or removable memory, such as RAM, ROM, EEPROM, solid-state memory, hard disk, CD, DVD, flash memory, etc. The machine learning model 146 may include, but is not limited to, image recognition and classification algorithms based on deep learning models and other methods. Although the database 150 is shown as being stored on a local memory, it should be understood that the database 150 can be, for example, an external database, a database on the cloud, multiple databases, or a combination thereof.

[0089] In at least one embodiment, the machine learning model 146 may include one or more convolutional neural networks, one or more recurrent neural networks, or a combination of one or more convolutional neural networks and one or more recurrent neural networks. A convolutional neural network (CNN) may be designed to recognize images or patterns. A CNN may perform a convolution operation, for example, a convolution operation may be used to classify image regions and determine the edges of objects identified in image regions. A recurrent neural network (RNN) may be used to recognize sequences, such as text, speech, and time evolution, so RNNs may be applied to data sequences to predict what will happen next. Thus, a CNN may be used to read what is happening on a given image at a given time, while an RNN may be used to recognize one or more sequences, optionally make one or more predictions, and provide information messages. Alternatively or additionally, a machine learning model may use computer vision, for example, alone or in combination with a CNN and / or RNN.

[0090] Program 142 includes program code that, when executed, configures processor unit 124 to operate in a particular manner to implement various functions and tools of system 100 , such as the functions discussed with respect to methods 1400 , 1500 , 1600 , and 1700 .

[0091] For example, input module 144 may include program code for obtaining input data and / or operating parameters from various sources including, for example, a user and / or memory.

[0092] For example, output module 148 may include program code for displaying various data and / or storing calibration data, image data, patient data, and / or operating parameters in a file such as database 150 .

[0093] See now Figure 2 , which shows an exemplary embodiment of an endoscope tracker 250. In the embodiments of the endoscope tracker 250 described herein, some or all of the system 100 may be used to perform any one or more functions requiring computer functionality (e.g., digital input, digital output, software processing, image processing, etc.).

[0094] Figure 2 An example setup is shown with an endoscope tracker 250 having an endoscope 255 inserted to the 85 cm line mark. The control portion of the endoscope appears on the left, with its insertion tube passing through the sensor sleeve on the right. A leader Raspberry Pi ("RasPi", sometimes referred to as "RPi") computer 260 is in the foreground, and a follower RasPi computer 270 is in the background. Two trackballs (see, e.g., Figure 3) is connected to the leader RasPi 260 via USB. Each RasPi computer is connected to the camera via a camera serial interface (CSI) and to the light source via an internal integrated circuit (I2C). The endoscope 255, the leader RasPi 260 and / or the follower RasPi 270 can be considered to be part of the I / O hardware 132 of the system 100. The leader RasPi 260 may alternatively be referred to as a leader computer (or "host computer"), and the follower RasPi 270 may alternatively be referred to as a follower computer (or "server computer"). Each RasPi computer may have some or all of the functionality of the server 120. It should be understood that the embodiments of the present invention described herein should not be limited to a particular brand or model, such as a Raspberry Pi computer, and other single board computers (SBCs), for example, may be used instead. Some SBCs can be connected to two cameras, which can allow the program instructions (or code) of the leader computer to be combined with the program instructions (or code) of the follower computer so that all calculations are performed on one processor. Examples of SBCs that can be connected to two or more cameras include various Jetson TM SBC, Raspberry Pi Compute Module 4 (2 cameras), and Raspberry Pi 5 (2 cameras).

[0095] In order to maximize the use of endoscope trackers, the position and orientation of the distal end of the endoscope must be accurately measured to locate any lesions found during endoscopy. In current practice, the position of the endoscope's distal camera is estimated using graduated lines on the endoscope's insertion tube. Each line represents its distance from the endoscope tip. This allows the insertion length of the endoscope into the patient to be manually recorded at key points during endoscopy, such as when a lesion is found. However, traditionally, the orientation of the endoscope's camera could not be determined because its rotation was not measured.

[0096] According to the teachings of this document, a real-time endoscope motion tracker continuously records the position and rotation of the endoscope and the insertion speed and rotation speed (and optional acceleration) of the endoscope insertion tube at the orifice position. The motion tracker uses a sensor sleeve that surrounds the endoscope insertion tube at a point outside the patient's body. The movement of the endoscope insertion tube at the orifice (i.e., the movement of the endoscope sheath at its entry position) is related to the movement of the distal tip inside the patient's body because the length of the endoscope does not change and has rotational rigidity. The motion tracker mechanically tracks the movement of the endoscope using a tracking ball and uses a camera to detect the scale and correct the accumulated errors caused by any slippage in the movement of the endoscope relative to one or two tracking balls. The motion tracker can determine the position of the endoscope tip based on, for example, the insertion depth of the endoscope. Alternatively or additionally, the position of the tip can be inferred from the movement of the endoscope axis and the analysis (e.g., machine learning) of the simultaneous movement or changes of endoscopic image features.

[0097] Combining motion tracking data from the endoscope tip with video simultaneously captured by an endoscopic camera at the endoscope tip is designed to improve localization of polyps and cancers. The tracker's data can be used to stitch frames together to create a 3D reconstruction, or unfolded map, of the colon mucosa.

[0098] Return to reference Figure 2 , the real-time endoscope tracker 250 includes a sensor sleeve 265 and a real-time motion tracking program. The sensor sleeve 265 can be considered as part of the I / O hardware 132 of the system 100. The real-time motion tracking program can be considered as part of the program 142 of the system 100. The program can be run simultaneously on two Raspberry Pi (RasPi) computers 260, 270 working together in a leader / follower model (leader RasPi 260, follower RasPi 270). For example, when including computer hardware and / or software that work together to track a portion of an endoscope (e.g., the endoscope tip), the endoscope tracker 250 can be more generally referred to as a real-time endoscope tracker system.

[0099] Figure 3Several views of an example embodiment of a sensor sleeve 300 are shown, showing its housing (in this case 2 attachments) and its sensors. The top view shows a top housing piece 380 in which a top tracking ball 310 is mounted, the middle view shows a schematic side view of the sensor, and the top view shows a bottom housing piece 382 in which a bottom tracking ball 320 and two cameras 330, 340 are housed. The coordinate system of the endoscope 305 is labeled next to the endoscope 305 in the side view. The positive x (rotation) and y (insertion) tracking ball axes are marked on the tracking ball with arrows at right angles to each other. The top housing piece 380 has two apertures 380a, 380b for allowing the endoscope 305 to pass therethrough. The bottom housing piece 382 has two holes 390a, 390b for allowing the endoscope 305 to pass therethrough. The sensor sleeve 300 can be part of a real-time endoscope tracker 250. The sensor sleeve 300 can be, for example, used with Figure 2 The sensor kit 300 may be considered to be part of the I / O hardware 132 of the system 100.

[0100] Inside the sensor sleeve 300 are two trackballs 310, 320, two cameras 330, 340 and two light sources 360, 370, which can be as Figure 3 The tracking balls 310, 320 measure the insertion and rotation of the endoscope over time. For example, by using the cameras 330, 340 to detect white tick marks that appear on the insertion tube of the endoscope 305 at regular intervals (e.g., every 5 cm), the accumulated uncertainty (i.e., accumulated error) in the measurements of the tracking balls 310, 320 can be mitigated. The tracking balls 310, 320 and / or the cameras 330, 340 can be considered part of the I / O hardware 132 of the system 100.

[0101] method

[0102] A method of using an endoscopic tracker includes steps associated with one or more of a sensor sleeve, a tracker ball, a camera, data acquisition, and tracker performance evaluation.

[0103] A.Sensor set

[0104] The sensor sleeve 300 can be fixed in position outside the patient or training phantom. The sensor sleeve 300 can be prism-shaped, or other suitable shape, with the endoscope 305 passing horizontally through its center. The endoscope 305 can be considered as part of the I / O hardware 132 of the system 100. The endoscope 305 can be manually inserted through the inlet (collectively referred to as the orifices 380b, 390a) and the outlet (collectively referred to as the orifices 390a, 390b) in the sensor sleeve 300, which places the insertion tube of the endoscope vertically between the top tracking ball 310 and the bottom tracking ball 320, and horizontally between the cameras 330, 340 at each end of the sensor sleeve 300. The endoscope 305 can be clamped vertically between the two tracking balls 310, 320 and can be viewed by the cameras 330, 340 located at the left and right ends of the sensor sleeve 300. In at least one embodiment, bottom tracking ball 320 supports endoscope 305 and prevents it from moving downward within sensor cuff 300 due to gravity, which can make images easier to process. In alternative embodiments, the endoscope tracker can be modified for different orientations of sensor cuff 300, in which case 320 can be located in different positions, such as along different planes (e.g., horizontal) on either side of the endoscope, although this may not operate accurately / robustly unless the weight of endoscope 305 is supported by the housing of sensor cuff 300 in a different manner.

[0105] More generally, the directions "horizontal" and "vertical" may be used to refer to relative positions. In practice, the cameras 330, 340 may be arranged 180° from each other to cover a complete view of the tube portion of the endoscope 305. The two trackballs 310, 320 may be arranged relative to each other to provide a pressure / fit to the tube (e.g., maintain alignment, reduce slippage). The relative positions of the trackballs 310, 320 and the cameras 330, 340 need not be completely orthogonal to each other, as long as the trackballs 310, 320 do not obstruct the cameras 330, 340. The lateral position of the trackballs 310, 320 and / or the cameras 330, 340 may be moved along the length of the endoscope 305 to achieve a complete view of the tube portion of the endoscope.

[0106] The housing of the sensor sleeve 300 can be 3D printed in two pieces 380, 390. The top piece 380 holds one trackball 310 in place and has slots at each end for cables. The bottom piece 390 aligns the cameras 330, 340 and another trackball 320 with the endoscope 305. Four screws or other suitable fasteners can connect the two halves, one at each corner of the housing. The screws can be adjusted so that the trackball has the desired tracking force on the endoscope to ensure accurate tracking. Figure 3 The top and bottom pieces 380, 390 of the tracker's sensor sleeve 300 are shown, along with a schematic diagram showing how the sensors are positioned.

[0107] The endoscope 305 is in direct contact with two trackballs 310, 320, so they rotate in place as the endoscope 304 moves. The position of the endoscope 305 (e.g., insertion length) is measured using the y-axis of the trackballs 310, 320, while the rotation is measured using the x-axis of the trackballs 310, 320. The measurement uncertainty of each trackball is proportional to its rotation on each axis. The uncertainty (or accumulated measurement error) generated in the position and rotation of a portion of the endoscope 305 can be corrected by recording images using cameras 330, 340, which can be used to detect scale markings that appear at regular intervals (e.g., 5 cm intervals) along the insertion tube of the endoscope 305. The two cameras 330, 340 of the sensor sleeve 300 observe the markings from opposite sides of the endoscope 305 to allow its rotation to be corrected on the fly (e.g., at different intervals, continuously, at predetermined degrees, such as every 180°). In an alternative embodiment with three cameras, the measurement uncertainty (or accumulated measurement error) is corrected for example every 120°.

[0108] The diameter of the inlet and outlet and the distance between the two trackballs 310, 320 can be customized for a particular type of endoscope based on its shaft diameter. This allows for good contact without a significant amount of slippage as the endoscope 305 moves within the sensor sleeve 300.

[0109] According to the teachings of this paper, all endoscope tracker calibrations and results presented herein are for a prototype device used with a 1 m long gastroscope (GIF-XQ10, Olympus Canada), but the prototype device can be adjusted to work with different endoscope models with graduated markings indicating insertion length. The prototype device was tested on three common commercial endoscope insertion tube axes with similar results to those reported herein.

[0110] In an example setup, two RasPi (model 3BC, RasPi Trading; Cambridge, England, UK) read the motion of the tracking ball, capture and process images, and calculate information about the endoscope's motion in real time. The output of the tracker can be displayed in a separate monitor (current setup) and / or integrated with the endoscope controller. Each RasPi is connected to a RasPi camera (version 1.3, RasPi Trading) and a white light-emitting diode (LED) ring light (Pi light, Mindsensors.com; Henrico, VA, USA) for illumination. One of the RasPis is designated as the "leader" and the other is called the "follower." The leader RasPi 260 is connected to one camera and two USB tracking balls. The follower RasPi 270 controls only the second camera. The leader RasPi 260 combines all sensor data and then calculates the position, rotation, and motion of the endoscope.

[0111] Fig.14 A flow chart of an exemplary embodiment of a method 1400 of tracking an endoscope from the perspective of a sensor sheath is shown. The method 1400 may be performed using some or all portions of the system 100 and / or some or all portions of the endoscope tracker 250.

[0112] At 1410, the sensor sleeve is secured in position external to the patient or training phantom.

[0113] At 1420, an endoscope is manually inserted through the inlet and outlet holes in the sensor sheath.

[0114] At 1430, an insertion tube of an endoscope is positioned between tracking balls (eg, two tracking balls) and between cameras (eg, two cameras).

[0115] At 1440, the position of the endoscope is measured using the trackball y-axis, and the rotation of the exoscope is measured using the trackball x-axis.

[0116] At 1450, the image data obtained from the camera is used to correct the uncertainty of the position and rotation of the endoscope. The correction of uncertainty can be achieved by one or more of the corrections for position error or rotation error described herein.

[0117] At 1460, an output is provided (eg, displayed) of the movement of the endoscope external to the patient or training phantom.

[0118] Figure 4 The interior of the sensor sleeve 410 is shown, along with the mounting location 420 for the trackball and camera.

[0119] B. Trackball Settings

[0120] Two trackballs (eg, the X13 trackball manufactured by Cursor Controls, Inc.) directly measure the motion of the endoscope. Figure 3 The side view in shows that the insertion of the positive endoscope 305 is guided into the page and the positive rotation is clockwise. The positive y-axis of the trackballs 310, 320 is aligned with the positive insertion axis of the endoscope. Similarly, the positive x-axis of the trackballs '310, 320 is aligned with the positive rotation direction of the endoscope. Two laser trackballs (e.g., X13, Cursor Controls; Newark, Nottinghamshire, UK) are connected to a leader RasPi 260 and convert the movement of the endoscope into trackball rotations. The rotation of the trackballs can also be measured optically without direct contact between the trackballs and sensors 350 and 360, which are USB.

[0121] In this exemplary prototype, each trackball has a diameter of 12.7 mm and is configured to output 300 ± 10% counts per revolution in linear mode

[10] . Therefore, the linear resolution of a single trackball (l spec )for:

[0122]

[0123] This equates to approximately 380 counts measured for every 5 cm of endoscope insertion.

[0124] The angular resolution that can be achieved by tracking the endoscope using a single tracking ball depends on the diameter of the endoscope insertion tube (d it ). The test was performed using an Olympus GIF-XQ10 gastroscope with a diameter of 9.8 mm. Angular resolution (a spec ) is given by:

[0125]

[0126] Each trackball is configured to generate approximately 231 counts per 360°. Depending on the diameter of the endoscope shaft, other counts may be generated per revolution (e.g., 100 counts per 360°, 360 counts per 360°, 480 counts per 360°, etc.).

[0127] The trackballs 310, 320 generate more counts than expected during tracking.The counts from the two trackballs 310, 320 are combined to produce a more robust estimate of the endoscope motion, as described in the Calibration and Calculation section.

[0128] C. Camera Settings

[0129] Two Raspberry Pi cameras 330, 340 observe the endoscope 305 from opposite ends of the sensor sleeve 265. The cameras 330, 340 are set to acquire grayscale images as detailed in Table 1. They have a rolling shutter with a shutter speed of 11111 μs at 90 Hz. The two cameras 330, 340 are placed at a working distance of 3.5 cm and their focus is adjusted. Camera illumination is provided by two Mindsensors Pi-Lights, each containing four RGB LEDs

[11] .

[0130] The leader camera on the leader RasPi 260 and the follower camera on the follower RasPi 270 are used to detect markers that appear every 5 cm along the insertion tube of the endoscope. The leader camera may alternatively be referred to as the "master camera" and the follower camera may alternatively be referred to as the "server camera". Images of these markers are used to correct accumulated position and rotation errors in the trackball measurements. The cameras have a fixed field of view (FOV) that includes the region of interest (ROI) where the insertion tube appears. A 2.7 cm long insertion tube is visible in each image. The leader camera and follower cameras observe opposite sides of the insertion tube of the endoscope, such as Figure 4 The software for the cameras includes different program code paths for running on the leader RasPi 260 and the follower RasPi 270, wherein the program is selected, for example, using a flag on the command line when the software is started, and wherein the software also includes different calibration / cropping constants for the two cameras, which can be determined empirically during calibration.

[0131] More specifically, Figure 4 An example of a still image taken by a leader camera and a follower camera at nearly the same time is shown. The follower (or "server") image has been rotated 180° to reverse its orientation. The image shows a 40 cm line 410 with no gaps and a gap in the 40 cm line 420. These image features can be used to correct for insertion and rotational position errors of the endoscope 305. The figure also shows two "40" text marks appearing on both sides of the endoscope.

[0132] The open source AVA RaspiCam code

[18] was modified to configure the RasPi camera and capture images at 90 frames per second. OpenCV (CCC) was used for fast image processing. The settings used for each camera 260, 270 are summarized in Table 1. In addition, the stock focal length of each camera lens was shortened to match the 3.5 cm working distance, and a ring light was mounted on each camera to provide illumination.

[0133] Table 1: Raspberry Pi Camera Settings

[0134]

[0135] Each white line on the gastroscope 255 has a black line gap. The line gaps are all aligned along one side of the insertion tube. The real-time motion tracking program uses the white scale lines to correct for cumulative errors in the insertion length, and the black line gaps to correct for the rotation angle. Using the leader camera and follower camera, the length position (also called the insertion length) can be corrected every 5cm, and the rotation angle can be corrected every 180 degrees. Figure 4 An example image 400 showing white tick marks 410 and black line gaps 420 .

[0136] The real-time motion tracking program detects two types of image features: white tick marks and black line gaps. The origin is in the upper left corner of the image. The leader camera uses a rotation setting of 180° because this aligns the positive image x-axis with the positive endoscope insertion direction. The follower camera does not require rotation. With this setting, the tick mark image features move to the right as the endoscope is inserted through the sensor sleeve, and the line gap image features move downward during clockwise (CW) rotation of the endoscope. Next, both images are cropped to show the endoscope ROI before the image features are detected. For example, the ROI for white tick mark detection is the insertion tube of the endoscope. Figure 4 A cropped image 400 captured by a leader camera and a follower camera and detected image features are shown.

[0137] A GUI, such as may be provided by the GUI engine 154, may provide the user with the ability to connect / reconnect to and / or disconnect from the system 100. The GUI may allow the user to view video from both cameras 330, 340. The GUI may draw a box around the white tick mark and black line gap image features. The GUI may also provide the user with controls for testing some or all of the cameras 330, 340, the LED lights, or the system 100. The GUI may display the insertion length (and possibly rotation and / or speed) of the endoscope 305 in real time (e.g., at 90 Hz).

[0138] Figure 4 A pair of images 400 recorded simultaneously by two cameras is shown. The perpendicular image of the leader camera is used to define the coordinate system of the two cameras. The image of the follower camera is rotated 180° prior to feature detection. Rotating the image means that the observed direction of motion appears to be the same in both cameras because the cameras are placed on opposite sides of the endoscope. When the endoscope is inserted, the image feature moves to the right. This increases the x-pixel coordinate of the feature. A positive clockwise rotation of the endoscope causes the image feature to move downward, increasing its y-pixel coordinate.

[0139] The camera images two types of features that appear on the endoscope: white scale lines 410 and black line gaps 420. Each white scale line 410 has a small black gap 420. These gaps are aligned with each other and appear on one side of the endoscope insertion tube. The camera uses the white scale lines to correct for insertion length and the black line gaps to correct for rotation. This makes it possible, for example, to correct for insertion length (e.g., at predetermined distances, such as every 1 cm, every 5 cm, continuously). Figure 4 White tick mark features 410 and black line gap features 420 can be seen.

[0140] In at least one embodiment, a motion tracking program is one of the programs 142 and is used to search for all features from all cameras (e.g., 2, 3, or more cameras). The motion tracking program then calculates the position (interpolation and rotation) based on the position of the feature and the position of the camera. In at least some cases, a particular feature (e.g., the number "10") may appear in the images (or FOVs) of multiple cameras. The motion tracking program may only need one image from one camera to calculate the position; however, using all images may be preferred. The motion tracking program may employ an algorithm (or subroutine) that averages the position calculations to obtain better results.

[0141] In an embodiment with two camera setups (e.g., Figures 13A-13D As shown in Figure 1, with or without a trackball), some specific "features" (e.g., gaps) may only appear in the FOV of one camera. On the other hand, the white tick marks are usually in images acquired from both cameras. A motion tracking program can use images from both cameras to detect the white tick marks while searching for gaps on both (since it is impossible to predict in which image the feature will appear). In a three-camera (or multi-camera) setup, the cameras' FOVs may overlap.

[0142] In the case where the endoscope is rotated so that both cameras that detect only white tick mark features or black gap features have images with the same time stamp, the motion tracking program can correct (e.g., in the case of using cameras and trackballs) or determine (e.g., in the case of using cameras but not trackballs) the position and rotation measurements. The motion tracking program can calculate the position using the relative position of each camera and its FOV with the help of prior calibration.

[0143] In the case where images from two cameras that detect neither white tick mark features nor black gap features are at the same timestamp, the program can correct (e.g., in the case of using a camera and a trackball) or determine (e.g., in the case of using a camera but not a trackball) position and rotation measurements. As described above, in many images, a particular feature (e.g., the number "10") may appear in the images (or FOVs) of multiple cameras. The motion tracking program can use data from only one image from one camera to calculate the position; however, using all images may be preferred. The program can employ an algorithm (or subroutine) that averages the position calculations to obtain better results.

[0144] Fig.15 A flow chart of an exemplary embodiment of a method 1500 of tracking an endoscope from the perspective of a trackball and a camera is shown. The method 1500 may be performed using some or all portions of the system 100 and / or some or all portions of the endoscope tracker 250.

[0145] At 1510, a trackball is connected to one of a plurality of cameras.

[0146] At 1520, a camera is placed or positioned to view the endoscope from the opposite end of the sensor sheath.

[0147] At 1530, the rotation of the trackball is measured.

[0148] At 1540, an image is captured by the camera showing markings along the insertion tube of the endoscope.

[0149] At 1550, the images are used to correct errors in the position and rotation of the endoscope. For example, the errors may be corrected every 5 cm lateral and 180° rotation by resetting the trackball measurements to the position (rotation / lateral) determined from the imaging (camera) data. Alternatively, these corrections may be made at different intervals, i.e., not limited to 5 cm / 180°. The correction of the errors may be accomplished by one or more of the corrections for positional or rotational errors as described herein.

[0150] At 1560, an output of the movement of the endoscope external to the patient or training phantom is provided (eg, displayed).

[0151] D. Data Collection Settings

[0152] The real-time endoscope motion tracker 250 can acquire data using a computer program running on two RasPis 260, 270. The leader RasPi 260 is connected to the two trackballs 310, 320 and polls every 8ms to update the measurement data of the movement of the trackballs 310 and 320, which in turn is used to monitor the movement of the endoscope. In addition, both the leader RasPi 260 and the follower RasPi 270 use their cameras to acquire images and detect image features in real time.

[0153] The follower RasPi 270 runs a remote procedure call (RPC) server that communicates the position determined by the image data it detects to the leader RasPi 260. The RPC server is initiated by the endoscope tracking program of the follower RasPi 270; the RPC server is part of the endoscope tracker program (i.e., the motion tracking program) running on the follower RasPi 260. In particular, pixel coordinates (i.e., the x-pixel coordinates of the center of the white lines and the y-pixel coordinates of the gaps between the black lines) are sent. Alternatively, in at least one case, the calculation of the position can be based on the geometric relationship of the endoscope insertion tube surface and the camera, which is confirmed by empirical calibration of endoscopes with the same tube diameter. The leader RasPi 260 periodically polls the follower RasPi 270 for updates. To improve connection reliability, a pair of RasPis 260, 270 is connected using wired Ethernet.

[0154] At least in some cases, the RPC server is started as a child program in the follower program. The RPC server is provided by the gRPC library, and the code in the follower program defines how it starts, replies to remote procedure call queries (using image feature coordinate data from the most recent frame), and shuts down. The leader program acts as a client and sends remote procedure call queries to the RPC server running on the follower program. The follower program provides the RPC server with a queue containing the latest information about the location of white lines and black line gaps that can be found in the most recent image frame if the follower's camera has seen them. The RPC server child program running on the follower program replies to the query from the leader program using the formatted message provided in the follower program queue. Information comes from camera -> follower program -> image features are found and stored as messages -> message queue (valid data) -> RPC server child program -> reply message to query from leader -> leader program receives and decodes message -> leader program uses information for tracking logic

[0155] The leader RasPi 260 combines the lateral and rotational positions determined by the two cameras to update its real-time estimate of the endoscope's motion. Alternatively or additionally, position error can be corrected each time the center of the white line (image feature) crosses the center of the x-axis in the image acquired by the leader camera. As described above, for images obtained from either camera, rotation error is corrected each time the center of the black line gap (image feature) crosses the center of the cropped endoscope ROI on the y-axis. Thus, position and rotation can be corrected at predetermined distances, such as every 5 cm (or other intervals, such as 1 cm or 10 cm, or continuously) and at predetermined degrees, such as 180° (or other spacings, such as 90° or π radians, or continuously), respectively.

[0156] The Endoscope Tracker 250 can estimate the motion of an endoscope in real time using two Raspberry Pi 3B+ (RasPi) computers. The program can be written in C++ and is multi-threaded to best utilize the limited resources of the RasPi platform. In an example setup, each RasPi is connected to a Pi camera and its associated Pi Light. It is not possible to connect both cameras to one RasPi, so a client-server model is used. The first RasPi is called the leader (or "host" or "client") RasPi, while the second is called the follower (or "server") RasPi. The cameras are similarly designated as leader (or "host") and follower (or "server").

[0157] The leader RasPi combines information from all the tracker sensors. It reads updates from both trackballs and uses a camera to detect endoscope image features (e.g., image of the endoscope axis). At the same time, it also acts as a leader for the follower RasPi and receives endoscope features from the follower cameras. Finally, the leader RasPi measures the motion of the endoscope outside the patient. This allows real-time endoscope tip motion tracking, as the endoscope motion inside and outside the patient is correlated.

[0158] The follower RasPi is dedicated to detecting endoscope image features using its camera, as described below, and providing these features to the leader RasPi. The follower RasPi communicates with the leader RasPi over wired Ethernet by performing remote procedure calls using gRPC. During normal operation, the leader RasPi periodically sends update queries to the follower RasPi. The follower RasPi replies by sending the image features of the latest image it has processed.

[0159] In at least one embodiment, each set of endoscopic features can be time stamped. Statistics can be recorded to measure the time lag in acquiring features from follower cameras. This can be measured by subtracting the latest timestamp of the leader camera from the latest follower camera feature timestamp. During use, the time lag (delay) may be small. However, in at least one embodiment, the leader camera can be used for some decisions because it has less lag. Alternatively, the leader camera and follower cameras (or more cameras if there are three or more cameras) can be connected to a single board computer (SBC) to eliminate network lag.

[0160] In at least one embodiment, the endoscope tracker tracks motion at the sensor sheath external to the patient. This external motion is correlated to the motion of the distal end of the endoscope, since the endoscope is rotationally rigid.

[0161] Fig.16 A flow chart of an example embodiment of a method 1600 for tracking an endoscope from a data acquisition perspective is shown. The method 1600 may be performed using some or all portions of the system 100 and / or some or all portions of the endoscope tracker 250.

[0162] At 1610, the trackball is polled at regular intervals to update the movement of the endoscope.

[0163] At 1620, an image is acquired by a camera, and image features of the endoscope are detected in real time.

[0164] At 1630, the trackball data and image data are combined to update a real-time estimate of the endoscope's motion, with the position measured by the trackball being periodically corrected by the position determined by the imaging data. The trackball data provides measurement information, while the image data is used for correction. For example, in the case where the endoscope is in direct contact with two trackballs, the trackballs rotate in place as the endoscope moves. The position (insertion length) of the endoscope can be measured using the y-axis of the trackballs, while the rotation can be measured using the x-axis of the trackballs. The measurement uncertainty of each trackball is proportional to its rotation on each axis. The resulting uncertainty in the endoscope's position and rotation can be corrected by using a camera. For example, a camera can detect scale markings that appear at regular (e.g., 5 cm) intervals along the insertion tube. In addition, for example, the camera can obtain images of markings from the opposite side of the endoscope to allow correction of its rotation at regular (e.g., every 180°) intervals (or partial rotations).

[0165] At 1640, the image features are used to correct errors in the position and rotation of the endoscope. Examples of these corrections are described in more detail below.

[0166] At 1650, an output is provided (eg, displayed) of the movement of the endoscope external to the patient or training phantom.

[0167] E. Tracker Performance Evaluation

[0168] The real-time endoscopic motion tracker was tested using an Olympus GIF-XQ10 gastroscope with a maximum insertion length of 102.5 cm and an insertion tube diameter of 9.8 mm.

[0169] The positional accuracy of the system was evaluated. Each trial began with the tracker's sensor cuff positioned so that the 20cm line was aligned with the outer edge of the sensor cuff. When the trial began, the endoscope was inserted 80cm. The camera first imaged the 20cm line and then continued to translate until the 100cm line reached the outer edge of the sensor cuff. Finally, the endoscope was retracted until it reached the 20cm starting position again.

[0170] Calibration and calculation

[0171] A calibration and calculation method for an endoscope tracker includes steps related to one or more of tracker ball calibration, motion estimation, camera calibration, and camera feature detection. The tracker ball and camera of the tracker are calibrated before being used to accurately record the motion of the endoscope. The information from all sensors is then combined to estimate the motion of the endoscope in real time.

[0172] A. Trackball Calibration and Motion Estimation

[0173] The X13 trackball is a USB device designed to let humans interact with computers. The desired tracking force of the trackball can be easily applied by the user's hand. The situation is different in the sensor cuff of an endoscope motion tracker. There, the endoscope is sandwiched between two trackballs, and the tracking force applied to each trackball varies as the operator moves the endoscope via the sensor cuff. If the tracking force is too low, then the endoscope may slip past the trackball. If it is too high, the trackball does not rotate smoothly. Therefore, when the trackball tracking force is outside the expected range, the trackball usually records a lower number of counts. This situation is mitigated by periodically reconciling the inputs from the two trackballs and retaining the results from the trackball that recorded more counts during that time. The absolute values ​​of the new position and rotation counts are added in an orthogonal manner to decide which trackball recorded the most counts. In this case, the maximum of the two values ​​is less sensitive to error than the average.

[0174] The real-time endoscope motion tracking program reads input from both trackballs by polling them every 8ms (this is determined by the camera frame rate, and for higher speed cameras the frame rate can be reduced to, for example, 4ms, 2ms, 1ms, etc.). In at least one embodiment, the 8ms time value can be the low-level USB polling rate used by Linux, for example, where the program is running on an ARM CPU and Linux, and the USB polling rate is negotiated by the operating system with the trackballs to be 8ms. For each trackball, the program receives a timestamped reply if the trackball recorded motion. If both trackballs measure changes in position and rotation, up to four changes can be recorded in a single polling cycle. Each cycle in which one or more changes are recorded is counted and this value is referred to as the sample count (n sample ) is very useful. Polling continues until the sample count reaches n rec =50 (can be changed to higher counts such as 20, 100, 400, etc. for devices with such capabilities) and then compare the results from the two trackballs. A sample count of 50 provides approximately 8 corrections for every 5 cm of displacement. The number of samples needs to be high enough to statistically select between the top and bottom differences between the top and bottom trackballs. The sample count is preferably low enough to make several corrections for each 5 cm segment to reduce the percentage uncertainty. The sample count typically varies in proportion to the number of counts per 5 cm (approximately =400 to 420 counts in Table 2). This results in a dynamic sampling rate that coordinates the trackball results more frequently when the trackball is moving continuously and does not coordinate the trackball at all when the endoscope is stationary. During use, counts will not increase unless the trackball records movement; if there is no movement, corrections will not be made until after movement begins again.

[0175] Table 2: Trackball displacement calibration. Shows the counts returned by the two trackballs for every ±5 cm displacement.

[0176]

[0177] The best estimate of the endoscope motion is updated each time the tracker ball is coordinated using the more reliable tracker ball results from each time interval. rec = 50 can be configured to change the frequency of the trackball coordination. During the trackball rotation calibration process, set n rec =50, the trackball is coordinated 11-13 times for every 360° rotation of the endoscope. The speed of the motion can be calculated by dividing the length / rotation value by the time. Similarly, the acceleration of the motion can be calculated by dividing the speed value by the time. The tracker can also measure the average linear and angular velocity of the endoscope at each coordination of the trackball, because the data obtained from the trackball measurement is time-stamped.

[0178] Table 3: Trackball rotation displacement calibration. Shows the trackball counts returned per ±360° rotation.

[0179]

[0180] The trackball is calibrated by recording its output during motion, including insertion, removal, clockwise rotation, and counterclockwise rotation. This calibration process is completed multiple times during the development of the tracker.

[0181] Position calibration was achieved using four insertion movements and four withdrawal movements. All insertion movements started at an insertion length of 20 cm and ended at 100 cm. The maximum insertion length of the GIF-XQ10 endoscope is 102.5 cm. The withdrawal movement was identical but in reverse. Ground truth points were recorded manually every ±5 cm, for a total of 16 points per ±80 cm movement. Reference Figure 5 , the start, end, and ground truth points are recorded when their corresponding white lines align with the left edge of the sensor sleeve.

[0182] The position calibration shown in Table 2 is a combination of insertion and removal results, as there is no significant difference between them. The results include 124 data points (8 movements x 16 translations – 4) because 4 of the ±5 cm translations were outliers that could not accurately track the endoscope motion.

[0183] Table 2 includes distance conversion factors for the bottom trackball, top trackball, and coordinate position. In most translations of ±5 cm, the top trackball registers less motion than the bottom trackball. This is because the forces exerted on the two trackballs are different as the endoscope moves through the sensor sleeve. If the friction of the endoscope on the trackball is too low, it may not register parts of the endoscope's motion.

[0184] It was experimentally determined that the most accurate tracking is achieved by periodically comparing the motion measured by the two trackballs (finding the maximum of the two readings), which is used to update the real-time estimate of the endoscope motion. This allows the endoscope tracker to be calibrated based on the number of coordinated trackball counts produced during a known calibration motion. This approach is reasonable because the relative uncertainty of the coordinated positions in Table 2 (approximately 3%) is less than the >5% for one trackball.

[0185] Rotational calibration was performed by completing one motion of +2880° (8 CW rotations) and one motion of -2880° (8 reverse CW rotations). After each rotation (±360°), a ground truth point was recorded by aligning temporary arrows on the insertion tube and the sensor sleeve. There was no significant difference between clockwise and counterclockwise rotations. The results are shown in Table 3. In at least one embodiment, the calibration factors used during actual use are the coordinated trackball values. These can be hard-coded into the motion tracking program for the endoscope model being used. These calibration values ​​are used to convert the counts of the USB trackball (whose internal motion unit is counts) into actual position changes (in cm) and rotation changes (in degrees).

[0186] In at least one embodiment, for example, calibration performed using a trackball can be used to derive a position slip factor for every 5 cm of insertion and a rotation adjustment factor for every 180° of rotation based on the distance of the actual position and rotation measurements from the coordinated position and rotation measurements. Thus, after these factors are determined, they are used to multiply the actual measurements to obtain calibration-corrected position and rotation values. The trackball calibration can then allow the trackball counts on both axes to be converted to real-world cm and degrees, respectively. The average slip can then be processed by comparing (coordinating) the two trackballs and maintaining the larger value during calibration. The same adjustments can then make the measurements reliable during operation.

[0187] B. Camera Calibration

[0188] The leader and follower cameras were calibrated by capturing multiple images while using the camera settings summarized in Table 1. To remove noise, the camera calibration routine saved an average image of a series of 256 images that were cropped to the endoscope ROI. Three main images were acquired during the calibration process for each camera: a background reference image, an image showing white line features, and an image showing white lines with black line gap features facing the camera. Camera calibration can be used for at least one purpose, such as: determining a threshold value (such as line, gap, or glare); checking that the constants used to crop the camera image to display the endoscope are correct; and / or obtaining images required for image subtraction.

[0189] Background reference image I ref The black part of the endoscope insertion tube and the glare of the LED light reflected from its surface are shown. The glare is similar to Figure 4 I ref The glare that appears in the endoscope is captured by inserting the endoscope into the sensor sleeve at a position in the camera FOV where no white lines or numbers are visible. These images are then loaded by the endoscope tracker 250 to perform background subtraction during real-time feature detection. This enhances the signal from the white line features and helps to mitigate glare.

[0190] Manually inspect images showing white and black line gap features. Select the white line features (e.g., as regions of interest in GIMP or ImageJ), and use a pixel histogram (made from the intensity values ​​of all pixels in the cropped image) to select a value of 50 as the threshold to distinguish the white lines from the black background (T line ). The same threshold of 50 is selected to detect the black line gap (T gap ) edge. When the position of glare in the image changes, the glare threshold (T glare )180 to eliminate glare. In at least one embodiment, there is a background subtraction step in the code that removes glare that appears in the reference background image of the black portion of the endoscope. However, as the endoscope bends and passes through the sensor sleeve, the glare appears to move a few pixels. The glare threshold is chosen to try to correct for this. The values ​​50 and 180 are convenient thresholds for 8-bit values ​​(i.e., values ​​ranging from 0 to 255), where the pixel values ​​from the camera are 8-bit integers. For the camera, 0 represents black and 255 represents 8-bit white. Table 1 shows that for each image pixel, the camera grayscale intensity value is an 8-bit value in the range of 0 to 255. The value 50 is the threshold used to find white lines (by averaging all pixels in the image column). Values ​​greater than 180 (or less than -180) after subtracting the background are considered glare. This means that these pixels are of no use and are set to 0 (no difference) in the image after background subtraction.

[0191] Finally, the process of imaging the feature is repeated, positioning the line at different locations in the camera's field of view. This step measures how the apparent width of the white line and the height of the black line gaps change as they move within the FOV. These values ​​are used to check that the effective size of the imaged feature is close to the expected value to reduce false positives.

[0192] For example, in at least one embodiment, a calibration performed on the imaging can be performed to determine background image pixel values, which can then be used to perform image subtraction on images obtained during actual use of the endoscope tracker during the procedure. This can include using the background image for background selection and selecting 8-bit (in the range of 0-255) image processing constants. This can also include setting cropping parameters and the correct size range (in pixels) of image features (white lines and black line gaps in white lines).

[0193] C. Camera feature detection

[0194] The software for the endoscope tracker can be a multithreaded C++ code where feature detection uses its own dedicated thread. This means that to achieve real-time performance at 90Hz, the software must process each image within 11.1ms. The algorithm can be designed to detect valid white line future and black line gap features if they appear in the camera's FOV. Black line gap features are always found within larger white line features. At this point, image rotation and endoscope ROI cropping have been completed. The feature detection algorithm is divided into five stages: background subtraction, glare correction, white line detection, black line gap detection, and image feature verification.

[0195] Background subtraction is used to enhance the signal of the white scale lines on the endoscope and correct for most non-uniformities in the illumination across the camera FOV. To perform background subtraction, for the acquired image being processed, convert the image pixels (I0) to signed 16-bit integers and then subtract the reference image, I from the result. ref , as shown in equation (3). If the intensity value of the pixel is less than -T glare or greater than +T glare , then most of the remaining glare is corrected by setting pixels with extreme grayscale values ​​to value 0, as shown in equation (4).

[0196] I1=int16 (I0)-I ref (3)

[0197]

[0198] Next, the algorithm attempts to detect white line features if they appear in the image. First, the image (μ col ) is the average intensity of the pixels in each column:

[0199]

[0200] Then, compare the value of each column with the threshold T line = 50 to detect white image features. Finally, if there is a white scale line in the frame, a one-dimensional (1D) distance transform is used to find the center of the white scale line and determine whether the width of the line is close to the expected value to determine whether it is valid.

[0201] If white tick mark features have been found, a similar approach is used to detect possible black line gap features. First, the image is cropped to the ROI of the white line features. Then, the pixel row average is calculated and compared with T gap = 50 to detect black areas. Finally, a 1D distance transform is used to locate black line gaps and verify their heights (eg, the determined height is close to the actual or expected height).

[0202] Testing Procedure

[0203] Many tests were completed to test the performance of the endoscope tracker throughout the development process. This section describes the tests performed on the prototype endoscope tracker. These tests and their results accurately reflect the overall performance of the endoscope tracker. Other omitted tests showed similar results with few missing white line features and black line gap features.

[0204] Three position tracking trials were completed to test the position error of the endoscope tracker and how the error relates to the average speed during the movement. The position error is defined as the difference between the measured position and the ground truth position. Each trial consisted of an insertion movement of 80 cm and a removal movement of 80 cm. These movements were very similar to the calibration movements, except that: the movement speed was varied between trials (slow, medium, fast) and the camera recorded 15 translations (± 5 cm each) for each movement. One less translation was recorded for each movement because the camera data corrected the error of the box center, which was 19 mm from the left edge of the sensor sleeve.

[0205] The data from the position tracking trials can be organized into three groups by considering each translation individually and classifying it according to its average speed. Group 1 consists of 31 slow translations with speeds between -10 mm / s and +10 mm / s. Group 2 includes fast exit translations with speeds between -40 mm / s and -10 mm / s, and Group 3 includes fast insertion translations with speeds greater than +10 mm / s.

[0206] Two trials were completed to test the rotational error of the device and its ability to measure rotational velocity. Rotational error is defined as the difference between the measured rotation angle and the ground truth. Both trials started with a movement of +1440° (4 CW rotations) followed by -1440° (four reverse CW rotations). The movement was half the size compared to during calibration, and the RasPis connected to the two cameras could run software that attempted to correct the accumulated error after each rotation of ±180°.

[0207] result

[0208] Figure 5 The measured positions for a single calibration test are plotted against the ground truth 500. The inset shows the regions where the raw displacements 520 before correction have the largest positive and negative errors compared to the ground truth. Line 510 shows the displacements in an ideal situation and line 530 shows the displacements after correction.

[0209] Figure 6 The position displacement ground truth is plotted versus the position error from the ground truth 600. Line 610 shows the original displacement error and line 620 shows the corrected displacement.

[0210] Figure 7 Position displacement ground truth is plotted versus position error (relative to ground truth) 700. Line 710 shows the bottom tracking ball error, line 720 shows the top tracking ball error, line 730 shows the optimum before correction, and line 740 shows the ideal case of zero displacement error. Figure 7 Shown is a graph of the output from two trackballs that were coordinated every 5 cm to create the best combined estimate before correction.

[0211] The rotation counts measured by the two trackballs are reconciled every 50 trackball updates that occur. An update occurs every time one or both trackballs record a count. The reconciled output is shown as the uncorrected rotation error. The corrected rotation error uses the ground truth data recorded every 360° to correct the accumulated uncertainty to zero. The rotation error is corrected in a similar manner to how the lateral interpolated error is corrected.

[0212] Figure 8 The rotational displacement ground truth is plotted versus the rotation error from the ground truth 800. Line 810 shows the original rotation error and line 820 shows the corrected rotation error. Figure 8 A graph showing the error in tip rotation, which can be reduced by dynamic correction to avoid cumulative error.

[0213] The position outputs of the two trackballs are coordinated and a timestamp is recorded whenever 25 trackball updates occur. This allows an average velocity to be calculated each time. The best estimate selects the trackball with the highest velocity. The tracker is unique in that it can measure insertion velocity in real time during the endoscopy procedure. This can be done in real time because velocity and acceleration along the axis of insertion and rotation are measured, in contrast to traditional methods of measuring tip position and velocity / acceleration in random 3D space, which is less accurate / useful. The graph on the left shows that the 5cm insertion did not occur in one smooth motion, but rather in two separate movements.

[0214] Fig. 9 The trackball update number is plotted versus rotational speed 900. Line 910 shows the bottom trackball insertion speed, line 920 shows the top trackball insertion speed, and line 930 shows the best combined estimate. Fig. 9 A graph showing real-time measurement of the insertion speed of the endoscope is shown.

[0215] The location tracking results are as follows: Figure 10-11 shown. Fig.10 A graph of position error and average velocity results 1000 is shown. Fig.11A graph of position error versus average velocity for each translation in group 11-3 1100 is shown.

[0216] exist Fig.10 In the , arrows show the direction of insertion or exit in each plot. No white line features were omitted in these trials. Fig.10 Results 1000 of (a) slow insertion 1010, (b) slow removal 1020, (c) fast insertion 1030, and (d) fast removal 1040 are shown. Fig.10 In the figure, the white line image features are defined as the ground truth and they are successfully detected by the camera every 5 cm to reset the position error to zero. In the figure, the translation of every 5 cm is shown as a diagonal line and the error correction is shown as a vertical line. Line 1050 shows the average speed and line 1060 shows the position error.

[0217] exist Fig.10 The relationship between position error 1060 and average velocity 1050 is shown in FIG. For (a) insertion and (b) exit at speeds less than 10 mm / s, position error 1060 is small and distributed around zero. When the speed exceeds 10 mm / s, the position error is generally in the same direction of travel, i.e., positive for (c) insertion and negative for (d) exit. This shows that at speeds > 10 mm / s, the position error is positively correlated with the average velocity.

[0218] exist Fig.11 The same trend can be observed in Fig.11 The position error versus velocity for all three trials 1100 is plotted. Fig.11 , the three groups are clearly separated along the x-axis. For Group 1 1110, the slope of the segmented trendline is set to 0 because the slope of the best t-line through the origin is -0.05±0.04s (p-value = 18%). The position error of the translation in Group 1 1110 is uncorrelated with its velocity. This indicates that the device is well calibrated at speeds below 10 mm / s. The trendline of Group 2 1120 passes through the point (-10, 0) with a slope of 0.46±0.05s. The trendline of Group 3 1130 passes through (10, 0) with a slope of 0.61±0.06s. The uncertainty quoted in the slope of each group of trendlines 1140 is the standard error.

[0219] Next, the translations in Group 1 1110 were used to calculate the absolute position error statistics for the endoscope tracker since it was well calibrated at speeds ≤10mm / s. The median absolute error (AE) for position tracking was 0.88mm, or 1.8% for a 50mm translation. The 10th percentile AE was 0.19mm and the 90th percentile was 2.2mm. The position errors of the faster translations in Group 2 1120 and Group 3 1130 were positively correlated with their speed. The maximum position error measured was approximately 10mm (20%) at speeds <40mm / s. The device did not function perfectly during the rotational tracking trials. It failed to detect the black line gap image feature on three occasions.

[0220] Fig.12 A graph 1200 is shown presenting (a) CW rotation tracking data from a first trial 1210 and (b) counter CW rotation data from a second trial 1220. The arrows show the direction of clockwise or counterclockwise rotation in each figure. In (a), the zero black line gap feature is missed. While in (b), when the three black line gap features pass through the center (vertical) of the leading image or the following image, they are not correctly detected.

[0221] exist Fig.12 In (a), the black line gap image features are defined as the ground truth. In (a), the camera detects all line gaps (every 180°) to reset the rotation error to zero. Line 1230 shows the rotation error and line 1240 shows the average speed. This means that the line gap is tracked correctly every time it appears in front of the leader camera or the follower camera. The corrected error is usually around ±10°, occasionally reaching a maximum of ±30°. In (b), the camera fails to accurately track the line gap and correct the rotation error three times at 1420°, 1240°, and 700°. Although the device is not perfect in (b), this case shows that it can self-correct once the next line gap is detected after rotating 360°. In this case, the rotation error for a 360° rotation is still less than 20°.

[0222] The rotational tracking AE statistics of the device were calculated based on 18 segments collected during two trials, each segment was 180° in size. The median AE of rotation was 11° (6.8%). The 10th and 90th percentile AE were 1.8° and 21°, respectively. The device was tested at angular velocities less than 40° / s.

[0223] discuss

[0224] Embodiments of the endoscope tracker described herein are capable of measuring in real time the insertion length (position) and direction (rotation) of the tube as well as their movement. This information can be continuously recorded and correlated with video, which can improve surgical documentation, relocation of biopsy sites and lesions, and navigation. This technology can also be used for physician training and quality assurance.

[0225] The endoscope tracker uses a dual-modality approach, with a tracking ball measuring the motion of the endoscope tube axis while a camera images the scale markings to correct for accumulated errors. The prototype device achieved a median AE of 0.88 mm and 11° for position and rotation, respectively, and a 90th percentile AE of 2.2 mm and 21°, respectively. This result is encouraging because a paper used MEI technology and computed tomography (CT) to locate points within a colon model and found that the position error of approximately 8 out of 12 points was greater than 10 mm

[16] .

[0226] Although locating the distal tip of an endoscope is a common problem in endoscopy, the traditional methods used vary greatly depending on the application

[16] ,

[20] ,

[21] ,

[22] ,

[23] . There are many traditional solutions for measuring the position of the distal tip of an endoscope. For example, external imaging modalities have been reported, including ultrasound

[21] , X-ray fluoroscopy

[22] , and CT

[16] ,

[20] ,

[23] . However, these modalities often significantly increase the cost of the procedure, while fluoroscopy and CT may pose additional radiation risks.

[0227] A large study of colonoscopy findings found the prevalence of large (>9 mm) tumors / polyps in average-risk screening patients to be 6.6%.

[24] Therefore, although bronchoscopic lung cancer resection may justify CT scanning, CT is unlikely to benefit patients undergoing screening or surveillance colonoscopy.

[0228] There are also MEI methods

[16] , as well as hybrid methods that combine MEI with endoscopic video or data from an inertial measurement unit (IMU)

[20] ,

[25] . Pentax, Olympus, and Fujifilm offer MEI products that detect coils that are integrated at regular intervals along the endoscope insertion tube and position them in 3D space. The coils provide the basis for a 3D image that displays the insertion tube and any loops or bends that may have formed in the colon. Information about loops and bends is very useful for navigating the endoscope within the colon, especially in training. However, MEI generally requires modification of the endoscope itself (i.e., purchasing a new endoscope or accessories), and each manufacturer uses its own proprietary technology

[12] ,

[26] ,

[27] .

[0229] Furthermore, these techniques have primarily focused on measuring the absolute 3D position of the endoscope tip. While this information is important, these techniques are limited in their ability to account for the insertion length and orientation of the lesion from the orifice. During endoscopy, particularly in the colon, the absolute 3D position of the colon (and its segments) changes within the abdominal cavity

[16] . For example, even during the same procedure, the 3D position of the endoscope tip will change as the colon moves as the patient changes body position, whereas the tip may not move within the colon itself. Likewise, this 3D position is not useful when compared to subsequent procedures.

[0230] Finally, wireless capsule endoscopes can be tracked by combining IMU data with a camera that uses visual odometry to estimate distance traveled

[28] . However, the accumulated error is large and an external device is required to provide correction.

[0231] Compared to conventional devices and techniques, the bimodal approach described herein for locating the endoscope tip is found to be less error prone and more suitable for endoscopic examinations than existing solutions. At least one embodiment of the endoscope tracker directly described herein measures the motion of the endoscope insertion tube outside the patient's body, which is correlated to the insertion and rotational motion of its distal tip. This correlation should be present for most commercial gastrointestinal endoscopes, as they have a fixed-length, rotating, rigid insertion tube.

[0232] Existing solutions focus primarily on positioning the endoscope tip in absolute 3D space, with little consideration of their relative position in the colon. For example, a hybrid tracking approach for bronchoscopy using CT, MEI methods, and endoscopic video achieved an error of 2.5 mm and 4.7°

[20] , which is adequate in clinical practice. However, the 90% error of the prototype endoscope tracking technology described in this paper is 2.2 mm and 21°. Positional accuracy is similar and rotational error is adequate for colonoscopy. For example, if a 12-hour clock face is used to describe the position of a polyp in an image, the 20° accuracy of the prototype endoscope tracking technology described in this paper is lower than the typical 1 hour / 30° error used in clinical practice.

[0233] While the endoscope tracking technology described in this article can be used alone, it can also be combined with other tracking solutions to create a more accurate hybrid solution that measures the movement of the endoscope outside the patient and its distal camera.

[0234] The inventors have discovered that rotation (or orientation) of the endoscope tip provides additional positional information of the biopsy / lesion site and is useful for detecting and / or avoiding looping

[29] and improving navigation. Current commercial 3D localization techniques are not able to measure rotation. Bernhardt et al. reported rotational motion tracking of intraoperative CT images

[23] . For most endoscopic procedures, CT radiation is avoided whenever possible. Other previously reported techniques use endoscopic video

[16] ,

[23] and IMU motion sensors

[16] ,

[25] ,

[29] . These approaches are limited to bronchoscopy applications, where endoscopic images have significantly more landmarks than colonoscopy

[16] or have significantly more accumulated errors

[25] ,

[29] .

[0235] Second, the above results (e.g. Figure 5 and Figure 6 ) also demonstrated that insertion speed can be calculated and recorded based on real-time insertion length measurements. Although insertion speed is not typically measured in clinical practice, it is an important parameter for training new endoscopists as well as for quality assurance. Such applications do not typically require regulatory approval, thus opening up a faster path to conversion. Therefore, in at least one embodiment, feedback that can be provided to the user can include insertion speed. This can be displayed in real time during the procedure, and / or can be provided after the procedure is completed in a post-operative report.

[0236] Third, GI endoscopic screening and surveillance, especially colonoscopy, is a large-scale, population-wide program. The cost of equipment / facility and access to qualified endoscopists should be considered when developing new technologies. The endoscope tracker technology described in this article is designed to be an add-on adjunct device that is compatible with most endoscopes on the market. This means that it can be used with existing endoscopes after a separate regulatory approval process. This add-on approach can significantly reduce the barriers and costs of technology conversion when using the new endoscope tracker technology described in this article.

[0237] Notes and motivation

[0238] Colonoscopy is the gold standard for detecting colorectal cancer. However, the tumor location (colon segment) reported during preoperative colonoscopy remains inaccurate for 10–17% of patients

[12] . Accurately recording the location of tumors and polyps found during colonoscopy is often a challenge for endoscopists. Colon polyp surveillance monitors patients at high risk for polyps through regular colonoscopy. Improved polyp localization would allow the polypectomy site to be examined for recurrent polyps during subsequent colonoscopies. It should be understood that other endoscopic procedures (e.g., EGD, bronchoscopy) face similar challenges.

[0239] In clinical practice, insertion and rotation speeds vary widely between different phases of the endoscopic procedure and even more so between different endoscopists. However, these observations are anecdotal and qualitative, as there is no existing technology to objectively measure the speed of this movement.

[0240] The real-time endoscopic motion tracker described in this paper improves on previous approaches. An externally mounted device records the motion of an endoscope in real time with the goal of improving the accuracy of lesion localization during procedures such as preoperative colonoscopy. The device uses a tracking ball to measure the position and rotation of the endoscope and uses camera-based feature detection to periodically correct for the accumulated uncertainty in these measurements. Each time the position or rotation of the endoscope is corrected by camera-based feature detection, the associated velocity is calculated. A prototype device based on the teachings of this paper was found to accurately measure the motion of the endoscope axis with a median AE of 0.88 mm for position and 11° for rotation. The real-time motion record produced by the endoscope tracker is a valuable tool for clinical endoscopic surgery as well as for answering a number of research questions. For example, it can be synchronized with recorded video frames to allow for more accurate recording of lesion location during colonoscopy. The recorded position and tip orientation (rotation) can be combined with the video to create a 3D reconstruction of the colon or an unfolded view of its luminal surface

[30] ,

[31] . This technology can also provide objective and quantitative measurements of endoscopic surgery for new residents in training.

[0241] Alternatives to the above motion trackers can offer additional benefits. Because the tracker ball is in direct contact with the endoscope, the tracker must be thoroughly cleaned and sterilized after use. A non-contact, fully camera-based tracker can eliminate the sterilization requirement.

[0242] Reference now Fig.13A , which shows a perspective view of an embodiment of an endoscope tracker 1310 that is capable of measuring the movement of an endoscope in real time through a fixed sensor device. The endoscope passes through the center of the endoscope tracker 1310, while three cameras 1310a, 1310b, 1310c are mounted on three separate distal ends of a star-shaped (e.g., Y-shaped) device. In the embodiments of the endoscope tracker 1310 described herein, some or all of the system 100 can be used to perform any one or more functions requiring computer functions (e.g., digital input, digital output, software processing, image processing, etc.). Therefore, the tracker 1310 includes at least one processor, such as in the various embodiments described for the system 100, in addition to the possible presence of two followers, because there are three cameras.

[0243] Fig. 13B A photograph of an embodiment of a bracket 1320 for an endoscope tracker 1310 (or "endoscope tracker system") is shown. Fig. 13CA perspective view of an embodiment of a bracket 1330 for an endoscope tracker 1310 is shown. Fig.13D A close-up photograph of an embodiment of an internal housing 1340 of an endoscope tracker 1310 is shown. The support 1330 can have a housing 1340 with spaced apart distal ends and a hole in the center of the housing 1340 for receiving an endoscope with an insertion tube.

[0244] The sensor device uses at least two cameras (e.g., 2, 3, 4 or more cameras) to cover a full 360-degree view of the endoscope insertion tube 305 around its transverse axis to measure changes in position and rotation over time. For 4 or more cameras, the housing can be modified to provide a mounting position so that the cameras are spaced apart according to the position of 360 degrees / n relative to the orifice of the insertion tube of the endoscope, where n is the number of cameras. For example, three cameras 1310a, 1310b, 1310c can be used, wherein each camera 1310a1310b and 1310c is fixed at a point 120° relative to the longitudinal axis of the endoscope insertion tube, which provides an uninterrupted view of the markings on the insertion tube and its sheath. Also for example, the at least two cameras can be mounted at one of the distal ends so that each camera points to the center of the housing 1340, and each camera has an uninterrupted view of the endoscope when it is received in the hole in the center of the housing 1340.

[0245] This design of this embodiment is in contrast to other designs that use a contact point (e.g., a trackball) that physically contacts the insertion tube and directly measures its position and rotation changes. In this design, motion can be measured optically by cameras 1310a, 1310b, 1310c without any physical contact between the sensor (e.g., camera) and the endoscope insertion tube. However, in an alternative embodiment, two of the three cameras 1310a, 1310b, and 1310c can be used as described below.

[0246] In at least one implementation of this embodiment, the camera is configured to capture video of the endoscope (or a portion thereof, such as a sheath of the endoscope (e.g., the endoscope insertion tube)) as it moves relative to the fixed field of view (FOV) of the cameras 1310a, 1310b, 1310c. Each frame of the camera video is processed in real time after it is captured. The software of the endoscope tracker 1310 tracks the apparent movement of white markers that appear on the black surface of the endoscope insertion tube. Specifically, most commercial endoscopes include white line markers that appear at regular intervals (e.g., every 5 cm) along the length of the endoscope insertion tube. The software of the endoscope tracker 1310 uses these to correct for any accumulated errors that build up in the measurement when large changes in position occur. The line markers also have a black gap at a point on their circumference to correct for rotational errors. The software of the endoscope tracker 1310 and the corresponding functions performed can be, for example, similar to the software described with reference to the endoscope tracker 250, but modified for use without a trackball, in which case measurements of position and / or rotation can be refined to a scale corresponding to one or more pixels (e.g., 1 pixel, 2 pixels, 5 pixels, 8 pixels, etc.).

[0247] In at least one implementation of this embodiment, the endoscope tracker 1310 runs program instructions to track in real time the position and orientation of the distal end of the endoscope based on an image of a portion of the endoscope in the FOV of at least two cameras. The program instructions may include, for example, (a) obtaining a real-time image of the portion of the endoscope in the FOV of the at least two cameras from the at least two cameras; (b) measuring lateral and rotational movement of the endoscope based on markers printed on the endoscope and captured in the real-time image; and (c) correlating the real-time images from the at least two cameras to calculate changes in the measured lateral and rotational movement to determine the position and orientation of the distal end of the endoscope. For example, the correlation may be performed by performing steps 5 to 7 described below.

[0248] Fig.17 A flow chart of an exemplary embodiment of a method 1700 of tracking an endoscope using a contactless tracker is shown. The method 1700 can be performed using some or all portions of the system 100 and / or some or all portions of the endoscope tracker 1310.

[0249] At 1710, the camera captures motion video of the endoscope as it moves relative to the camera's fixed field of view, thereby generating endoscopic image data (or video data). Image features can be obtained individually as pixel positions (or pixel values) for each camera.

[0250] At 1720, the apparent motion of the endoscope is tracked using markings (e.g., scale markings) that appear on the insertion tube of the endoscope. Alternatively or additionally, the pixel values ​​can be converted to calibrate the position and rotation values ​​based on geometry and the distance from the endoscope to the camera. For example, with three cameras in a Y configuration, there is more overlap in the field of view of each camera, which will make the geometry-based approach more reliable.

[0251] At 1730, the accumulated error established in the measurement of the apparent motion is corrected. The correction of the accumulated error (or uncertainty) can be achieved by one or more of the corrections for position error or rotation error as described herein. For example, image data of the endoscope and scale markings along the insertion tube acquired from cameras (e.g., two cameras facing each other, three cameras in a Y configuration, four cameras in a diamond configuration, etc.) can be used to correct for position error and rotation error.

[0252] At 1740, an output is provided (eg, displayed) of the movement of the endoscope external to the patient or training phantom.

[0253] The endoscope tracker 1310 may use some or all of the above-described methods, functions, and / or procedures as they relate to images obtained from cameras that function similarly to a two camera setup with two tracking balls, but are modified to work without the tracking balls.

[0254] For contactless (or camera-only) endoscope trackers, the motion tracking program / software can use images that include the "white line" and "black line gap" features to measure position. The motion tracking program can estimate the current position (e.g., measured changes in position and rotation angles) using the "history" of the endoscope procedure - for example, entry of the distal tip into the orifice, followed by insertion into the rest of the endoscope; and during withdrawal, withdrawal of the proximal end followed by withdrawal of the distal end. In effect, the "black line gap" image feature is found within the "white line" image feature; the "black line gap" cannot exist without being part of the larger "white line" image feature.

[0255] Position and rotation calculation and calibration

[0256] The endoscope tracker 1310 may employ a contactless (or camera-only) endoscope motion tracking program that calculates and calibrates position and rotation by running some or all of the following program instructions:

[0257] (1) Using a single effective viewpoint camera model, the distortion parameters for each camera are determined. These parameters are likely to be similar for a given brand and model of camera units, assuming they have not been modified since leaving the factory.

[0258] (2) Dedistort the image from each camera using the parameters of the camera model. The camera scene can be reprojected onto a screen at a fixed distance from the camera, which is equal to the distance between the endoscope and the camera.

[0259] (3) Using triangulation and the known distance from the endoscope to the camera and / or the known diameter of the endoscope, the pixel distance in the dedistorted image is converted to distance units (e.g., cm or mm) for position tracking using white line features detected in the image. The triangulation and real-world distances can be used to create similar triangles (same angles) to convert pixels to real-world units.

[0260] (4) Using the known diameter of the endoscope and the known distance from the endoscope to the camera, find the center of the black line gap in pixel coordinates of at least one camera and use triangulation to find the corresponding angle (in degrees) for rotational tracking - approximating the endoscope surface as a cylinder within the camera FOV.

[0261] (5) Record the starting position (e.g., in mm or cm) and update the position in real time using the position tracking information from multiple cameras. If the position tracking information from different cameras is generally consistent, a suitable algorithm can be used to combine (e.g., average) the results from the cameras. If the position tracking information from different cameras appears to reflect the same true measurement results without significant measurement errors or lags (time) between the measurements, then the information is "consistent". In at least some cases, the measurements (from the cameras) remain close compared to the measurement resolution of the device. For example, assume that the linear distance (in mm) counted by the endoscope tracker is about 0.12 mm per count. In this case, a difference in the measurements that is significantly greater than 0.12 mm would indicate an inconsistency. In this case, the angular distance in degrees may be about 1.3°. These values ​​represent the resolution limits of a particular embodiment (in mm and degrees (°)). For example, the resolution in any particular embodiment may depend on how many pixels each camera has, the distance from the camera to the endoscope, and whether the image is clear enough to measure the position with an accuracy as low as one pixel (or a small number of pixels). If there is noise in the position tracking information and / or if the cameras are inconsistent about the position of the endoscope, a Kalman filter (or similar noise reduction algorithm) can be used. For example, a mean can be used, in which case the filter is a mean filter. Another option that handles noise well is a median filter, which takes the median of the last N measurements rather than the average, e.g. the median of three cameras. The Kalman filter takes a series of measurements over time from one or more noisy signals and models how the actual values ​​measured change over time. The Kalman filter can be used on noisy data when a smoother output signal is preferred.

[0262] (6) Record the starting rotation angle in degrees and update the rotation in real time using the rotation tracking information from multiple cameras. The recording can be done "intelligently", that is, if the image processing results are confused due to other elements in the camera FOV, the goal is to filter out outliers, erroneous data, empty data, etc. The vertical center of a camera (on the y-axis) can be used as the origin for measuring the rotation angle. An offset of 360 / n degrees can be applied between adjacent cameras, where n is the number of cameras used. For camera tracking only embodiments, the number of cameras can be at least two, but three or more cameras can give better results. If black line gaps appear in the FOV of multiple cameras, the rotation results of these cameras can be averaged. A Kalman filter can also be used in this case to filter out noise.

[0263] (7) In a three-camera setup, in addition to the white line and black gap, the motion tracking program / software uses other markings on the surface of the endoscope tube (e.g., 5 cm, 10 cm, 20 cm, etc.). In at least some cases, the marking (e.g., 20 cm) explicitly states the position to which the line next to it corresponds. This can be used to start recording at any insertion length (position) within the colon, restart recording in the event of an error, confirm that there are no errors in position tracking, and / or correct position tracking errors that may occur due to obstructions between the camera and the endoscope or other reasons. An image recognition algorithm can be used to identify the marking and compare it to prior knowledge of the marking and its location on a particular type of endoscope (e.g., surgical history, medical device data, video data, image data). The image recognition algorithm can be, for example, a text recognition algorithm such as optical character recognition (OCR). There are several image recognition techniques that can be used to detect image features, of which OpenCV, an open source computer vision library, is an example of an image processing technique that can be used that is known to those skilled in the art. For example, an image recognition algorithm can be used to crop out the desired portion of the image and find the average intensity of the image columns (for white lines) or rows (for black line gaps), as previously described; the algorithm can then take the row average or column average of the image or a portion thereof for further processing, as previously described for the camera plus trackball embodiment. The parameters used for cropping can be determined empirically.

[0264] In at least one embodiment of the endoscope tracker 130 (e.g., when used with the system 100), an ROI in an image obtained by a camera can be detected using, for example, an edge detector, a rectangle detector, or a blob detector. Once the endoscope is found in one frame, techniques such as active contours can be employed. Starting with the ROI of the previous frame, active contours (e.g., edge detection) can find a new ROI in a new frame.

[0265] One of the advantages of at least one embodiment of the endoscope tracker 1310 (e.g., when used with the system 100) is that it improves patient outcomes for medical endoscopy. Background research has focused primarily on improving colonoscopy to treat colon cancer. The endoscope tracker 1310 allows for precise localization of abnormal or cancerous colon tissue and polyps, for example, by improving localization to within 1 mm compared to a rough estimate of location using 5 cm markers. This allows for measurement of disease progression and treatment effectiveness at specific sites during a series of colonoscopies. Improved localization also allows for correct treatment selection for larger tumors and allows for better planning of minimally invasive surgeries to remove tumors.

[0266] Another advantage of at least one embodiment of the endoscope tracker 1310 (e.g., when used with the system 100) is that it improves the training process for new endoscopy operators. It can be used to monitor how smoothly a trainee inserts and moves an endoscope. It can also record how an expert operator completes a colonoscopy procedure, providing a useful benchmark for trainees.

[0267] In at least one embodiment of the endoscope tracker 1310 (e.g., when used with system 100), positional (or lateral) and rotational information (or data) from the endoscope tracker 1310 can be combined with real-time video from a camera at the tip of the endoscope to determine whether the insertion tube of the endoscope is forming a loop.

[0268] In at least one embodiment of the endoscope tracker described herein, the system 100 measures position and rotation, and corrects position errors by imaging white lines that are present at regular intervals (e.g., every 5 cm) along the endoscope. Since the black gaps in each white line only appear on one side of the endoscope, rotation correction can be omitted. Therefore, the gaps are typically hidden outside the field of view of one camera. Variations on the implementation (e.g., changing the color of the lines, changing their spacing, providing rotation correction) may be made where it is desired or necessary to achieve the same result.

[0269] One of the advantages of at least one embodiment of the endoscope trackers described herein (e.g., when used with system 100) is that they can be used to provide training for endoscope operators. It can monitor how smoothly a trainee inserts and moves an endoscope into a training phantom or patient. This can quantify how their technique differs from that of their peers, or from that of an expert operator who has performed many endoscopic procedures.

[0270] Another advantage of at least one embodiment of the endoscopic tracker described herein (e.g., when used with system 100) is that they are beneficial in more complex applications such as clinical use. The endoscopic tracker can allow for more precise localization of abnormal or cancerous colon tissue and polyps during colonoscopy compared to traditional methods. Better localization can allow for more or different treatment options. It can also improve the planning of minimally invasive surgical resections of large colon tumors by better determining the location of the tumor.

[0271] In at least one embodiment of the endoscope tracker 1310 described herein, software for the endoscope tracker determines how the endoscope moves over time. The software uses an image processing pipeline, and each camera frame captured contains a timestamp. The time required for each part of the tracker logic can be measured and compared to the time used to acquire each frame (e.g., at 90fps) (e.g., 11ms).

[0272] In at least one embodiment of the endoscope tracker 1310, a dual camera setup is used. In the case of two cameras, there may be disagreements between the cameras due to the delay in the second camera recording the frame. Therefore, the system is programmed to compensate for the delay.

[0273] In at least one embodiment of the endoscope tracker 1310, it uses a standalone design without a "housing". For example, this configuration can allow for a completely independent camera. In this case, a calibration procedure allows for registering the relative position and orientation between the camera and the endoscope to perform measurements.

[0274] In at least one embodiment, the endoscope tracker described herein records the movement of the endoscope outside the patient and infers how the tip of the endoscope may have moved in the same time. This information can be combined with real-time video from the endoscope tip camera to determine if the endoscope is forming a loop, or to generate an image mosaic that maps the entire inner surface of the colon to locate tumors and polyps. The graph can be expanded to show the colon in one image summary. This can reduce the number of times the full video from the endoscope camera must be viewed. Clicking on a feature of interest in the expanded graph can jump to the time code in the video when the feature was viewed.

[0275] In at least one embodiment of the endoscope tracker 1310, there may be additional motion sensors on the endoscope, additional motion sensors on the operator, and / or environmental sensors mounted in the endoscope to improve motion tracking.

[0276] In at least one embodiment of the endoscope tracker 1310 (eg, when used with the system 100), the optical system design (ie, two or more cameras) has an uninterrupted view of the endoscope insertion tube, and a lighting design that minimizes reflections.

[0277] Another advantage of at least one embodiment of the endoscope tracker 1310 (e.g., when used with the system 100) is that, when used with the system 100, it provides algorithms for correlating real-time images from multiple cameras, calculating pattern changes, and determining position.

[0278] Another advantage of at least one embodiment of the endoscope tracker 1310 (e.g., when used with the system 100) is that it can provide better coverage using three cameras (or more cameras) of a 360° view compared to a two camera setup, where, while it can also provide a 360° view, the two camera setup may have poor resolution (or image processing) at the edges of the FOV.

[0279] In at least one embodiment of the endoscope tracker 1310 (e.g., when used with system 100), it uses two or more cameras (e.g., three cameras) external to the endoscope, where the cameras do not contact the endoscope and capture endoscopic images for detecting movement of the endoscope.

[0280] Another advantage of the endoscope tracker 1310 (e.g., when used with the system 100) is that it does not require a contact point, such as a trackball. This eliminates the problem of physical contact between the trackball and the endoscope, which requires additional cleaning of the endoscope and tracker device. The non-contact nature of the design also allows for future modifications, allowing the use of a stand-alone design without a "sleeve".

[0281] Yet another advantage of the endoscope tracker 1310 (e.g., when used with the system 100) is that it reduces errors compared to contact-based sensors. Technically, mechanical sensors (tracking balls) can accumulate errors. Without contact, any errors can be corrected in real time by video-based sensors.

[0282] In at least one embodiment, the tracking system may include one trackball for providing position and rotation measurements and one camera for obtaining image data from which white line scale and black gap features are detected and whose positions are used to correct for accumulated errors in the measurements provided by the trackball. Such an embodiment would operate similarly to that described for the embodiment with two trackballs and two cameras, except that since only one trackball is used, there are no coordinated position and rotation measurements from the trackball, and since only one trackball is used, there is no need to combine the results of multiple images of the same time step from the two cameras.

[0283] It should be noted that the white scale lines and black gap markings may appear on the outer surface of the endoscope or on the outer surface of the insertion tube in which the endoscope is placed. It should also be noted that based on the standards that define the location, spacing, and size of the white line scale lines and black gap markings, these markings may appear in different locations or at different spacings. Regardless of which standard is used to present these markings, it should be understood that the trackers described herein may be modified to account for these characteristics during calibration so that the tracking system can use these markings (with respect to the standards used to define them) during use.

[0284] Although the applicant's teachings described herein are used in conjunction with various embodiments for illustrative purposes, it is not intended that the applicant's teachings be limited to such embodiments for which the embodiments described herein are intended as examples. Rather, the applicant's teachings described and illustrated herein include various alternatives, modifications, and equivalents without departing from the embodiments described herein, the general scope of which is defined in the appended claims.

[0285] Citations

[0286] [1] RL Siegel et al., “Colorectal cancer statistics, 2017,” CA. Cancer J. Clin., vol. 67, no. 3, pp. 177–193, 2017. doi: https: / / doi.org / 10.3322 / caac.21395.

[0287] [2] RL Siegel, K D Miller, H E Fuchs, and A Jemal, “Cancer Statistics, 2021,” CA Cancer J. Clin., volume 71, issue 1, pages 7–3321, 2021, doi: https: / / doi.org / 10.3322 / caac.21654.

[0288] [3] L. M. Fernandez, R. N. M. Brahim, I. Mizrahi, G. Da Silva, and S. D. Wexner, “How accurate is preoperative colonoscopic localization of colonic neoplasia?,” Surg. Endosc., vol. 33, no. 4, pp. 1174–1179, April 2019, doi: 10.1007 / s00464-018-6388-5.

[0289] [4] S.A. Acuna, M. Elmi, P.S. Shah, N.G. Coburn, and F.A. Quereshy, “Preoperative localization of colorectal cancer: a systematic review and meta-analysis,” Surg. Endosc., vol. 31, no. 6, pp. 2366–2379, June 2017, doi:10.1007 / s00464-016-5236-8.

[0290] [5] A. Azin et al., “A comparison of endoscopic localization error rate between operating surgeons and referring endoscopists in colorectal cancer,” Surg. Endosc., vol. 31, no. 3, pp. 1318–1326, March 2017, doi: 10.1007 / s00464-016-5114-4.

[0291] [6] R. Yap, D. Ianno, and A. Burgess, “Colonoscopic localization accuracy for colorectal resections in the laparoscopic era,” Am. J. Surg., vol. 212, no. 2, pp. 258–263, August 2016, doi: 10.1016 / j.amjsurg.2015.12.014.

[0292] [7] N. Piscatelli, N. Hyman, and T. Osler, “Localizing Colorectal Cancer by Colonoscopy,” Arch. Surg., vol. 140, no. 10, pp. 932–935, October 2005, doi: 10.1001 / archsurg.140.10.932.

[0293] [8] P. Ellul et al., “Colonic tumor localization using an endoscope positioning device,” Eur. J. Gastroenterol. Hepatol., vol. 23, no. 6, 2011, [online], available at: https: / / journals.lww.com / eurojgh / Fulltext / 2011 / 06000 / Colonic_tumor_localization_using_an_endoscope.7.aspx.

[0294] [9] A. S. Bryce, M. S. Johnstone, and S. J. Moug, “Improving lesion localisation at colonoscopy: an analysis of influencing factors,” Int. J. Colorectal Dis., vol. 30, no. 1, pp. 111–118, January 2015, doi: 10.1007 / s00384-014-2052-2.

[0295]

[10] “LT1371D Apem|Mouser,” Mouser Electronics. https: / / www.mouser.ca / ProductDetail / 642-LT1371D (accessed May 7, 2020).

[0296]

[11] “Pi-Light for Pi-Pan.” http: / / www.mindsensors.com / rpi / 83-pi-light-for-pi-pan (accessed May 7, 2020).

[0297]

[12] S.A. Acuna, M. Elmi, P.S. Shah, N.G. Coburn, and F.A. Quereshy, “Pre-operative localization of colorectal cancer: A systematic review and meta-analysis,” Surgical Endoscopy, vol. 31, no. 6, pp. 2366–2379, June 2017, doi: 10.1007 / S00464-016-5236-8.

[0298]

[13] A. Azin et al., “A comparison of endoscopic localization error rate between operating surgeons and referring endoscopists in colorectal cancer,” Surg. Endosc., vol. 31, no. 3, pp. 1318–1326, March 2017, doi: 10.1007 / S00464-016-5114-4-016-5114-4.

[0299]

[14] R. Yap, D. Ianno, and A. Burgess, “Colonoscopic localization accuracy for colorectal resections in the laparoscopic era,” Amer. J. Surg., vol. 212, no. 2, pp. 258-263, August 2016, doi: 10.1016 / J.AMJSURG.2015.12.014.

[0300]

[15] N. Piscatelli, N. Hyman, and T. Osler, “Localizing Colorectal Cancer by Colonoscopy,” Arch. Surg., vol. 140, no. 10, pp. 932–935, October 2005, doi: 10.1001 / ARCHSURG.140.10.932.

[0301]

[16] M. Oda et al., “Colonoscope tracking method based on shape estimation network,” Proc. SPIE, vol. 10951, March 2019, no. 109510Q, doi: 10.1117 / 12.2512729.

[0302]

[17] Mouser Electronics. LT1371D APEM | Mouser. Accessed January 18, 2022, [Online]. URL: https: / / www.mouser.ca / ProductDetail / 642-LT1371D.

[0303]

[18] RM Salinas. Raspicam. Aplicaciones of the Visual Artificial, University of Córdoba. Accessed October 14, 2022, [Online version], URL: https: / / www.uco.es / investiga / grupos / ava / portfolio / raspicam / .

[0304]

[19] D. Plowman. Raspberry Pi Documentation Camera. Raspberry Pi Ltd. Accessed on October 14, 2022, [Online version], URL: https: / / www.raspberrypi.com / documentation / accessories / camera.html.

[0305]

[20] X. Luo and K. Mori, “Robust endoscope motion estimation via an animated particle filter for electromagnetically navigated endoscopy,” IEEE Trans. Biomed. Eng., vol. 61, no. 1, pp. 85-95, January 2014, doi: 10.1109 / TBME.2013.2277609.

[0306]

[21] M. Zhang, L. Yan, S. Li, Y. Li, and P. Huang, “Ultrasound-guided transforaminal percutaneous endoscopic lumbar discectomy: A new guidance method that reduces radiation doses,” Eur. Spine J., vol. 28, no. 11, pp. 2543-2550, November 2019, doi: 10.1007 / S00586-019-05980-9.

[0307]

[22] M. Sippey, S. Maskal, M. Anderson, and J. Marks, “Use of fluoroscopy inendoscopy: Indications, uses, and safety considerations,” Ann. Laparosc. Endosc. Surg., vol. 4, p. 59, June 2019, doi: 10.21037 / ALES.2019.06.05.

[0308]

[23] S. Bernhardt, S. Nicolau, V. Agnus, L. Soler, C. Doignon, and J. Marescaux, “Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery,” Med. Image Anal., vol. 30, pp. 130-143, May 2016, doi: 10.1016 / j.medi.2016.01.008.

[0309]

[24] D.A. Lieberman et al., “Colonoscopy utilization and outcomes 2000 to 2011,” Gastrointestinal Endoscopy, volume 80, issue 1, pages 43–133, July 2014, doi: 10.1016 / J.GIE.2014.01.014.

[0310]

[25] H. Ren, D. Rank, M. Merdes, J. Stallkamp, ​​and P. Kazanzides, “Multisensor datafusion in an integrated tracking system for endoscopic surgery,” IEEE Trans. Inf. Technol. Biomed., vol. 16, no. 1, pp. 106–111, January 2012, doi: 10.1109 / TITB.2011.2164088.

[0311]

[26] P. Ellul et al., “Colonic tumor localization using an endoscopepositioning device,” Eur. J. Gastroenterol. Hepatol., vol. 23, no. 6, pp. 488–491, June 2011, doi: 10.1097 / MEG.0B013E328346974B.

[0312]

[27] A. S. Bryce, M. S. Johnstone, and S. J. Moug, “Improving lesion localisation at colonoscopy: An analysis of influencing factors,” Int. J. Colorectal Disease, vol. 30, no. 1, pp. 111–118, January 2015, doi: 10.1007 / S00384-014-2052-2.

[0313]

[28] S.S. Vedaei and K.A. Wahid, “A localization method for wireless capsule endoscopy using side wall cameras and IMU sensor,” Sci. Rep., vol. 11, no. 1, pp. 1–16, December 2021, doi: 10.1038 / S41598-021-90523-W.

[0314]

[29] M. Bruce and J. Choi, “Detection of endoscopic looping during colonoscopy procedure by using embedded bending sensors,” Med. Devices, Evidence Res., vol. 11, pp. 171-191, May 2018, doi: 10.2147 / MDER.S146934.

[0315]

[30] S. Sahli, R. C. C. Wang, A. Murthy, D. Armstrong, M. J. Deen, and Q. Fang, “A 360-degree side view endoscope for lower GI tract mapping,” Phys. Canada, Vol. 71, No. 1, pp. 18, 20, 2015. [Online version], URL: https: / / pic-pac.cap.ca / index.php / Issues / showpdf / article / v71n1.0-a2394.pdf

[0316]

[31] R.C.C. Wang, M.J. Deen, D. Armstrong, and Q. Fang, “Development of catadioptric endoscope objective with forward and side views,” J. Biomed. Opt., vol. 16, no. 6, 2011, no. 066015, doi: 10.1117 / 1.3593148.

Claims

1. A real-time endoscope tracker system comprising: a stent having a housing with spaced apart distal ends and a hole in the center of the housing for receiving an endoscope having an insertion tube; at least two cameras, each camera mounted at one of the distal ends such that each camera is directed toward a center of the housing and each camera has an uninterrupted view of the endoscope when the endoscope is received in the hole in the center of the housing; and a processor in communication with a non-transitory computer-readable medium storing program instructions that, when executed by the processor, cause the processor to track in real time a position and orientation of a distal end of the endoscope based on an image of a portion of the endoscope that is in a field of view (FOV) of the at least two cameras, the processor being configured, when executing the program instructions, to: obtaining from the at least two cameras a real-time image of the portion of the endoscope within the FOVs of the at least two cameras; measuring lateral and rotational movement of the endoscope based on markers printed on the endoscope and captured in the real-time image; and Real-time images from the at least two cameras are correlated to calculate changes in the measured lateral and rotational motion to determine the position and orientation of the distal end of the endoscope.

2. The system of claim 1, wherein the at least two cameras cover a full 360 degree view of the insertion tube of the endoscope around its lateral axis for measuring lateral and rotational movement of the endoscope.

3. A system according to claim 1 or 2, wherein the at least two cameras include three cameras, each of which is fixed at a point so that two consecutive cameras are 120° from the axis of the insertion tube to provide an uninterrupted view of the insertion tube and its markings.

4. The system of any one of claims 1 to 3, wherein the at least two cameras are configured to capture video of the endoscope as the endoscope moves relative to a fixed FOV of the at least two cameras.

5. The system of any one of claims 1 to 4, wherein the processor, when executing program instructions, is further configured to identify white line markings along the length of the insertion tube to correct for accumulated errors built up in the measurements.

6. The system of any one of claims 3 to 5, wherein the processor, when executing program instructions, is further configured to identify a black gap at a point on the circumference of the insertion tube to correct for errors in rotation.

7. A system according to any one of claims 1 to 6, wherein the processor, when executing program instructions, is configured to combine data regarding lateral and rotational movement with real-time video from a camera on the tip of the endoscope to determine whether the insertion tube is forming a loop.

8. A system according to any one of claims 1 to 7, wherein the processor, when executing program instructions, is further configured to combine data regarding lateral and rotational movement with real-time video from a camera on the tip of the endoscope to generate an image mosaic that maps the inner surface of the colon being imaged to locate tumors and polyps.

9. The system of claim 1, wherein the at least two cameras include two cameras, and the processor, when executing program instructions, is further configured to compensate for a delay in recording a frame from one of the two cameras.

10. The system of any one of claims 1 to 9, wherein the processor, when executing program instructions, is further configured to use computer vision to detect movement of the endoscope based on images obtained from the at least two cameras.

11. The system of any one of claims 1 to 10, wherein the processor, when executing program instructions, is further configured to provide at least one of a position or an orientation of a distal end of the endoscope to an endoscope training simulator.

12. A method for tracking an endoscope, comprising: capturing, by the camera, video of the movement of the endoscope as the endoscope moves relative to a fixed field of view of the camera; tracking the apparent movement of the endoscope using markings present on an insertion tube of the endoscope; and An output of the movement of the endoscope external to the patient or training phantom is provided.

13. The method of claim 12, wherein the method is performed by an endoscope tracker having at least two cameras for obtaining images of portions of the endoscope that are in a field of view (FOV) of the at least two cameras, and the method further comprises: obtaining from the at least two cameras a real-time image of the portion of the endoscope within the FOVs of the at least two cameras; measuring lateral and rotational movement of the endoscope based on markers printed on the endoscope and captured in the real-time image; and Real-time images from the at least two cameras are correlated to calculate changes in the measured lateral and rotational motion to determine the position and orientation of the distal end of the endoscope.

14. The method of claim 12 or 13, wherein capturing video comprises: Determine the distortion parameters for each camera; dedistorting an image from a video captured by the camera based on the distortion parameters; Convert pixel distances in the dedistorted image to distance units for position tracking; and The known diameter of the endoscope and the known distance from the endoscope to the camera are used to determine the corresponding angle for rotational tracking.

15. The method according to any one of claims 12 to 14, wherein said tracking of apparent motion further comprises: Recording a starting position and updating the starting position in real time using position tracking information from a camera; and The starting rotation angle is recorded and updated in real time using the rotation tracking information from the camera.

Images