Apparatus and method for tracking the position of an endoscope within a patient's body

By capturing and converting reference marker data in the endoscopic handle and external optical tracker system, the problem of inaccurate position during endoscopic surgery is solved, and the accurate positioning and better manipulation of the endoscopic in the patient's body is achieved. Improves the efficiency and diagnostic effect of the surgery.

CN114795472BActive Publication Date: 2024-11-08ANDROX CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210467674.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2015-10-28
Filing Date
2016-10-26
Publication Date
2024-11-08
Estimated Expiration
2036-10-26

AI Technical Summary

Technical Problem

During endoscopic surgery, it is difficult for the surgeon to understand exactly where the endoscopic is located relative to the patient's body, resulting in improper manipulation that can lead to discomfort or pain.

Method used

Using multiple directional marker and sensor systems, the reference marker data is captured and converted through the endoscope handle and external optical tracker, generating an image of the exact position of the endoscope in the patient's body and projecting it onto the patient's body, providing Augmented reality environment to show the exact location of the endoscope.

Benefits of technology

It significantly shortens the endoscopic navigation time, improves the accuracy and manipulation of the surgery, ensures the correct distance between the endoscopic and the patient's organs, and enhances the diagnostic effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114795472B_ABST
    Figure CN114795472B_ABST
Patent Text Reader

Abstract

The present invention discloses a system and method for tracking the position of an endoscope in a patient's body during endoscopic surgery. The apparatus and method include determining the position of the endoscope in the patient's body in the endoscope's coordinate system, capturing a fiducial marker attached to the endoscope in an image by an external optical tracker, transforming the captured fiducial marker from the endoscope's coordinate system to the optical tracker's coordinate system, projecting a virtual image of the endoscope on a model of the patient's organ, and projecting or displaying the combined image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of invention patent 201680076675.1.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This specification relies on priority to U.S. Provisional Application No. 62 / 247,232, filed on October 28, 2015, entitled "DEVICES AND METHODS FOR TRACKING THE POSITION OF AN ENDOSCOPYRIGHTED OBJECT WITHIN A PATIENT."

[0004] This specification is related to U.S. patent application No. 14 / 505,387 filed on October 2, 2014 and entitled "Endoscope with Integrated Sensor" and U.S. patent application No. 14 / 697,933 filed on April 28, 2015 and entitled "System and Method for Endoscopic Scanning of Body Cavity Using Multiple Viewing Elements."

[0005] The entire contents of the above-mentioned application are incorporated herein by reference. Technical Field

[0006] The present description relates generally to endoscopes and, more particularly, to images for displaying the position of an endoscope within a patient's body, the images being overlaid on the patient's body or superimposed on an image of the patient's body. Background Art

[0007] An endoscope is a medical device used to inspect and manipulate internal body cavities, such as the digestive tract, airway, gastrointestinal system, and other organ systems. A conventional endoscope is typically an elongated tubular shaft, rigid or flexible, with a camera and a fiber optic light guide for directing light from an external light source located at the proximal end of the tube to the distal tip. Furthermore, most endoscopes are provided with one or more channels through which medical devices, such as forceps, probes, and other tools can pass. In addition, during endoscopic surgery, fluids, such as water, saline, drugs, contrast agents, dyes, or emulsifiers, are typically introduced or evacuated through the shaft. Multiple channels may be provided within the shaft, each of which is used to introduce and aspirate liquids.

[0008] Endoscopes have gained great acceptance in the medical community because they provide a tool to perform procedures with minimal trauma to the patient while enabling the physician to view the patient's internal anatomy. Over the years, many endoscopes have been developed and categorized according to specific applications, such as cystoscopy, colonoscopy, laparoscopy, upper gastrointestinal (GI) endoscopy, etc. An endoscope can be inserted into a natural orifice of the human body or through an incision in the skin.

[0009] Some endoscopes have a viewing element for viewing internal organs, such as the colon, and an illuminator for illuminating the field of view of the viewing element. The viewing element and the illuminator are located in the tip of the endoscope and are used to capture images of the inner wall of the body cavity scanned by the endoscope. The captured image is sent to a control unit connected to the endoscope via one of the channels present in the shaft of the scope to be displayed on a screen connected to the control unit.

[0010] During endoscopic surgery, the surgeon guides the endoscope within the patient's body using captured images displayed on a screen coupled to a control unit, such as a guide. However, the physician does not know the exact position of the endoscope within the body relative to the patient's internal organs. The physician manipulates the endoscope based largely on his / her knowledge of the patient's anatomy, experience, and displayed images of the internal organs.

[0011] Conventional guidance system endoscopes allow the surgeon to observe the position of the scope within the patient's body by displaying a representation of the scope within the body during endoscopic surgery. However, such a representation depicts the position of the scope relative to the endoscope coordinate system, not relative to the patient's coordinate system. As a result, the surgeon does not have an accurate understanding of the scope's position relative to the patient's body. This may cause the surgeon to manipulate the scope in a manner that causes discomfort or even pain to the patient.

[0012] Therefore, there is a need for an apparatus and method for displaying the exact position of an endoscope within a patient by combining the coordinates of the scope with the coordinates of the patient. There is a need for a method for combining patient and endoscope information to provide an augmented reality environment that clearly highlights the position of the endoscope relative to the patient's internal organs. Summary of the invention

[0013] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools, and methods, which are intended to be exemplary and illustrative, not limiting in scope.

[0014] The present specification discloses an endoscope system having an endoscope handle and an endoscope body suitable for being inserted into the gastrointestinal tract of a patient, the system comprising: a plurality of orientation markers located on the endoscope handle, wherein the orientation markers are distributed around the circumference of the endoscope handle; a plurality of sensors located at different positions longitudinally along the outer surface of the endoscope body, wherein each of the plurality of sensors is suitable for generating first orientation data; one or more cameras located outside the patient's body and suitable for detecting one or more of the plurality of orientation markers and generating second orientation data; and a controller suitable for receiving the first orientation data and the second orientation data and generating data indicating the position of the endoscope body in the gastrointestinal tract of the patient.

[0015] Optionally, the data indicative of the position of the endoscope body includes all positions of the endoscope body that have entered the gastrointestinal tract of the patient.

[0016] Optionally, the sensors are distributed longitudinally along the length of the endoscope body and are separated by a predetermined distance of at least 0.1 mm.

[0017] Optionally, the controller is adapted to generate an image of the endoscope body based on data indicative of the position of the endoscope body.

[0018] Optionally, the controller is further adapted to orient the image by performing a transformation of the orientation of the endoscope into a coordinate system defining the position of the patient and applying the transformation to the image.

[0019] Optionally, the controller is further adapted to superimpose the oriented image onto an image of a portion of the patient's gastrointestinal tract to generate an oriented superimposed image of the endoscope body.

[0020] Optionally, the first orientation data indicates a position of the endoscope body relative to an endoscope coordinate system.

[0021] Optionally, the second orientation data indicates a position of the endoscope handle relative to a patient coordinate system.

[0022] Optionally, the endoscope system further comprises a projector adapted to receive the oriented superimposed image of the endoscope body from the controller and project it onto the patient.

[0023] Optionally, the plurality of orientation markers comprise spheres positioned on a handle of the endoscope, each sphere having a diameter in the range of 0.5 to 2 cm.

[0024] Optionally, the plurality of directional markers comprises a pin-spot sized laser beam.

[0025] Optionally, the plurality of directional markers are made of a material that reflects or emits infrared light.

[0026] Optionally, the plurality of sensors include one or more of accelerometers, gyroscopes, magnetometers, and strips that measure bending and twisting of the insertion tube of the endoscope by one or more of electro-optical and mechanical methods.

[0027] Optionally, the plurality of sensors include one or more of inductive sensors, capacitive sensors, capacitive displacement sensors, photoelectric sensors, magnetic sensors, and infrared sensors positioned along one of the elongated axis and insertion tube of the endoscope, wherein each of the sensors corresponds to a unique identifier based on the position of the sensor along the insertion tube.

[0028] Optionally, the endoscope system also includes a distance sensor suitable for detecting distance markings located at different positions longitudinally along the outer surface of the endoscope body and generating distance data, wherein the distance sensor includes one or more of a depth sensor and a touch sensor for providing a distance that the insertion tube has traveled in the patient's gastrointestinal tract.

[0029] Optionally, the endoscopic system comprises two stereo calibrated cameras adapted to generate by triangulation the second orientation data comprising the 3D position of the reference in the camera's own coordinate system.

[0030] The present specification also discloses a method for tracking the position of an endoscope within an organ of a patient during endoscopic surgery, the method comprising: determining the position of the endoscope within the organ in the coordinate system of the endoscope; capturing multiple reference markers in an image through an external optical tracker; transforming the captured reference markers from the coordinate system of the endoscope to the coordinate system of the optical tracker; detecting the captured reference markers on a model of the patient's organ; and projecting an image of the endoscope with the reference markers onto an image of the patient with the reference markers.

[0031] Optionally, the external optical tracker is a camera placed over an endoscope used to perform the endoscopic procedure.

[0032] Optionally, the captured fiducial markers on the model of the patient's organ are detected by using an object detection algorithm. Still optionally, the captured fiducial markers on the model of the patient's organ are detected by using a Hough transform. Optionally, the method also includes projecting the position of the endoscope directly onto the patient's body by using a calibrated projector.

[0033] The present specification also discloses an endoscopic inspection system for tracking the position of an endoscope within a patient's organ during endoscopic surgery, the system comprising at least an endoscope connected to a plurality of reference markers; an optical tracker disposed outside the endoscope and a computing unit for at least processing images captured by the optical tracker, the optical tracker capturing a plurality of reference markers and the endoscope in images during endoscopic surgery, the computing unit transforming the captured reference markers from the coordinate system of the endoscope to the coordinate system of the optical tracker and projecting the image of the endoscope with the reference markers onto the image of the patient's organ with the reference markers.

[0034] Optionally, the external optical tracker is a camera placed over an endoscope used to perform the endoscopic procedure.

[0035] Optionally, the captured fiducial markers are transformed from the endoscope's coordinate system to the optical tracker's coordinate system by using an object detection algorithm. Still optionally, the captured fiducial markers are transformed from the endoscope's coordinate system to the optical tracker's coordinate system by using a Hough transform.

[0036] Optionally, the system also includes a calibrated projector for projecting the position of the endoscope directly onto the patient's body.

[0037] The above and other embodiments of the present specification will be described in more detail in the drawings and detailed description provided below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] As these and other features and advantages of the present specification will be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings, wherein:

[0039] Figure 1A A multi-viewing element endoscopy system is shown in which the methods of the present specification may be implemented;

[0040] Figure 1B A schematic diagram of a layout of a multi-viewing element endoscopy system and associated interface unit deployed in an operating room in which the methods of the present specification may be implemented;

[0041] Figure 1C is a schematic diagram of another arrangement of a multi-viewing element endoscopy system and associated interface unit deployed in an operating room in which the methods of the present specification may be implemented;

[0042] Figure 1D is a schematic diagram of yet another arrangement of a multi-viewing element endoscopy system and an associated interface unit deployed in an operating room in which the methods of the present specification may be implemented;

[0043] Figure 2 An endoscope using multiple viewing elements to measure depth, distance or position of an endoscope tip is shown, according to one embodiment of the present specification, the elongated shaft of the endoscope having multiple sensors attached thereto;

[0044] Figure 3 A flowchart illustrating a method for tracking the position of an endoscope within an internal organ of a patient during endoscopic surgery according to one embodiment of the present specification;

[0045] Figure 4A A three-dimensional (3D) model of an endoscope with fiducial markers according to one embodiment of the present specification is shown;

[0046] Figure 4B The optical tracker / camera is used to project the Figure 4A The 3D endoscope model shown in ; and

[0047] Figure 4Cshows a video image projected on top of an endoscope image captured by the same camera according to one embodiment of the present specification. Figure 4A Fiducial markers shown in . DETAILED DESCRIPTION

[0048] The present specification provides a method for displaying the position of an endoscope within a patient's body. In one embodiment, an image of an endoscope used in endoscopic surgery is projected directly onto the patient's body, which allows the surgeon to clearly determine the position of the endoscope within the body. In another embodiment, the position of the endoscope within the patient's body is displayed as an image on a monitor together with an image of the patient's internal organs, which allows the surgeon to easily manipulate the endoscope within the body.

[0049] In various embodiments, the method of the present specification allows the surgeon to accurately determine the position of the endoscope in the patient's body during endoscopic surgery, thereby significantly reducing the navigation time of the endoscope. The method also allows the surgeon to correctly determine the distance between the tip of the endoscope and the patient's cecum during GI surgery.

[0050] In one embodiment, the method of the present description allows for a three-dimensional reconstruction of a patient's colon. Images captured during endoscopic surgery can be displayed on top of a colon model, which enables better maneuverability of the endoscope and improved diagnosis of the colon.

[0051] It is noted that according to some embodiments, the term "endoscope" as described herein may particularly refer to a colonoscope, but is not limited to a colonoscope. The term "endoscope" may refer to any instrument used to examine the interior of a hollow organ or body cavity.

[0052] It should also be noted that various terms described below that appear in this specification may be used interchangeably or to refer to similar components and should in no way be interpreted as limiting:

[0053] • "Utility tube / cable" may also be referred to as "umbilical tube / cable".

[0054] The “main control unit” may also be referred to as a “controller unit” or a “main controller”.

[0055] • A "viewing element" may also be referred to as an image capture device / assembly, a viewing assembly, a camera, a television camera, or a video camera.

[0056] The “working channel” may also be referred to as the “service channel”.

[0057] • An "illuminator" may also be referred to as an "illumination source", and in some embodiments, is an LED.

[0058] • A "flexible shaft" may also be referred to as a flexure or vertebral mechanism.

[0059] • As used herein and throughout, "fiducial" may be used to refer to a standard or reference, for example, a fiducial marker.

[0060] As used in the specification, the term "optical assembly" is used to describe a set of components that allow an endoscopic device to capture light and transform the light into at least one image. In some embodiments, a lens / optical element is used to capture light, and an image capture device, such as a sensor, is used to transform the light into at least one image.

[0061] The image capture device may be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor, or other suitable device having a light sensitive surface that can be used to capture an image. In some embodiments, a sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor (for detecting reflected light received by an optical element) is employed.

[0062] In some embodiments, the optical element includes a plurality of optical devices, such as a lens assembly, a lens, and a protective glass, and is configured to receive reflected light originating from a target object.

[0063] According to one embodiment of the present specification, the top end cap can accommodate the top portion of the endoscope. The top portion with the top end cap can be rotated or manipulated by a flexible shaft, which can also be referred to as a bending portion, for example, a vertebral mechanism. The top end cap can be configured to be assembled on the interior of the top portion including an electronic circuit board assembly and a fluid channeling assembly, and provide protection for internal components in the interior, such as a body cavity. Subsequently, the endoscope is able to perform diagnostic or surgical procedures in the body cavity. The top portion carries one or more observation elements, such as a camera to observe areas within the body cavity that are the target of these operations.

[0064] The top cover may include a panel having a transparent surface, window or opening for an optical lens assembly for viewing elements. The panel and viewing elements may be located on the front and sides of the top portion. The optical lens assembly may include a plurality of static or movable lenses that provide different fields of view.

[0065] The electronic circuit board assembly can be configured to carry a viewing element that can be viewed through an opening in the panel. The viewing element can include an image sensor such as, but not limited to, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor.

[0066] The electronic circuit board assembly may be configured to carry an illuminator capable of providing illumination through an optical window of the illuminator. The illuminator may be associated with the viewing element and may be positioned to illuminate a field of view of the viewing element.

[0067] The present specification provides a method for determining the position of an endoscope within a patient's body by determining the coordinates of the endoscope within the body. In one embodiment, the present specification discloses the use of fiducial markers or points to track the position of an endoscope within a patient's body. In one embodiment, the fiducial markers are distributed around the entire handle of the endoscope and together with a sensing system provide data describing how the handle is turned or oriented in 3D space. In one embodiment, a camera system is used to capture the position of the fiducial markers, and using a known algorithm, the orientation of the endoscope is transformed relative to the camera system. In one embodiment, the position of the patient and the camera system is fixed so that the camera system can scale and transform the orientation of the endoscope into an image projected onto the patient's body.

[0068] In an embodiment, the position of the endoscope within the patient is tracked by sensors that measure the bending, rotation, or orientation of the endoscope body within the patient. In other embodiments, sensors are integrated along the insertion tube of the endoscope to provide real-time information about the distance traveled by the endoscope within the patient. In one embodiment, the orientation of the endoscope obtained from the fiducial markers and the bending, rotation, or orientation information obtained by the sensors together provide a precise orientation of the entire endoscope within the patient. In one embodiment, the position of a first position of the endoscope is determined by using fiducial markers, and the position of each point along the endoscope body is determined by using the rotation or orientation relative to the handle of the endoscope and the distance that the point has traveled within the patient.

[0069] This specification is directed to multiple embodiments. The following disclosure is provided to enable a person of ordinary skill in the art to practice this specification. The language used in this specification should not be interpreted as a general rejection of any particular embodiment or a claim that is used to limit the meaning of the terms used herein. Without departing from the spirit and scope of the specification, the general principles defined herein can be applied to other embodiments and applications. Moreover, the terms and wording used are to describe exemplary embodiments and should not be considered restrictive. Therefore, this specification should be given the broadest scope that includes multiple substitutes, modifications and equivalents consistent with the disclosed principles and characteristics. For the sake of clarity, details related to technical materials known in the technical field related to this specification are not described in detail to avoid unnecessary obscurity of this specification.

[0070] Throughout this document, it should be noted that any feature or component described in conjunction with a particular embodiment can be used and implemented with any other embodiment, unless explicitly stated otherwise.

[0071] Reference now Figure 1A, which shows a multi-viewing element endoscopy system 100. The system 100 may include a multi-viewing element endoscope 102. The multi-viewing element endoscope 102 may include a handle 104 from which an elongated shaft 106 extends. The elongated shaft 106 terminates in a top portion 108 that can be rotated by a bend 110. The handle 104 can be used to manipulate the elongated shaft 106 within a body cavity. The handle may include one or more buttons and / or knobs and / or switches 105 that control the bend 110 and functions such as fluid injection and aspiration. The handle 104 may also include at least one, and in some embodiments, one or more working channel openings 112, through which surgical tools can be inserted; and one or more side service channel openings.

[0072] A multipurpose cable 114, also referred to as an umbilical cord, may be connected between the handle 104 and the main control unit 199. The multipurpose cable 114 may include one or more fluid channels and one or more electrical channels therein. The electrical channels may include at least one data cable for receiving video data from the forward and side viewing elements, and at least one power cable for providing power to the viewing elements and discrete illuminators.

[0073] The main control unit 199 contains the controls required to display images of internal organs captured by the endoscope 102. The main control unit 199 can manage the power transmission to the tip portion 108 of the endoscope 102, such as for the viewing elements and illuminators of the tip portion. The main control unit 199 can also control one or more fluid, liquid and / or suction pumps that supply the corresponding functions to the endoscope 102. For human interaction with the main control unit 199, one or more input devices 118, such as a keyboard, touch screen, etc., can be connected to the main control unit 199. In Figure 1A In the embodiment shown in FIG. 1 , the main control unit 199 includes a screen / display 120 for displaying surgical information about an endoscopic procedure when using the endoscope 102. The screen 120 can be configured to display images and / or video streams received from the viewing elements of the multi-viewing-element endoscope 102. The screen 120 can also be operated to display a user interface for allowing a human operator to set various features of the endoscopy system.

[0074] In various embodiments, the position of the endoscope within the patient's body is determined by determining the endoscope's coordinates within the body. Various methods can be used to determine the position and pose of the endoscope within the scope's coordinate system or relative to an external coordinate system.

[0075] In one embodiment, the specification discloses the use of fiducial markers or points to track the position of an endoscope within a patient's lumen. As is known in the art, a fiducial marker or point is an object placed in the field of view of an imaging system that appears in the resulting image to serve as a reference point or measurement standard. It can be something placed in or on the imaging object, or a mark or group of marks in a reticle of an optical instrument. The position of the fiducial marker depends on the method used to calculate the posture of the scope within the patient.

[0076] In one embodiment, the fiducial markers or points are spheres (balls) placed on the handle of the endoscope that enable the position of the endoscope relative to the endoscope coordinates (internal coordinate system) to be measured. These spheres are easily identified in the image. For example, other recognizable shapes, such as but not limited to crosses, can also be used as fiducial markers. In one embodiment, the fiducial markers used in this specification are spheres with a diameter ranging from 0.5 to 2 cm. In another embodiment, a pinpoint-sized laser beam can also be used as a fiducial marker because it can be uniquely detected by the optical tracker / camera when in the field of view of the optical tracker / camera. The fiducial markers can be made of any material that can be easily detected by the optical tracker. In one embodiment, the fiducial markers can be made of materials that reflect or emit any form of light, particularly infrared light. Multiple optical trackers, such as cameras, can be used to detect the fiducial points on the handle portion of the endoscope. The fiducial points appear in the image captured by the optical tracker, which are used as reference points to associate the coordinates of the endoscope with the coordinates of the optical tracker.

[0077] In an embodiment, an optical tracker, which may be an external camera, is placed over a patient undergoing endoscopic surgery so that the camera captures the endoscope and the patient's body in the same image. Fiducial markers are placed on at least a portion of the endoscope. Thus, the external camera produces an image that displays the fiducial markers on the (partially invisible) endoscope along with the patient's body.

[0078] In one embodiment of the present specification, electromagnetic tracking technology is used to detect the position of an endoscope within a patient's body. As is known in the art, for electromagnetic tracking of an endoscope, a plurality of electromagnetic coils are wrapped around one or more portions of the endoscope. The coils emit electromagnetic signals that can be detected by an electromagnetic tracking device placed outside the endoscope. In one embodiment, a fiducial marker is attached to the electromagnetic tracking device, which is used to measure the position of the endoscope relative to the coordinates of the tracking device.

[0079] Sensing system:

[0080] In another embodiment of the specification, the position of the endoscope within the patient's body is tracked by sensors that measure the orientation of the insertion tube of the endoscope at several locations. In various embodiments, sensors such as accelerometers, gyroscopes, magnetometers (i.e., electronic compasses), and strips can be used that measure the bending and twisting of the insertion tube by electro-optical / mechanical methods. According to another embodiment, the endoscope includes sensors integrated along its insertion tube to provide real-time information about the distance traveled by the endoscope within the patient's lumen.

[0081] In one embodiment, a plurality of sensors are placed along the elongated shaft or insertion tube of the endoscope. In addition, each sensor has a unique identifier, code, signature or other identification according to its position along the insertion tube (such as the distance from the distal tip). In another embodiment, each identifier is not only unique to the sensor, but also indicates a specific position or distance occupied by the sensor. Several different types of sensors can be used, including but not limited to inductive sensors, capacitive sensors, capacitive displacement sensors, photoelectric sensors, magnetic sensors and infrared sensors. In one embodiment, a depth sensor is placed at the entrance of the body, where the endoscope is inserted and communicates with a main control unit, which is used together with the endoscope. In some embodiments, a sensor matrix is ​​used to achieve continuity in distance reading. In some embodiments, a touch sensor can be used. Therefore, for example, using touch sensors placed at regular intervals on the insertion tube, the number of touch sensors displaying the output will indicate the depth to which the insertion tube has traveled in the inner cavity of the body.

[0082] It is known in the art that the insertion tube has numbers or markings on it to indicate to the physician the distance of the insertion tube within the patient's body. Therefore, in another embodiment, an imaging device, such as a CCD, CMOS, etc., is placed outside the patient's body, near the entry point of the insertion tube of the endoscope to capture an image of the markings of the insertion tube visible outside the body, thereby providing the distance of the insertion tube within the patient's body.

[0083] In yet another embodiment, the depth is measured by using sensors that respond to the physician's grip of the tube. The sensors are placed substantially throughout the length of the insertion tube, and each sensor has a unique identifier, code, signature, or other identification based on its position along the elongated axis of the insertion tube. Methods and systems for determining the position or distance of the endoscope tip within a patient are described in co-pending U.S. Patent Application No. 14 / 505,387, filed on October 2, 2014, and entitled "Endoscope with Integrated Sensor," which is incorporated herein by reference in its entirety. Thus, as described above, various methods may be used to determine the position and attitude of an endoscope within the scope's coordinate system.

[0084] Direction determination:

[0085] The 3D coordinates of the endoscope can be reconstructed from the orientation calculated by the integration of the bending matrix. The bending matrix provides a measure of the degree of bending of the endoscope. The 3D coordinates of the endoscope can be reconstructed from the bending information provided by the bending matrix. In one embodiment, fiducial markers with known coordinates in the coordinate systems of the endoscope and the optical tracker are used to obtain matching between the two coordinate systems.

[0086] In one embodiment, the 3D mirror model is projected onto a two-dimensional (2D) plane by an optical tracker / camera. According to one embodiment of the specification, the optical tracker captures the fiducial markers relative to the coordinates of the camera and matches the fiducial markers to a model of a human organ, such as a colon, for endoscopic scanning. An object detection algorithm is used to detect the fiducial markers captured within the frame / model of the patient's organ. In one embodiment, the detected fiducial markers are represented by a circle or demarcation (in this case orange) surrounding the fiducial markers.

[0087] The transformation method used to transform the fiducial marker from the endoscope coordinates to the optical tracker coordinates depends on the characteristics of the optical tracker. If the optical tracker is composed of two stereo calibrated cameras, these cameras calculate the 3D position of the marker in its own coordinate system by triangulation. The transformation of these 3D points of the known 3D structure with the fiducial marker can be calculated by any point cloud registration algorithm that calculates the transformation that minimizes the average distance between the points, such as Horn's algorithm. However, if the optical tracker is composed of a single camera, the transformation can be calculated by any algorithm that solves the PnP problem, such as EPnP, DLT and POSSIT. As known in the art, the PnP problem aims to determine the position of the camera based on a comparison between a set of 3D points with known positions in some arbitrary coordinate system and their 2D projections through the camera.

[0088] In one embodiment, a target detection algorithm, such as the "Hough Transform", is used to detect fiducial markers captured within a frame / model of the patient's internal organs. In some embodiments, various other target detection algorithms may also be used. The Hough Transform is an algorithm commonly used to detect parametric shapes in an image. For example, it can be used to detect spheres. The algorithm first calculates the number of pixels that are consistent with any parametric combination of the shape, and secondly determines a threshold for the calculated matrix.

[0089] In other embodiments, the distance that the endoscope has traveled within the lumen of the patient can be obtained using data obtained from sensors that measure the bending and twisting of the insertion tube by electro-optical / mechanical methods, such as accelerometers, gyroscopes, magnetometers (i.e., electronic compasses), and strips. In one embodiment, the distance that the endoscope has traveled within the lumen of the patient is obtained by using data obtained from sensors placed along the elongated shaft or insertion tube of the endoscope, such as inductive sensors, capacitive sensors, capacitive displacement sensors, photoelectric sensors, magnetic sensors, depth sensors, infrared sensors, and touch sensors. The unique identifier of each of these sensors provides information about the specific position (relative to the position of the distal tip of the insertion tube) or distance occupied by the sensor, thereby providing the orientation of the scope within the patient's organ.

[0090] Image Projection

[0091] In an embodiment, the orientation of the endoscope determined using fiducial markers and sensors is converted to the camera's coordinate system, scaled based on the patient's position and size and relative camera position, and projected onto the patient.

[0092] In one embodiment, the captured fiducial markers are projected onto an image of the patient's internal organs. The 3D image of the internal organs displaying the captured fiducial markers is obtained by using computer software. In another embodiment, the captured fiducial markers corresponding to the endoscope are projected onto the image of the endoscope. The 3D image of the endoscope displaying the captured fiducial markers is obtained by using computer software. In yet another embodiment, the internal organs are displayed together with the image of the fiducial markers and the image of the endoscope with the fiducial markers so that the surgeon can clearly determine the position of the endoscope within the organ.

[0093] In one embodiment, a projector connected to the control unit of the endoscope is used to project a virtual model of the scanned patient's organ in which the position of the endoscope is shown directly onto the patient's body. In one embodiment, the projector is calibrated to express its position relative to the position of the endoscope and the patient in the coordinate system of the endoscope and the patient. The calibration also provides the internal parameters of the projector, such as the direction of the rays originating from the projector and the patient. By using the internal parameters, the accurate illumination pattern of the projector can be calculated, which in turn enables the real-time projection of a virtual model (holographic) of the endoscope's position in the patient's body on top of the patient.

[0094] Figure 1BSchematically depicts the layout of an endoscopic system and associated interface unit 132 deployed in an operating room, in which an optical tracker and fiducial markers as described above may be employed. When a physician 138 uses the endoscopic portion 140 of the endoscopic system 130 in an endoscopic procedure, a patient 134 is supported on a bed 136. The endoscope 140 is connected to a main controller 142 via a multipurpose cable 144. A plurality of fiducial markers 146 are provided on the endoscope 140. An optical tracker or external camera 148 is placed above the patient 134 so that the camera 148 captures the endoscope 140, the fiducial markers 146, and the patient's body 134 in the same image displayed on an external display unit 152. In another embodiment, more than one optical tracker or external camera 148 may be positioned above the patient 134.

[0095] Thus, an optical tracker, or in this case an external camera, is placed over the patient undergoing endoscopic surgery, such that the camera captures the endoscope and the patient's body in the same image. Fiducial markers are placed on at least a portion of the endoscope. Thus, the external camera produces an image that shows the fiducial markers on the (partially invisible) endoscope along with the patient's body. Next, an object detection algorithm, such as a "Hough transform," is used to detect the fiducial markers captured within the frame / model of the patient's internal organs (relative to the Figure 3 In various embodiments, after the parameters of the external camera are determined (performed by calibration), a virtual model of the patient's organ (e.g., colon) is provided on top of the patient's image taken by the external camera. In one embodiment, the virtual model of the patient's organ showing the endoscope position is projected directly onto the patient's body using a calibrated projector.

[0096] The endoscope 140 provides one or more endoscopic views (which may be simultaneous) using one, two, three or more cameras housed in the tip of the endoscope 140. The main controller 142 is connected to at least one display screen 154 (not shown) or multiple display screens, such as three display screens 154a, 154b and 154c, respectively, wherein each display screen is configured to display a corresponding view of the three endoscopic views provided by the endoscope system 130. The display screen 154 is positioned facing the physician 138 and can be raised so that the physician 138 can perform endoscopic surgery by looking at the screen display and having an undisturbed line of sight thereto. In one embodiment, the position and posture of the scope are determined relative to an external coordinate system. In this embodiment, the fiducial markers are attached to the external coordinate tracking system.

[0097] Figure 1C shows that it is placed in acceptance as in Figure 1BAn electromagnetic field generator with reference markers 156 is placed in close proximity to the patient 134, for example, on / under the patient's bed 136 or on a stand (not shown) placed near the patient 134, for example, during endoscopic surgery.

[0098] Figure 1D An endoscope according to one embodiment of the present specification is shown to be projected directly onto the patient's body using a projector while receiving an image such as in Figure 1B 140 and 134. The position of the patient's body for endoscopic surgery as shown in . As shown, a projector 158 connected to a controller 142 is used to project a virtual model 160 of the patient's colon in which the position of the endoscope is shown directly onto the patient's body 134. In one embodiment, the projector 158 is calibrated, whereby the calibration projector 158 expresses its position relative to the position of the endoscope 140 and the patient 134 in the coordinate system of the endoscope and the patient. The calibration also provides internal parameters of the projector 158, such as the direction of rays originating from the projector and the patient 134. By using the internal parameters, the accurate illumination pattern of the projector 158 can be calculated, which in turn enables real-time projection of a virtual model 160 (holographic) of the position of the endoscope 140 in the patient's body on top of the patient 134.

[0099] According to another embodiment, the endoscope includes sensors integrated along its insertion tube to provide real-time information about the distance traveled by the endoscope within the patient's lumen. Figure 2 As shown in , multiple sensors 2015 are placed along the elongated shaft or insertion tube 2306 of the endoscope. In addition, each sensor has a unique identifier, code, signature or other identification according to its position along the insertion tube 2306 (such as the distance from the distal tip). Therefore, by way of example but not limitation, the sensor will be placed at a distance of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 or 20 centimeters, or any increment therein, from the distal end of the tube 2306. The next sensor can be placed at a similar or different distance and will have an identifier different from the identifier programmed into the first sensor. In another embodiment, each identifier is not only unique to the sensor, but also indicates a specific position or distance occupied by the sensor. Therefore, in one embodiment, multiple sensors are placed at 10 centimeters increments along the length of the insertion tube 2306, wherein each sensor 2015 has a different identifier and wherein each identifier indicates the distance increment occupied by the sensor.

[0100] Additionally, the depth sensor is placed at the entrance of the body where the endoscope is inserted and communicates with the main control unit that is used with the endoscope. As a non-limiting example, let us consider an endoscopic procedure performed on a patient's colon 2022. The depth sensor 2020 is placed outside the body 2024, near the rectum 2026, which is the entry point for the endoscope to enter the colon 2022. During the procedure, the depth sensor 2020 detects alignment with the sensor 2016 that is closest to the entry site outside the body. In one embodiment, each sensor 2015, 2016 is preprogrammed to read according to its position so that a 10 cm sensor will transmit a different output than a 20 cm sensor. In one embodiment, the output of the depth sensor 2020 is transmitted to a controller or main control unit, which records and provides a display of the distance traveled by the distal end of the scope.

[0101] Figure 3 Flowchart showing a first method of tracking the position of an endoscope within an internal organ of a patient during endoscopic surgery according to one embodiment of the present specification. In step 302, a reference position of the endoscope within the patient is determined in a coordinate system (internal) or external to the scope.

[0102] In one embodiment, fiducial markers are placed on the handle of the endoscope in order to determine the position of the scope within the patient using the scope's coordinate system.

[0103] In another embodiment, the position of the mirror within the patient's body can be determined by an external coordinate system, such as by using a bending matrix, electromagnetic tracking, or by using one or more sensors as described above. In such an embodiment, the fiducial markers are placed on an external reference coordinate tracking system.

[0104] At step 304, fiducial markers located on the handle of the scope or an external coordinate tracking system are captured by an optical tracker. In one embodiment, the optical tracker is a camera that captures the fiducial markers relative to the coordinates of the camera. In the case where the fiducial markers are rigidly attached to the endoscope, the markers can also be described relative to the coordinates of the endoscope.

[0105] Figure 4A A three-dimensional (3D) model of a mirror with fiducial markers according to one embodiment of the present specification is shown. Figure 4B The optical tracker / camera is used to project the Figure 4A 3D mirror model shown in . As shown, the reference marker 402 is captured in the image by an optical tracker (not shown in the figure) relative to the endoscope 404. In one embodiment, the optical tracker is a camera that captures the reference marker 402 relative to the coordinates of the camera. Therefore, according to one embodiment of the present specification, Figure 4B Shown is a match to the human colon type Figure 4A The object detection algorithm is used to detect the fiducial marker 402 captured within the frame / model 406 of the patient's colon. The circle or demarcation (in orange in this case) surrounding the fiducial marker 402 is an indication of the fiducial marker 402 detected (and marked) by the fiducial detection algorithm, as shown in the figure below. Figure 3 As explained in step 306 in .

[0106] In step 306, the transformation of the fiducial marker from the coordinates of the endoscope to the coordinates of the optical tracker is obtained. The transformation method depends on the characteristics of the optical tracker. If the optical tracker consists of two stereo calibrated cameras, these cameras calculate the 3D position of the marker in its own coordinate system by triangulation. However, if the optical tracker consists of a single camera, the transformation can be calculated by any algorithm that solves the PnP problem, such as EPnP, DLT and POSSIT.

[0107] At step 308, the captured fiducial markers are projected onto an image of the patient's internal organs. A 3D image of the internal organs showing the captured fiducial markers may be obtained using computer software. At step 310, the captured fiducial markers corresponding to the endoscope are projected onto an image of the endoscope. A 3D image of the endoscope showing the captured fiducial markers may be obtained using computer software, such as in Figure 4C as shown in . Figure 4C is shown projected on top of an image captured by the same optical tracker / camera according to one embodiment of the present specification. Figure 4A As shown in the figure, Figure 4A The fiducial marker 402 shown in FIG. 4 is projected onto the image of the endoscope 404. In one embodiment, the accuracy of the position of the camera (not shown in the figure) can be estimated by calculating the error of the re-projection of the camera.

[0108] At step 312, the internal organ is displayed along with the image of the fiducial markers and the image of the endoscope with the fiducial markers to enable the surgeon to clearly determine the position of the endoscope within the organ. In one embodiment, after the parameters of the external camera are determined (performed by calibration), a virtual model of the patient's colon showing the position of the endoscope is augmented to the image of the patient taken by the external camera.

[0109] In various embodiments, the patient image can be enhanced with a virtual model of the patient's colon showing the position of the endoscope by using a display of a computer monitor. In other embodiments, a display method such as transparent glass can be used. In another embodiment, the enhanced image can also be displayed on the retina of the observer.

[0110] The above examples are merely illustrative of many applications of the present specification system. Although only a few embodiments of the present specification are described herein, it should be understood that the present specification may be embodied in many other specific forms without departing from the spirit or scope of the present specification. Therefore, the present examples and embodiments should be considered illustrative rather than restrictive, and the present specification may be modified within the scope of the appended claims.

Claims

1. An endoscope system, comprising an endoscope, wherein the endoscope comprises an endoscope handle and an endoscope body, and the endoscope system comprises: a plurality of orientation markers located on the endoscope; one or more cameras located outside the patient's body and adapted to detect one or more of the plurality of orientation markers and generate first orientation data; a controller adapted to receive the first orientation data and generate a virtual model of the patient's anatomy showing the position of the endoscope; as well as A projector adapted to receive the first orientation data and the virtual model from the controller and to project the virtual model onto the patient, wherein the projector is calibrated to express data of its position relative to the endoscope and the patient in the coordinate system of the endoscope and the patient, and to achieve real-time projection of the virtual model based on internal parameters of the projector, wherein the calibration of the projector provides the internal parameters of the projector. 2 . The endoscope system according to claim 1 , wherein the first orientation data includes all positions of the endoscope body that have entered the patient.

3. The endoscopic system according to claim 1 further includes at least one orientation marker located on a patient's bed, wherein the controller is adapted to generate an image of the patient located on the patient's bed.

4. An endoscopic system according to claim 1, wherein the controller is suitable for generating an image of the endoscope body based on data indicating the position of the endoscope body, and wherein the controller is further suitable for orienting the image by performing a transformation of the orientation of the endoscope to a coordinate system defining the position of the patient and applying the transformation to the image, wherein the controller is further suitable for superimposing the oriented image onto an image of a portion of the patient's anatomical structure to generate an oriented superimposed image of the endoscope body, wherein the endoscopic system further includes at least one screen, which is suitable for receiving the oriented superimposed image of the endoscope body from the controller.

5. The endoscope system of claim 1, wherein the first orientation data indicates a position of the endoscope body relative to the coordinate system.

6. The endoscope system according to claim 1 further includes a plurality of sensors located at different positions along the longitudinal direction of the endoscope body, wherein each of the plurality of sensors is suitable for generating second orientation data; wherein the second orientation data indicates the position of the endoscope handle.

7. The endoscopic system of claim 1, wherein the plurality of orientation markers comprise spheres disposed on the handle of the endoscope, each sphere having a diameter ranging from 0.5 to 2 cm.

8. The endoscopic system of claim 1, wherein the plurality of orientation markers comprise pinpoint-sized laser beams.

9. The endoscope system according to claim 1, wherein the plurality of orientation markers are made of a material that reflects or emits infrared light.

10. The endoscope system of claim 1, further comprising a plurality of sensors including one or more of accelerometers, gyroscopes, magnetometers, and strips that measure bending and twisting of the insertion tube of the endoscope by one or more of electro-optical and mechanical methods.

11. The endoscope system according to claim 1 further comprises a plurality of sensors, which include one or more of an inductive sensor, a capacitive sensor, a capacitive displacement sensor, a photoelectric sensor, a magnetic sensor, and an infrared sensor, wherein each of the plurality of sensors corresponds to a unique identifier based on the sensor.

12. The endoscope system according to claim 10 further comprises a distance sensor adapted to detect distance marks located at different positions longitudinally along the endoscope body and generate distance data, wherein the distance sensor comprises one or more of a depth sensor and a touch sensor for providing a distance that the insertion tube has traveled in the gastrointestinal tract of the patient.

13. An endoscopic system according to claim 1, wherein the one or more cameras include two stereo calibrated cameras, which are adapted to generate at least a portion of the first orientation data including a 3D position of a reference in the camera's own coordinate system by triangulation.

14. The endoscope system of claim 1, further comprising a plurality of touch sensors configured to measure an insertion depth of the endoscope body in response to a user's gripping of the endoscope body.

15. The endoscope system of claim 1, further comprising a plurality of electromagnetic coils wrapped around one or more portions of the endoscope, wherein the plurality of electromagnetic coils are configured to emit electromagnetic signals, wherein the controller is configured to receive data from the electromagnetic coils and determine a position of the endoscope.