Systems and methods for processing colon images and videos

Abstract

A method is provided for generating a plurality of instructions for presenting a graphical user interface (GUI), the plurality of instructions for dynamically tracking at least one polyp in a plurality of endoscopic images of a colon of a patient, the method comprising the steps of iterating through the plurality of endoscopic images: tracking a position of a region that depicts at least one polyp within the corresponding endoscopic image relative to at least one previous endoscopic image; when the position of the region is outside the corresponding endoscopic image: calculating a vector from the position of the region within the corresponding endoscopic image to the position outside the corresponding endoscopic image; creating an enhanced endoscopic image by enhancing the corresponding endoscopic image using an indication of the vector; and generating a plurality of instructions for presenting the enhanced endoscopic image within the GUI.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 16 / 430,461, filed on June 4, 2019, the contents of which are incorporated herein by reference in their entirety.

[0003] Technical field and background technology of the present invention

[0004] The present invention, in some embodiments thereof, relates to colonoscopy, and more particularly, but not exclusively, to systems and methods for processing colon images and videos and / or processing colon polyps automatically detected during a colonoscopy procedure.

[0005] Colonoscopy is the gold standard for detecting colon polyps. During a colonoscopy, a long, flexible tube called a colonoscope is advanced within the colon. A camera at the end of the colonoscope captures images and presents them to the doctor on a monitor. The doctor examines the surface inside the colon for the presence of polyps. Identified polyps are removed using the colonoscope's instruments. Early removal of cancerous polyps can eliminate or reduce the risk of developing colon cancer. Summary of the Invention

[0006] According to a first aspect, a method of generating a plurality of instructions for presenting a graphical user interface (GUI), the plurality of instructions for dynamically tracking at least one polyp in a plurality of endoscopic images of a colon of a patient, the method comprising the steps of: iterating over the plurality of endoscopic images: tracking a position of a region that depicts at least one polyp within the corresponding endoscopic image relative to at least one previous endoscopic image; when the position of the region is outside the corresponding endoscopic image: calculating a vector from the position of the region within the corresponding endoscopic image to the outside of the corresponding endoscopic image; creating an enhanced endoscopic image by enhancing the corresponding endoscopic image using an indication of the vector; and generating a plurality of instructions for presenting the enhanced endoscopic image within the GUI.

[0007] According to a second aspect, a method for generating a plurality of instructions for presenting a GUI, wherein the plurality of instructions are for dynamically tracking the 3D motion of an endoscopic camera that captures a plurality of 2D endoscopic images within a colon of a patient, the method comprising the following steps: iterating over the corresponding endoscopic image of the plurality of endoscopic images; inputting the corresponding 2D endoscopic image into a 3D reconstruction neural network; outputting a 3D reconstruction of the corresponding 2D endoscopic image through the 3D reconstruction neural network, wherein a plurality of pixels of the 2D endoscopic image are assigned a plurality of 3D coordinates; calculating a current 3D position of the endoscopic camera within the colon based on the 3D reconstruction; and generating a plurality of instructions for presenting the current 3D position of the endoscopic camera on a colonogram within the GUI.

[0008] According to a third aspect, a method for calculating a three-dimensional volume of a polyp based on at least one two-dimensional (2D) image is characterized in that the method further includes the following steps: receiving at least one 2D image of an inner surface of the colon captured by an endoscopic camera located in a lumen of the colon; receiving an indication of an area of ​​the at least one 2D image, wherein the two-dimensional image depicts at least one polyp; inputting the at least one 2D image into a 3D reconstruction neural network; outputting a 3D reconstruction of the at least one 2D image through the 3D reconstruction neural network, wherein a plurality of pixels of the 2D image are assigned a plurality of 3D coordinates; and calculating an estimated 3D volume of the at least one polyp within the area of ​​the at least one 2D image based on an analysis of the plurality of 3D coordinates of the plurality of pixels of the area of ​​the at least one 2D image.

[0009] In a further embodiment of the first aspect, the indication of the vector depicts a direction and / or orientation for adjusting an endoscopic camera to capture at least one further endoscopic image depicting the area of ​​the at least one image.

[0010] In a further embodiment of the first aspect, when the location of the area delineating at least one polyp appears in the corresponding endoscopic image, the enhanced image is created by enhancing the corresponding endoscopic image using the location of the area, wherein the indication of the vector is excluded from the enhanced endoscopic image.

[0011] In a further embodiment of the first aspect, the steps of: calculating a position of the region, the position of the region depicting at least one polyp within the colon of the patient; creating a colonogram by plotting a schematic diagram representing the colon of the patient, the schematic diagram representing an indication of the position of the region depicting at least one polyp; and generating instructions for presenting the colonogram within the GUI, wherein the colonogram is dynamically updated with the positions of newly detected polyps.

[0012] In a further embodiment of the first aspect, the steps are further included: translating and / or rotating at least one endoscopic image of a continuous subset of the multiple endoscopic images including the corresponding endoscopic image to create a processed continuous subset of the multiple endoscopic images, wherein the region depicting the at least one polyp is in the same approximate position in all of the images in the continuous subset of the multiple endoscopic images; inputting the processed continuous subset of the multiple endoscopic images into a detection neural network; outputting a current region through the detection neural network, wherein the current region depicts the at least one polyp of the corresponding endoscopic image; creating an enhanced image of the corresponding endoscopic image by enhancing the corresponding endoscopic image using the current region; and generating a plurality of instructions for presenting the enhanced image in the GUI.

[0013] In a further embodiment of the first aspect, when a previous endoscopic image sequentially prior to the corresponding endoscopic image provides the output of the neural network and the tracked position delineating the area of ​​at least one polyp within the corresponding endoscopic image is at a different location than the area output by the neural network for the previous endoscopic image, the enhanced image is created for the corresponding endoscopic image based on the tracked position.

[0014] In a further embodiment of the second aspect, the method further comprises the steps of: tracking a plurality of 3D positions of the endoscopic camera; and plotting the plurality of tracked 3D positions of the endoscopic camera in the colonogram GUI.

[0015] In a further embodiment of the second aspect, a marker is used to mark several forward-direction tracking 3D positions of the endoscopic camera on the colonogram presented in the GUI, the marker indicating a forward direction of the endoscopic camera entering deeper into the colon, and another marker is used to mark several reverse-direction tracking 3D positions of the endoscopic camera presented on the colonogram, the another marker indicating a reverse direction of removing the endoscopic camera from the colon.

[0016] In a further embodiment of the second aspect, the steps are further included: inputting the corresponding endoscopic image into a detection neural network; outputting an indication of an area of ​​the endoscopic image through the detection neural network, wherein the indication depicts at least one polyp; and calculating an estimated 3D position of the at least one polyp within the area of ​​the endoscopic image based on the 3D reconstruction, and generating a plurality of instructions for presenting the 3D position of the at least one polyp on a colonogram within the GUI.

[0017] In another embodiment of the second aspect, the steps of: receiving an indication for surgical removal of the at least one polyp of the colon; and marking the 3D location of the at least one polyp on the colonogram using the indication for removal of the at least one polyp are further included.

[0018] In another embodiment of the second aspect, the steps of tracking a 3D position of an endoscopic camera, wherein the endoscopic camera captures the plurality of endoscopic images, calculating an estimated distance from a current 3D position of the endoscopic camera to a 3D position of at least one polyp previously identified using earlier acquired endoscopic images, and generating instructions for presenting an indication within the GUI when the estimated distance is below a threshold, are further included.

[0019] In a further embodiment of the second aspect, the steps of analyzing the corresponding 3D reconstruction to estimate a portion of an inner surface of the colon depicted in the corresponding endoscopic image, tracking cumulative portions of the inner surface of the colon depicted in consecutive endoscopic images during a spiral scanning motion of the endoscopic camera during a colonoscopy procedure, and generating instructions for presenting in the GUI at least one of: an estimate of remaining portions of the inner surface not yet depicted in any previously captured endoscopic images, and an estimate of full coverage of the inner surface area, wherein the analyzing step, the tracking step, and the generating step are iterated during the spiral scanning motion.

[0020] In a further embodiment of the second aspect, each portion corresponds to a time window having an interval corresponding to an amount of time that the corresponding portion is covered during the spiral scanning motion, wherein an indication of sufficient coverage is generated when at least one image depicting a majority of the corresponding portion is captured during the time window, and / or another indication of insufficient coverage is generated when several images depicting a majority of the corresponding portion are not captured during the time window.

[0021] In a further embodiment of the second aspect, an indication is calculated by aggregating the several portions covered during the spiral scanning motion relative to the several portions not covered during the spiral scanning motion, the indication being an amount of the inner surface depicted in the several images relative to an amount of the inner surface not depicted.

[0022] In a further embodiment of the second aspect, the 3D reconstruction neural network is trained by a training dataset of a plurality of 2D endoscopic image pairs defining a plurality of input images corresponding to a plurality of 3D coordinate values ​​calculated for a plurality of pixels of the 2D endoscopic images calculated by a 3D reconstruction process defining ground truth.

[0023] In another embodiment of the second aspect, the steps of: receiving an indication of at least one anatomical landmark of the colon, wherein the at least one anatomical landmark divides the colon into a plurality of portions; tracking a plurality of 3D positions of the endoscopic camera relative to the at least one anatomical landmark; calculating an amount of time spent by the endoscopic camera in each of the plurality of portions of the colon; and generating a plurality of instructions for presenting the amount of time spent by the endoscopic camera in each of the plurality of portions of the colon in the GUI.

[0024] In a further embodiment of the third aspect, the indication of the region of the at least one 2D image delineating the at least one polyp is output by a detection neural network trained to segment polyps in a plurality of 2D images.

[0025] In another embodiment of the third aspect, the 3D reconstruction of the at least one 2D image is input to the detection neural network together with the at least one 2D image to output the indication delineating the region of the at least one polyp.

[0026] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. Although several methods and materials similar or equivalent to those described herein can be used in the practice or testing of several embodiments of the present invention, exemplary methods and / or materials are described below. In the event of a conflict, the patent specification (including definitions) shall prevail. In addition, the several materials, several methods and several examples described are illustrative only and are not intended to be necessarily limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Some embodiments of the present invention are described herein by way of example only and with reference to the accompanying drawings. With specific and detailed reference now to the accompanying drawings, it is emphasized that the details shown are by way of example and for purposes of illustrative discussion of several embodiments of the present invention. In this regard, the description in conjunction with the accompanying drawings will make clear to those skilled in the art how the several embodiments of the present invention may be practiced.

[0028] In the several figures:

[0029] Figure 1 is a flow chart of a method for processing images acquired by a camera on an endoscope positioned within a colon of a target patient according to some embodiments of the present invention;

[0030] Figure 2 is a block diagram of components of a system for processing images acquired by a camera on an endoscope positioned within a colon of a target patient according to some embodiments of the present invention;

[0031] Figure 3 is a flow chart of a process for tracking a position of a region delineating one or more polyps within a corresponding endoscopic image relative to one or more previous endoscopic images according to some embodiments of the present invention;

[0032] Figure 4 is a schematic diagram illustrating an example of feature-based KD-tree matching between two consecutive images according to some embodiments of the present invention;

[0033] Figure 5 According to some embodiments of the present invention Figure 4 A geometric transformation matrix between several frames;

[0034] Figure 6 is a schematic diagram depicting an ROI representing a bounding box of a detected polyp being tracked, and a sequentially later schematic diagram in which the tracked bounding box is no longer delineated within the image, with the addition of a presentation of an arrow indicating a direction in which the camera is moved to re-delineate the ROI of the polyp in the captured image, according to some embodiments of the present invention;

[0035] Figure 7 is a schematic diagram depicting a detected polyp and a later frame depicting the tracked polyp calculated using the transformation matrix described herein, according to some embodiments of the present invention;

[0036] Figure 8is a flow chart of a process for 3D reconstruction of 2D images captured by a camera within the colon of the patient, according to some embodiments of the present invention;

[0037] Figure 9 is a flow chart depicting an exemplary 3D tracking process for tracking 3D movement of the camera according to some embodiments of the present invention;

[0038] Figure 10 is an example of a 3D rigid body transformation matrix for tracking 3D movement of a colonoscopy camera according to some embodiments of the present invention;

[0039] Figure 11 is a schematic diagram of an assembled and / or panoramic image of the colon constructed for each pixel in the plurality of combined 2D images, wherein the 3D position in the single 3D coordinate system is calculated in the plurality of combined 2D images, according to some embodiments of the present invention;

[0040] Figure 12 is a schematic diagram depicting a volumetric image presented within a corresponding quarter of the inner surface of the colon depicted therein according to some embodiments of the present invention;

[0041] Figure 13 is a flow chart of a method for calculating the quarter block depicted by a frame according to some embodiments of the present invention;

[0042] Figure 14 is a schematic diagram depicting the process for calculating a volume of a polyp from a 3D reconstructed surface calculated from a 2D image according to some embodiments of the present invention;

[0043] Figure 15 is a flow chart of an exemplary process for calculating the volume of a polyp from a 2D image according to some embodiments of the present invention;

[0044] Figure 16 is a schematic diagram depicting a series of enhanced images for tracking an ROI of a polyp presented in the GUI according to the plurality of generated instructions according to some embodiments of the present invention;

[0045] Figure 17 is a schematic diagram of a colonogram showing movement trajectories of the endoscope, locations of detected polyps, removed polyps, and anatomical landmarks presented in the GUI according to some embodiments of the present invention; and

[0046] Figure 18is a schematic diagram according to some embodiments of the present invention, which depicts several quadrants of the inner surface of the colon depicted in one or more images and / or several quadrants of the inner surface of the colon that have not yet been depicted in several images. DETAILED DESCRIPTION

[0047] The present invention, in some embodiments thereof, relates to colonoscopy, and more particularly, but not exclusively, to systems and methods for processing colon images and videos and / or processing colon polyps automatically detected during a colonoscopy procedure.

[0048] As used herein, the terms "image" and "frame" are sometimes used interchangeably. The images captured by a camera of the colonoscope may be individual frames of a video captured by the camera.

[0049] As used herein, the terms "endoscope" and "colonoscope" are sometimes used interchangeably.

[0050] An aspect of some embodiments of the present invention relates to systems, methods, apparatus, and / or code instructions (i.e., stored on a memory and executable by one or more hardware processors to generate instructions for presenting a graphical user interface (GUI)) for dynamically tracking one or more polyps in two-dimensional (2D), optionally colored, endoscopic images of a patient's colon captured by a camera of an endoscope positioned within a lumen of the colon (e.g., during a colonoscopy procedure). A position of a region (e.g., a region of interest (ROI) within a previous endoscopic image) descriptive of the one or more polyps is tracked relative to one or more previous endoscopic images, optionally based on matching visual features between the current and previous images, such as based on speed robust features. Several features are extracted by processing the ROI using SURF (Super Surfactant Feature Recognition) features. When the position of the ROI depicting the polyp(s) is determined, a vector is calculated to point to the current image (which will be outside the several boundaries of the image). The vector points from a position within the current image to the position of the ROI outside the current image. The position within the current image can be, for example, the position of the ROI in several earlier images on the screen when the ROI was within the image. An enhanced endoscopic image is created by enhancing the current endoscopic image with an indication of the vector, for example, by injecting the indication of the vector into the endoscopic image as a GUI element and / or as an overlay. Several instructions are generated for presenting the enhanced endoscopic image on a display within the GUI.

[0051] The indication of the vector depicts a direction and / or orientation for adjusting the endoscopic camera to capture another endoscopic image depicting the ROI of the polyp(s). The enhanced endoscopic image is enhanced for display within the GUI using the indication of the vector.

[0052] The indication of the vector may be an arrow pointing to the location of the ROI outside the image. Moving the camera in the direction of the arrow may restore the polyp in the plurality of images.

[0053] The process is iterated over the plurality of captured images, assisting the operator in maintaining the polyp within the images. When the camera is moved and the polyp no longer appears in the current image, the indication of the vector instructs the operator on how to manipulate the camera to recapture the polyp within the plurality of images.

[0054] One aspect of some embodiments of the present invention relates to systems, methods, devices, and / or code instructions (i.e., stored in a memory and executable by one or more hardware processors) for generating instructions for capturing endoscopic images of a patient's colon using a camera that dynamically tracks the 3D movement of an endoscope. The captured 2D images (e.g., every image or every few images, such as every third or fourth image, or another number of images) are input to a 3D reconstruction neural network, optionally a convolutional neural network (CNN). The 3D reconstruction neural network outputs a 3D reconstruction of the corresponding 2D endoscopic image. Pixels of the 2D endoscopic image are assigned 3D coordinates. A current 3D position of the endoscopic camera (i.e., endoscope) within the colon is calculated based on the 3D reconstruction. For example, the 3D position of the endoscope is determined based on the 3D coordinate values ​​of the current image. Instructions are generated for displaying the current 3D position of the endoscopic camera on a colonogram within the GUI. The colonogram depicts a virtual map of the patient's colon.

[0055] The 3D positions of the endoscope may be tracked and plotted as a trajectory on the colonogram, for example, to track the path of the endoscope within the colon during the colonoscopy procedure.

[0056] The forward and backward directions of the endoscope may be marked, for example, by arrows and / or color codes.

[0057] The 3D locations of the detected polyps can be marked on the colonogram. The 3D locations of the detected polyps can be tracked relative to the 3D location of the camera. When the distance between the camera and the polyp falls below a threshold, instructions can be generated for presenting an indication within the GUI. The indication can be, for example, a mark for the polyp when the polyp is present in the image, an arrow pointing to the location of the polyp when the polyp is not depicted in the image, and / or information that the camera is in proximity to the polyp, optionally in the form of instructions on how to move the camera to capture images depicting the polyp.

[0058] Several polyps surgically removed from the colon may be marked on the colonogram.

[0059] Optionally, the portion of the inner surface of the colon depicted in the plurality of images is analyzed. For example, based on a virtual division of the inner surface into four equal parts. As the colonoscope is used to visually scan the inner wall of the colon, the coverage of the portion is cumulatively tracked, optionally continuously, for example as the colonoscope is pulled out of the colon (or moved forward in the colon) in a spiral movement. For example, the spiral movement is performed by the clockwise (or counterclockwise) direction of the camera as the colonoscope is slowly pulled out (or pushed in). Alternatively, the interior of the colon is imaged in steps, for example by pulling the colonoscope back (or forward) a certain distance, stopping the camera's movement in the opposite direction (or forward), and imaging the circumference by orienting the camera in a circle pattern (or cross pattern), wherein the step of pulling back (or pushing forward), the step of stopping, and the step of imaging are iterated over the length of the colon. Optionally, each portion (e.g., quarter) is primarily depicted by one or more images as the colonoscope is used to visually scan the inner wall of the colon. An estimate of the depicted inner surface and / or the remaining inner surface (e.g., quarter) can be generated and instructions generated for presentation within the GUI. The estimation can be performed in real time, for example, per quarter and / or based on aggregation of coverage of various portions (e.g., quarters) as a global estimate of the entire (or majority) colon. The prior coverage and / or remaining coverage of the inner surface of the colon helps the operator ensure that the entire inner surface of the colon has been captured in the image, thereby reducing the risk of missing polyps.

[0060] Optionally, an amount of time the colonoscope spends in one or more defined portions of the colon is calculated based on the 3D tracking of the colonoscope. Instructions can be generated for presenting the time to be presented within the GUI, such as presenting the amount of time spent in each portion of the colon on the corresponding portion of the colonogram.

[0061] An aspect of some embodiments of the present invention relates to systems, methods, apparatus, and / or code instructions (i.e., stored on a memory and executed by one or more hardware processors) for generating instructions for calculating a dimension (e.g., size) of a polyp. The dimension can be a 2D dimension (e.g., area and / or radius of a flat polyp) and / or a 3D dimension (e.g., volume and / or radius of a protruding polyp). An indication of a region of the 2D image describing the polyp(s) is received, e.g., manually delineated by an operator (e.g., using a GUI) and / or output by a detection neural network that is input with the 2D image(s) and trained to segment polyps in the 2D images. The 2D images are input to a 3D reconstruction neural network, which outputs 3D coordinates of the pixels of the 2D image. The dimensions of the polyp are calculated based on an analysis of the 3D coordinates of pixels of the ROI of the 2D image depicting the polyp.

[0062] Optionally, when the size of the polyp is above a threshold, instructions are generated for presenting an alert within the GUI. The threshold may define the minimum size of polyps that should be removed. Polyps below the threshold may be left in place.

[0063] At least some embodiments of the systems, methods, devices and / or code instructions described herein relate to the medical problem of treating a patient, particularly for identifying and removing polyps in a patient's colon. Using standard colonoscopy procedures, adenomas may be missed in up to 20% of cases and cancers may be missed in approximately 0.6% of cases, as evidenced by the eventual detection of these missed lesions by interval colonoscopy. The adenoma detection rate (ADR) is variable and depends on the patient's risk factors, physician performance, and instrument limitations. The patient's individual anatomy and the quality of bowel preparation are important determinants of colonoscopy quality. The physician's ability to perform a high-quality colonoscopy depends on factors such as successful cecal intubation, careful examination during extended downtime, and overall endoscopic experience. Endoscopist fatigue and inattention are risk factors for the physician missing polyps, and starting the procedure earlier in a segment is associated with better outcomes. Notably, several sites that underwent a quality improvement program experienced an increase in ADRs, and monitoring awareness or simply observed awareness had a positive impact on ADRs.

[0064] At least some embodiments of the systems, methods, apparatus, and / or code instructions described herein improve the detection and / or removal rate of polyps during colonoscopy procedures. Improvements are facilitated at least in part by the GUI described herein that helps guide an operator to: (i) indicate which direction to steer the colonoscope camera to recapture image(s) of the polyp(s) previously identified as having disappeared from the currently captured image, helping to ensure that the polyp is not missed or mistaken for another polyp; (ii) present and update a colonogram showing the 2D and / or 3D locations of the identified polyps to help ensure that all identified polyps have been assessed and / or removed; (iii) trace portions of the circumference of the inner surface of the colon to identify portions of the inner surface that were not captured by the image (and therefore not analyzed) to identify polyps to help ensure that portions of the colon are not imaged and that polyps are missed; (iv) calculate the volume of the polyps, which may help determine which polyps to resect and / or provide data to aid in diagnosing cancer, and / or (v) calculate the time the colonoscope spends in each portion of the colon. The GUI may present and adapt in real time to the plurality of images captured during the colonoscopy process, providing real-time feedback to help guide the physician operator to improve the polyp detection and / or removal rate.

[0065] At least some embodiments of the systems, methods, devices and / or code instructions described herein address the technical problem of improving the detection and / or removal rate of polyps. In particular, at least some embodiments of the systems, methods, devices and / or code instructions described herein improve image processing technology and / or GUI technology used by an operator to help improve polyp identification and / or detection rates by analyzing the code and / or GUI of the captured images. For example, compared to standard methods. For example, optics that achieve a wider field of view and improved image resolution, and distal colonoscope attachments (such as balloon caps or rings) to improve visualization behind mucosal folds. Such optics and attachment devices are passive and rely on the operator's skill to track the identified polyps. In contrast, the GUI described herein automatically tracks the identified polyps.

[0066] At least some implementations of the systems, methods, devices, and / or code instructions described herein address the technical problem of calculating the volume of a polyp. According to standard practice, the size of the polyp is measured only after the polyp is removed from the patient, for example, see Kume, Keiichiro et al., "Endoscopic measurement of polyp size using a novel calibrated hood," Gastroenterology research and practice 2014 (2014). The importance of measuring the size of the polyp is described, for example, in Summers, Ronald M. "Polyp size measurement at CT colonography: What do we know and what do we need to know?" Radiology 255.3 (2010): 707-720. In contrast, at least some of the systems, methods, devices, and / or code instructions described herein calculate the size of the polyp in vivo before the polyp has been removed, while the polyp is attached to the colon wall. Calculating the volume of the polyp prior to resection may provide advantages, for example, polyps above a threshold volume may be targeted for resection and / or polyps below the threshold volume may remain in the patient. The volume of the polyp calculated before resection may be compared to the volume after resection, for example, to determine whether the entire polyp has been resected, and / or to compare the volume of the polyp above the surface to the unseen portion of the polyp below the surface as a cancer risk, and / or to help stratify the polyp and / or cancer risk.

[0067] At least some embodiments of the systems, methods, apparatus, and / or code instructions described herein address the technical problem of neural network processing that is slower than the rate at which images are captured in a video by a colonoscope's camera. The process described herein for tracking polyps (and / or associated ROIs) based on extracted features compensates for delays in the process of detecting polyps using a neural network, which outputs, for the image data, an indication of the detected polyps and / or their locations. The neural network-based detection process is computationally more expensive than the feature extraction and tracking process (e.g., 25 milliseconds (ms) to 40 ms on a typical personal computer (PC) with an i7 Intel processor and an Nvidia GTK 1080TI GPU, while the typical time difference between consecutive frames is in the 20 ms to 40 ms range). Consequently, a delayed scenario can be created in that the detection result output by the neural network for frame number denoted as i is only ready when a subsequent frame (e.g., frame number denoted as i+2 or subsequent) has occurred. This delayed scenario can lead to strange situations when the initial frame depicts the polyp but the subsequent frame does not (e.g., the camera moves such that the polyp is not captured in the image), and the delay in the neural network can detect the polyp only when the polyp is no longer depicted, resulting in a situation where an indication of a detected polyp is provided when the rendered image does not contain the polyp. It should be noted that when frame number i+2 is available for presentation, the frame must be available (e.g., in real time and / or immediately) for presentation because a delay in frame presentation is unacceptable from a clinical and / or regulatory perspective, for example, as it could result in injury when attempting to remove the imaged polyp. Feature-based tracking, which is computationally efficient and results in fast processing (e.g., less than 10 milliseconds on a typical PC with an i7 Intel processor) compared to neural network-based processing, is used to transform the position of the ROI depicting the detected polyp in frame i to the position in frame i+2. The transformed position is the position displayed on the display for frame number i+2. Optionally, the bounding box of the ROI (e.g., only the bounding box of the ROI) is transformed to the i+2 frame, because the contour transformation may be more inaccurate due to the influence of several different 3D positions (without considering the two-dimensional transformation). It should be noted that the i+2 frame is an example and is not necessarily limiting, as other examples can be used, such as i+1, i+3, i+4, i+5, and higher numbers.

[0068] Before explaining at least one embodiment of the present invention in detail, it should be understood that the present invention is not necessarily limited in its application to the details of the construction and arrangement of the components and / or methods described in the following description or in the drawings and / or examples. The present invention is capable of other embodiments or of being practiced or implemented in various ways.

[0069] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0070] The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (e.g., punched cards or raised structures with instructions recorded in a groove), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through fiber optic cable), or electrical signals transmitted through a wire.

[0071] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a plurality of corresponding computing / processing devices or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the corresponding computing / processing device for storage.

[0072] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional processing programming languages ​​such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user computer, partially on the user computer, as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can be customized to perform aspects of the present invention by executing a plurality of computer-readable program instructions using state information of the plurality of computer-readable program instructions.

[0073] Several aspects of the present invention are described herein with reference to several flowcharts and / or several block diagrams of several methods, apparatuses (several systems), and several computer program products according to embodiments of the present invention. It should be understood that each block of the several flowcharts and / or several block diagrams, as well as several combinations of the several blocks in the several flowcharts and / or several block diagrams, can be implemented by several computer-readable program instructions.

[0074] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the plurality of instructions (executed by the processor of the computer or other programmable data processing device) creates means for implementing the functions / actions specified in the flowchart and / or block(s). These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing device, and / or other device to operate in a particular manner, such that the computer-readable storage medium having the plurality of instructions stored therein comprises an article of manufacture, the article of manufacture comprising the plurality of instructions that implement aspects of the functions / actions specified in the flowchart and / or block(s).

[0075] The plurality of computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause a series of operating steps to be executed on the computer, other programmable device, or other device to produce a computer-implemented process, such as instructions executed on the computer, other programmable device, or other device to implement the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0076] The flowcharts and block diagrams in the figures illustrate the architectures, functions, and operations of various possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, a segment, or a portion of instructions that includes one or more executable instructions for implementing the specific logical function(s). In some alternative embodiments, the functions noted in the blocks may not occur in the order noted in the figures. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the opposite order. It is also noted that each block in the block diagrams and / or flowchart illustrations, as well as combinations of the blocks in the block diagrams and / or flowchart illustrations, may be implemented by systems that perform the specific functions or actions or by dedicated hardware-based systems that perform the specific functions or actions or by combinations of dedicated hardware and computer instructions.

[0077] Now please refer to Figure 1 , Figure 1 A flow chart of a method for processing images acquired by a camera on an endoscope within a colon of a target patient according to some embodiments of the present invention for tracking polyp(s), tracking the camera movement, mapping the location of polyp(s), calculating coverage of the inner surface of the colon, tracking the amount of time spent in different parts of the colon, and / or for calculating the volume of the polyp(s). Figure 2 , Figure 2 is a block diagram of components of a system 200 for processing images acquired by a camera on an endoscope positioned within a colon of a target patient, tracking polyp(s), tracking movement of the camera, mapping locations of polyp(s), calculating coverage of the inner surface of the colon, tracking the amount of time spent in different locations of the colon, and / or calculating the volume of the polyp(s), according to some embodiments of the present invention. The system 200 may be implemented with reference to Figure 1 The actions of the described method are optionally performed by a hardware processor 202 of a computerized device 204 executing a plurality of code instructions stored in a memory 206 .

[0078] An imaging probe 212 (e.g., a camera located on a colonoscope) captures images of a patient's colon, such as those obtained during a colonoscopy procedure. The colon images can be 2D images or color images. The colon images can be obtained as a stream of video and / or a sequence of still images. The captured images can be processed in real time and / or offline (e.g., after the procedure is complete).

[0079] The captured images may be stored in an image repository 214 , optionally implemented as an image server, such as a picture archiving and communication system (PACS) server, and / or an electronic health record (EHR) server. The image repository may be in communication with a network 210 .

[0080] A computerized device 204 receives the captured images, e.g., directly in real time from the imaging probe 212 and / or from the image repository 214 (e.g., in real time or offline). The real-time images may be received during the colonoscopy procedure for guiding the operator, as described herein. The computerized device 204 may receive the captured images via one or more imaging interfaces 220, e.g., a wired connection (e.g., a physical port, e.g., output from the imaging probe 212 plugged into the imaging interface via a cable), a wireless connection (e.g., an antenna), a local bus, a port for connecting to a data storage device, a network interface card, other physical interface implementations, and / or virtual interfaces (e.g., a software interface, a virtual private network (VPN) connection, an application programming interface (API), a software development kit (SDK)).

[0081] The computerized device 204 analyzes the captured images as described herein and generates several instructions for dynamically adjusting a graphical user interface presented on a user interface (e.g., a display) 226, for example, several elements of the GUI are injected as an overlay on the several captured images and presented on the display, as described herein.

[0082] The computerized device 204 can be implemented as, for example, a dedicated device, a client, a server, a virtual server, a colonoscopy workstation, a gastroenterology workstation, a virtual machine, a computing cloud, a mobile device, a desktop computer, a thin client, a smartphone, a tablet computer, a laptop computer, a wearable computer, an eyeglass computer, or a watch computer. The computerized device 204 can include an advanced visualization workstation, sometimes attached to a gastroenterology and / or colonoscopy workstation and / or other device, for enabling the operator to view the GUI created from processing the colonoscopy images, such as by real-time rendering arrows pointing to polyps not currently seen in the image and / or colonogram, displaying 2D and / or 3D locations of polyps, and / or other features described herein.

[0083] The computerized device 204 may include locally stored software that executes reference Figure 1 One or more actions described and / or may act as one or more servers (e.g., network servers, web servers, a computing cloud, virtual servers) that provide services to one or more clients 208 (e.g., clients used by a user to view colonoscopy images, such as a colonoscopy workstation including a display presenting the images captured by the colonoscope 212, a remotely located colonoscopy workstation, a PACS server, a remote EHR server, a remote display for, for example, medical students to remotely view the procedure) (e.g., referring to Figure 1 Services may be provided over the network 210, such as providing software as a service (SaaS) to the client(s) 208, providing an application for local download to the client(s) 208 as a response to a web browser and / or a colonoscopy application, and / or providing functionality to the client(s) 208 using a remote access session, such as through a web browser, application programming interface (API), and / or software development kit (SDK), for example, for injecting GUI elements into the colonoscopy images and / or presenting the colonoscopy images within the GUI.

[0084] Several different architectures of system 200 can be implemented. For example:

[0085] The computerized device 204 is connected between the imaging probe 212 and the display 226, e.g., as components of a colonoscopy workstation. This implementation can be used for real-time processing of the images captured by the colonoscope during the colonoscopy procedure, and can be used for presenting the GUI described herein on the display 226 in real time, e.g., injecting the GUI elements and / or presenting the images within the GUI, e.g., displaying directional arrows pointing to currently invisible polyps, and / or displaying a colon map depicting the 2D and / or 3D locations of polyps and / or other features described herein. In such an embodiment, the computerized device 204 (e.g., including the imaging probe 212 and / or the display 226) can be installed for each colonoscopy workstation.

[0086] The computerized device 204 acts as a central server, providing services to a plurality of colonoscopy workstations, such as a plurality of clients 208 (e.g., including imaging probes 212 and / or displays 226), via a network 210. In such an embodiment, a single computerized device 204 may be installed to provide services to a plurality of colonoscopy workstations.

[0087] The computerized device 204 is installed as code on an existing device, such as a server 218 (e.g., a PACS server, an EHR server), to provide local offline processing for the corresponding device, such as performing offline analysis of multiple colonoscopy videos captured by different operators and stored in the PACS and / or EHR server. The computerized device 204 can be installed on an external device that communicates with the server(s) 218 ​​(e.g., a PACS server, an EHR server) via a network 210 to provide local offline processing for multiple devices.

[0088] The client(s) 208 may be implemented as, for example, a colonoscopy workstation, which may include an imaging probe 212 and a display 226, a desktop computer (e.g., running a viewer application for viewing colonoscopy images), a mobile device (e.g., a laptop, a smartphone, glasses, a wearable device), and a remote station server for remotely viewing colonoscopy images.

[0089] The hardware processor(s) 202 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), field programmable gate array(s) (FPGA), digital signal processor(s) (DSP), and application specific integrated circuit(s) (ASIC). The processor(s) 202 may include one or more processors (homogeneous or heterogeneous) that may be configured for parallel processing, as a cluster, and / or as one or more multi-core processing units.

[0090] The memory 206 (also referred to herein as a program memory and / or data storage device) stores code instructions executed by the hardware processor(s) 202, such as random access memory (RAM), read-only memory (ROM), and / or a storage device, such as non-volatile memory, magnetic media, semiconductor storage devices, hard disk drives, removable storage, and optical media (e.g., DVD, CD-ROM). For example, the memory 206 may store code 206A and / or GUI code 206B, wherein the code 206A implements the instructions described with reference to FIG. Figure 1 One or more actions and / or features of the method described herein, the GUI code 206B generates the plurality of instructions for presentation within the GUI, and / or displays the GUI described herein based on the plurality of instructions (e.g., injection of GUI elements into the colonoscopy images, an overlay of GUI elements on the images, presentation of the colonoscopy images within the GUI, and / or presentation and dynamic updating of the colonogram described herein).

[0091] The computerized device 204 may include a data storage device 222 for storing data, such as the plurality of received colonoscopic images, the colonogram, and / or the plurality of processed colonoscopic images presented within the GUI. The data storage device 222 may be implemented as, for example, a memory, a local hard drive, a removable storage device, an optical disk, a storage device, and / or a remote server and / or computing cloud (e.g., accessed via the network 210).

[0092] The computerized device 204 may include a data interface 224, optionally a network interface, for connecting to the network 210, such as one or more of the following: a network interface card, a wireless interface for connecting to a wireless network, a physical interface for connecting to a cable for network connection, a virtual interface implemented in software, network communication software providing higher-level network connection, and / or other implementations. The computerized device 204 may use the network 210 to access one or more remote servers 218, for example, to download updated imaging processing code, updated GUI code, and / or to obtain images for offline processing.

[0093] It should be noted that the imaging interface 220 and the data interface 224 can be implemented as a single interface (e.g., a network interface, a single software interface) and / or two separate interfaces, such as several software interfaces (e.g., as an API, a network port) and / or several hardware interfaces (e.g., two network interfaces) and / or a combination (e.g., a single network interface and two software interfaces, two virtual interfaces on a common physical interface, several virtual networks on a common network port). The term / component "imaging interface 220" may sometimes be interchanged with the term "data interface 224."

[0094] The computerized device 204 can use a network 210 (or another communication channel, such as via a direct link (e.g., cable, wireless) and / or an indirect link (e.g., via an intermediate computerized device, such as a server, and / or via a storage device)) to communicate with one or more of the following: server(s) 218, imaging probe 212, image repository 214, and / or client(s) 208, for example, in accordance with several different architectural embodiments described herein.

[0095] The imaging detector 212 and / or the computerized device 204 and / or the client(s) 208 and / or the server(s) 218 ​​include or communicate with a user interface 226 that includes a user interface for inputting data (e.g., marking a polyp for resection) and / or viewing the GUI that includes the colonoscopy images, the directional arrows, and / or the colonogram. Exemplary user interfaces 226 include, for example, one or more of a touch screen, a display, a keyboard, a mouse, augmented reality glasses, and voice-activated software using speakers and microphones.

[0096] In 100, the endoscope is inserted and / or moved into the colon of the patient. For example, the endoscope is advanced forward (e.g., from the rectum to the cecum), retracted (e.g., from the cecum to the rectum), and / or the direction of at least the camera of the endoscope is adjusted (e.g., up, down, left, right), and / or the endoscope is left in place.

[0097] It should be noted that the endoscope can be adjusted based on the GUI, for example manually by the operator and / or automatically by the user, for example the user can adjust the camera of the endoscope according to the presented arrows to recapture a polyp that has moved out of the several images.

[0098] At 102, the camera of the endoscope captures an image. The image is a 2D image with optional color. The image depicts the interior of the colon and may or may not depict a polyp.

[0099] Several images can be captured as a video stream, and individual frames of the video stream can be analyzed.

[0100] As described herein, the plurality of images may be analyzed individually and / or as a set of sequential images. Each image in the sequence may be analyzed, or some intermediate images may be ignored, optionally a predefined number, such as every third image being analyzed with the intermediate two images being ignored.

[0101] Optionally, one or more polyps depicted in the plurality of images are treated. The plurality of polyps may be treated by endoscopy. The plurality of polyps may be treated by surgical resection, for example, and sent to a pathology laboratory. The plurality of polyps may be treated by ablation.

[0102] Optionally, a plurality of treated polyps are marked, for example, manually by the physician (e.g., by making a selection using the GUI, by pressing a "polyp removal" icon) and / or automatically by code (e.g., detecting movement of the surgical resection device). As described herein, the plurality of marked treated polyps can be tracked and / or presented on the colonogram presented in the GUI.

[0103] At 104, the plurality of images are input to a detection neural network, which outputs an image output indicating whether a polyp is depicted in the image (or not depicted). The detection neural network may include a segmentation process that identifies the location of the detected polyp in the image, such as by generating a bounding box and / or other outline of the polyp in the 2D frame.

[0104] An exemplary neural network-based process for detecting multiple polyps is the automatic polyp detection system (APDS) described in reference to International Patent Application Publication No. WO 2017 / 042812, “A SYSTEM AND METHOD FOR DETECTION OF SUSPICIOUS TISSUEREGIONS IN AN ENDOSCOPIC PROCEDURE,” by the same inventors as the present application.

[0105] The automatic polyp detection process implemented by the detection neural network can be performed in parallel and / or independently of features 106 to 114, for example on the same computerized device and / or (several) processors and / or another physically connected computer graphics device and / or a platform connected to the computerized device that performs the several features described with reference to 106 to 114.

[0106] When the output of the neural network is calculated and provided for a previous endoscopic image that is sequentially earlier than the corresponding endoscopic image, and when the tracking position of the region depicting the polyp(s) within the corresponding endoscopic image (i.e., as described in reference 106) is located at a different position than the region output by the neural network for the previous endoscopic image, the enhanced image is created for the corresponding endoscopic image based on the calculated tracking position of the polyp. This may occur when the frame rate of the plurality of images is faster than the processing rate of the detection neural network. The detection neural network completes processing of an image after capturing one or more consecutive images. If the plurality of results of the detection neural network are used in this case, the calculated polyp positions for the plurality of older images may not necessarily reflect the polyp positions of the current image.

[0107] Optionally, one or more endoscopic images, including a continuous subset of the plurality of endoscopic images that delineate the tracked ROI (as described in reference 106), including the corresponding endoscopic image, and one or more images sequentially located earlier than the corresponding endoscopic image (e.g., captured before the corresponding endoscopic image) are input to the detection neural network. The continuous subset of the plurality of images can be input to the detection neural network (as described in reference 106) in parallel with the tracking process. Alternatively, the subset of the plurality of images can first be processed by the tracking process described in reference 106. One or more of the post-processed images can be translated and / or rotated to create a subset of the plurality of endoscopic images in which the region delineating the polyp(s) (e.g., ROI) detected by the tracking process is at the same approximate location in all of the plurality of images, e.g., at the same pixel location on the display of all images. The processed continuous subset of the plurality of images is input to the detection neural network for outputting the current region delineating the polyp(s). The regions detected by the detection neural network may be used to enhance the image.

[0108] Alternatively or additionally, as described with reference to 106 , the calculated tracking position depicting the region of the polyp within the image(s) and / or the output of the tracking process (e.g., a 2D transformation matrix between consecutive frames) are input to the detection neural network. The tracking position and / or 2D transformation matrix can be input to the neural network when the tracked position of the polyp is within the image or when the tracked position of the polyp is outside the image. The tracking position can be input to the neural network alone or in conjunction with one or more images (e.g., the current image and / or previous images). The output of the tracking process (e.g., the tracking position and / or 2D transformation matrix) can be used by the neural network process, for example, to improve the accuracy of correlation between polyp detections in consecutive frames. The output of the tracking process can increase the confidence level in the polyp detection process based on the neural network (e.g., when the tracked polyp was detected in previous frames) and / or can reduce false positive detections.

[0109] Optionally, in the event of a mismatch between the tracked position of the polyp calculated based on step 106 (e.g., an ROI delineating a polyp) and the output of the detection neural network, the position of the neural network is used. The position output by the neural network is used to generate a plurality of instructions for creating the enhanced image having an indication of the position of the polyp. The position of the polyp output by the neural network can be considered more reliable than the tracked position calculated as described with reference to step 106, although the position calculated by the tracking step is computationally more efficient and / or can be executed in a shorter time than the processing step of the neural network.

[0110] Optionally, as described with reference to 108, the 3D reconstruction of the 2D image is input to the detection neural network alone and / or in combination with the 2D image for outputting the indication delineating the region of the at least one polyp.

[0111] Now back Figure 1 At 106 , a position of a region (eg, ROI) delineating one or more polyps is tracked within the current endoscopic image relative to one or more previous endoscopic images.

[0112] It is important to note that the camera movement (e.g., direction, forward, backward) is tracked indirectly by tracking the movement of the ROI between images because the camera is moving while the positions of the polyps within the colon remain stationary. It is also important to note that some movement of the ROI between frames may be due to peristalsis and / or other natural movements of the colon itself, regardless of whether the camera is stationary or moving.

[0113] Optionally, the position is tracked in 2D. The polyp may be tracked by tracking the ROI that delineates the polyp. The ROI and / or polyp may be detected in one or more previous images by the detection neural network of feature 104.

[0114] The position of the polyp is tracked even when the polyp is not depicted in the current image, eg, the camera is positioned such that the polyp is no longer present in the image captured by the camera.

[0115] Optionally, a vector is calculated from a position within the current image to the position of the polyp and / or ROI located outside the previous image. The vector may be calculated, for example, from the position of the ROI on the last (or earlier) image depicting the ROI, from the center of the screen, from the center of a quadrant of the screen closest to the external ROI position, and / or from another area of ​​the image closest to the position of the external ROI (e.g., a predefined distance from the image border position closest to the position of the external ROI).

[0116] The indication of the vector may depict a direction and / or orientation for adjusting the endoscopic camera to capture another endoscopic image depicting the area of ​​the image.

[0117] Optionally, the tracking algorithm is feature-based. The features may be extracted from an analysis of the endoscopic image. Features may be extracted based on the SURF (Speed ​​Robust Features) extraction method, see Bay, Herbert, Tinne Tuytelaars, and Luc Van Gool. "Surf: Speeded up robust features." European conference on computer vision. Springer, Berlin, Heidelberg, 2006. For example, the kd-tree method may be selected based on the observation that the main motion during the colonoscopy is the motion of the endoscopic camera in the colon. Features may be matched between consecutive images based on their descriptions. For example, the best homography can be estimated using the Random Sample Consensus (RANSAC) method of the camera movement from frame to frame by calculating the two-dimensional affine transformation matrix (and its closest two-dimensional geometric transformation matrix), for which please refer to the document Vincent, Etienne, and Robert Laganiére. "Detecting planar homographies in animage pair." ISPA 2001. Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis. In conjunction with 23rd International Conference on Information Technology Interfaces (IEEE Cat.. IEEE, 2001), and the two-dimensional affine transformation matrix may be referred to the document Agarwal, Anubhav, CV Jawahar, and PJ Narayanan. "A survey of planar homography estimation techniques." Centre for Visual Information Technology, Tech. Rep. IIIT / TR / 2005 / 12 (2005).

[0118] A specific region of interest (ROI), optionally a bounding box representing the location of one or more polyps, can be tracked. The location of the ROI is tracked as the ROI moves outside of the image (e.g., video) frame. The ROI can be tracked until the ROI returns (i.e., is depicted again) in the current frame. For example, the ROI can be continuously tracked outside the boundaries of the plurality of image frames based on a tracking coordinate system defined outside the plurality of image frames.

[0119] During the time interval between when a polyp is detected (e.g., automatically by code and / or manually by an operator) and when the camera stops moving (e.g., when the operator focuses on the detected polyp), the polyp may move out of the frame. Instructions can be generated to present an indication of where the polyp (or a ROI associated with the polyp) is currently located outside the presented image. For example, the current image can be enhanced by presenting a directional arrow. The arrow points to the location where the operator should move the camera (i.e., the endoscope tip) so that the polyp (or the ROI associated with the polyp) is once again depicted in the new frame(s).

[0120] Alternatively, the arrow (or other indicator) may be presented until the camera moves too far from the tracked polyp (e.g., greater than a defined threshold). The predefined threshold may be defined, for example, as the distance between the new position of the polyp and the center of the current frame being greater than three times the length of the diagonal of the frame, in pixels, or other numerical values.

[0121] Now refer to Figure 3 , Figure 3 is a flow chart for tracking a position of a region delineating one or more polyps within a corresponding endoscopic image relative to one or more previous endoscopic images, according to some embodiments of the present invention.

[0122] A new (ie, current) image, optionally a new frame of a video captured by the camera of the colonoscope, is received at 302. The image may or may not depict one or more polyps being tracked.

[0123] At 304, irrelevant or misleading portions are removed from the image, such as the periphery that is not part of the colon, such as the lumen that is the dark area in the colon image from which light does not return to the camera (e.g., representing the distal portion of the colon), and / or such as reflections that are inconsistent in the consecutive frames because they depend on the light source moving with the camera.

[0124] At 306, the contrast-limited adaptive histogram equalization (CLAHE) process (e.g., Reza, Ali M. "Realization of the contrast-limited adaptive histogram equalization (CLAHE) for real-time image enhancement." Journal of VLSI signal processing systems for signal, image and video technology 38.1 (2004): 35-44) and / or other processing is performed to enhance the image. The speed-robust features (SURF) method is used to extract the features, optionally after the CLAHE (e.g., without significant delay).

[0125] In 308, when the current frame is the first frame of the sequence and / or the first frame of a detected polyp, iterate 302 to obtain the next frame. When the current frame is not the first frame, then implement 310 to process the current frame with the previous frame processed in the previous iteration.

[0126] At 310 , a geometry matrix is ​​optionally calculated based on a homography for the camera movement between two consecutive frames (eg, a frame number denoted as i and a frame number denoted as i+n, ie, a jump of n frames between two frames processed together).

[0127] In 312, when no transformation is found, the next image (e.g., frame) in the sequence is processed with the previous (i.e., current) frame (i.e., when number of frames i and i+n cannot be matched, then try to match number of frames i and i+n+1)) (through iteration 302), otherwise execute 314.

[0128] In 314, when a transformation between the previous two or more frames cannot be found (i.e., the position of the tracked object (i.e., polyp) in the last frame is not and / or cannot be calculated), execute 318, otherwise when the position of the polyp is calculated, execute 316.

[0129] At 316, when an object (i.e., a polyp) for tracking is identified in the image, the region being tracked is input to the current frame according to the transformation found in 310. Execution 322 is performed to generate a plurality of instructions for delineating the object (i.e., a polyp) in the current frame, for example, by a visual marker (e.g., a box).

[0130] Alternatively, at 318, the transforms for the intermediate frames for which no transform is found are interpolated based on the last transform found between the two frames before and after the intermediate frames.

[0131] At 320, when there is an object being tracked (i.e., a polyp), the region being tracked is input to the intermediate frames based on the interpolated transforms found at 318. Execution 322 generates instructions for delineating the object (i.e., a polyp) in these frames. Execution 316 may also be performed to delineate the object (i.e., a polyp) in the current frame.

[0132] When the object (i.e., polyp) is outside the frame, the process is iterated to track it at 324. The next frame to be processed may be a number of skip steps, such as ignoring two or three or more frames until a new frame is processed, or alternatively processing each frame.

[0133] Now refer to Figure 4 , Figure 4 FIG2 is a schematic diagram illustrating an example of a KD-tree matching between two consecutive images 402A and 402B, which are matched together by corresponding features (features are shown as plus signs, one labeled element 404, and matching features are marked with lines, one labeled element 406), for tracking a polyp within a region 408A and 408B, according to some embodiments of the present invention. Image 402A can be represented as frame number i. Image 402B can be represented as frame number i+3. As described herein, region E08A represents the original bounding box of the region, and region E08B represents the transformation of E08A, based on the transformation matrix calculated from the homographies of the matches between the keypoints.

[0134] Now refer to Figure 5 , Figure 5 According to some embodiments of the present invention Figure 4 A geometric transformation matrix 502 between the plurality of frames 402A to 402B.

[0135] Now refer to Figure 6 , Figure 66 and a sequentially later diagram 606 (e.g., two frames later) according to some embodiments of the present invention. Diagram 602 depicts a bounding box ROI 604 representing a detected polyp being tracked. In sequentially later diagram 606, the tracked bounding box is no longer depicted within the image, and the image is augmented with the presence of an arrow 608 indicating a direction in which to move the camera to re-delineate the ROI of the polyp in the captured image.

[0136] Now refer to Figure 7 , Figure 7 is a schematic diagram depicting a schematic diagram 702 of a detected polyp 704 and a sequentially later frame (ie, three frames later in time) 706 depicting the tracked polyp 708 calculated using the transformation matrix described herein, according to some embodiments of the present invention.

[0137] Now back to the reference Figure 1 At 108, a 3D reconstruction is calculated for the current 2D image. The 3D reconstruction may define a plurality of 3D coordinate values ​​for a plurality of pixels of the 2D image.

[0138] Optionally, a 3D reconstruction neural network is trained to output a 3D image from an input of a 2D image. The trained 3D reconstruction neural network produces more accurate 3D images from a plurality of 2D images compared to only a 3D reconstruction process (e.g., a standard 3D reconstruction process). The 3D reconstruction process is a different process than the 3D reconstruction neural network. The 3D reconstruction process can be based on a plurality of standard 3D reconstruction processes that, for example, use only a single 2D image to calculate the 3D reconstruction. The 3D reconstruction neural network is trained using a training dataset of a plurality of 2D images captured by the colonoscope camera and a plurality of corresponding 3D images created from the plurality of 2D images by the 3D reconstruction process. The 2D image is designated as input and the reconstructed 3D image is designated as ground truth.

[0139] Optionally, a process for calculating the 3D reconstruction of the current 2D image is based on a 3D neural network that outputs the 3D reconstruction. The 3D reconstruction neural network can be trained using a training dataset of pairs of 2D endoscopic images defining input images and corresponding 3D coordinate values ​​calculated for pixels of the 2D endoscopic images calculated by a 3D reconstruction process defining ground truth. The neural network, which can be trained on a large training dataset (e.g., on the order of 10,000 to 100,000 or 100,000 to 1,000,000 pixels), can provide higher accuracy than the 3D reconstruction process alone.

[0140] The neural network can be trained using pairs of in-vivo colon image data captured by the endoscopic camera (represented as input) and corresponding 3D values ​​estimated by a 3D reconstruction process (represented as ground truth). The 3D values ​​can be 3D coordinates calculated for 2D pixels of the images. The 3D reconstruction process can be trained using a large dataset (e.g., at least 100,000 colon images from at least 100 different colonoscopy videos, or other smaller or larger datasets).

[0141] An exemplary process for 3D reconstruction of the several 2D colon images is now described: the 3D geometry of the portion of the inner surface area of ​​the colon can be reconstructed for each frame, for example, using the Shape from Shading (SfS) process described in Zhang, Ruo et al. "Shape-from-shading: a survey." IEEE transactions on pattern analysis and machine intelligence 21.8 (1999): 690-706 and / or Prasath, VB Surya et al. "Mucosal region detection and 3D reconstruction in wireless capsule endoscopy videos using active contours." 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2012. For example, the camera's shift parameters can be calculated based on a Shape from Motion (SfM) process, as described in Szeliski, Richard, and Sing Bing Kang. "Recovering 3D shape and motion from image streams using nonlinear least squares." Journal of Visual Communication and Image Representation 5.1 (1994): 10-28. The 3D position of one or more feature points can be calculated, optionally used to integrate the partial surfaces reconstructed by the SfS algorithm. The SfS algorithm processes the moving local light and light attenuation. The real situation of endoscopic examination in human organs can be simulated. An exemplary advantage of the 3D reconstruction process described here compared to other SfS-based processes is that the 3D reconstruction process described here can calculate a clear reconstructed surface for each frame. The 3D reconstruction process can use an intensity threshold to delete the non-Lambertian (e.g., the more specular) areas so that the SfS process is applicable to the other areas.The SfS process implemented by several inventors (see the document Prados, E., Faugeras, O.: Shape from shading: a well-posed problem?. IEEE Conference on Computer Vision and Pattern Recognition, 870–877 (2005)) creates the several explicit surfaces. This can be done by considering 1 / r. 2 Light attenuation and / or assuming that the point light source is connected to the center of the camera projection to calculate an explicit surface, so the image brightness is expressed as E = ∝I cosθ / r^2, where θ is the angle between the surface normal and the incident light, α is the albedo, and r is the distance between the light source and the surface point. The 3D reconstruction process described herein can use the motion information obtained by the SfM process to integrate parts of the surface obtained from different frames. Several non-uniform areas can be identified as several feature points. The SfM process can use these features to estimate the several extrinsic parameters of the camera for each frame, for example, as described in Kaufman A., Wang J. (2008) 3D Surface Reconstruction from Endoscopic Videos. In: Linsen L., Hagen H., Hamann B. (eds) Visualization in Medicine and Life Sciences. Mathematics and Visualization. Springer, Berlin, Heidelberg. Compared to the ICP (Iterative Closest Point) algorithm (e.g., see Kaufman A., Wang J. (2008) 3D Surface Reconstruction from Endoscopic Videos. In: Linsen L., Hagen H., Hamann B. (eds) Visualization in Medicine and Life Sciences. Mathematics and Visualization. Springer, Berlin, Heidelberg), the calculated information provides higher accuracy for the integration of partial surfaces of various frames.

[0142] According to Kaufman A., Wang J. (2008) 3D Surface Reconstruction from Endoscopic Videos. In: Linsen L., Hagen H., Hamann B. (eds) Visualization in Medicine and Life Sciences. Mathematics and Visualization. Springer, Berlin, Heidelberg, the average reprojection error for the selected feature points is 0.066 pixels, and the described 3D reconstruction process is applicable to several colonoscopy video frames.

[0143] The standard 3D processing described herein can be used to calculate the estimate of the 3D reconstruction of the colon for each 2D frame. The 2D frames and the corresponding calculated ground truth (i.e., the 3D values ​​of the 3D reconstructions of the 2D frames) can be split to create a training dataset (e.g., 70% of the data) and a test dataset (e.g., 30% of the data). For example, a convolutional neural network (CNN) based on an implementation similar to an encoder-decoder architecture (e.g., see Shan, Hongming, et al. "3-Dconvolutional encoder-decoder network for low-dose CT via transferlearning from a 2-D trained network." IEEE transactions on medical imaging 37.6 (2018): 1522-1534) can be trained and tested using training and validation datasets to robustly predict the 3D reconstruction of each 2D frame.

[0144] In an exemplary architecture, the encoder and decoder portions of the 3D reconstruction CNN can be constructed from several 2D convolutional layers. The last layer can be multiplied 3 times to predict the 3D coordinate value of each pixel of the input image. The 3D reconstruction CNN can be used in real time to predict several 3D values ​​for each pixel in each 2D frame of the colon surface in the colonoscopy video. The CNN (trained on the data output by the 3D reconstruction process) can optionally use a large number of processed 2D images (e.g., on the order of 100,000 or more or less) to process more robustly and / or more accurately than the 3D reconstruction alone.

[0145] Now refer to Figure 8 , Figure 8is a flow chart of a process for performing 3D reconstruction of 2D images captured by a camera within a patient's colon, according to some embodiments of the present invention. The 3D reconstructions may be defined as ground truth values ​​(associated with the 2D images defined as input values) for creating a training dataset for training a 3D reconstruction neural network (as described herein).

[0146] In 802, the plurality of images are provided, optionally as a video stream of a plurality of frames.

[0147] At 804, the current frame, denoted as n, is processed. Optionally, a SfS process of non-specular regions is used to compute the 3D reconstruction of frame n.

[0148] At 806, a plurality of images are processed, including a number of images acquired before and after the current image. The number of frames may be denoted as n-2 through n+2, or other numbers may be used. The SfM process may be used to compute a general 3D reconstruction of frame #n based on the number of frames #n- through #n+2 and the ongoing number of updated features based on the estimate of the intrinsic camera parameters.

[0149] In 808 , the 3D reconstructions computed in 104 and 106 are integrated into frame n.

[0150] At 810, a 3D reconstruction is provided for each pixel in 2D frame #n, representing the observed colon region. Each pixel in 2D frame #n is assigned an (x, y, z) value in a 3D coordinate system. The origin of the 3D coordinate system can be relative to the center point of the frame.

[0151] An exemplary data flow for a 3D reconstruction CNN is now described. In a first stage shown, several consecutive 2D frames (i.e., an odd number of at least 3, the number of consecutive input frames is denoted as n) are fed into the 3D CNN and passed through a layer of 3D convolution kernels (n*3*3). The first layer is designed to process all 3 color channels of the 2D frame by replicating it 3 times. The data flow continues through a Batch Normalization and several Relu layers, and a Max Pooling layer, where the Max Pooling layer reduces the size of the output by 2 in each frame dimension. The data flow passes through another four sets of layers, but with several 2D convolution kernels, which represent the encoder part of the 3D CNN. The data continues through an additional four sets of layers with several 2D convolutions, but using upsampling layers instead of max pooling layers, and then through a fifth layer repeated three times, which represents the encoder part of the CNN. The resolution of the output is the same as the resolution of the several input 2D frames. Outputs three values ​​for each 2D pixel, which are the pixel (x, y, z) 3D coordinate values ​​of the 2D pixel.

[0152] Optionally, the calculated 3D values ​​for the pixels of the 2D frame can be fed into the polyp detection process (e.g., the APDS detection process) described with reference to 104. The calculated 3D values ​​provide additional input (e.g., to the neural network) for detecting a polyp in the current frame and / or for determining the location of the polyp in the current frame. For example, the 3D values ​​can be information indicating the flatness and / or protrusion of the colon tissue region suspected of being a polyp (relative to its surroundings).

[0153] Now refer to Figure 10 At 110, the current 3D position of the polyp and / or endoscope is calculated and / or tracked. The 3D position may be calculated based on the output of the 3D reconstruction output by the 3D reconstruction neural network.

[0154] The 3D position can be calculated based on a 3D rigid body transformation matrix calculated between the plurality of consecutive 2D endoscopic images based on the plurality of 3D coordinates of the plurality of matching features extracted from the plurality of consecutive 2D endoscopic images and / or from the 3D reconstruction of the plurality of consecutive 2D endoscopic images. In other words, the 3D transformation matrix is ​​calculated for the plurality of matching features using the plurality of 3D coordinates of the plurality of pixels corresponding to the plurality of features, for example, similar to the process described herein for tracking a plurality of 2D images based on a plurality of corresponding features between the plurality of 2D images, but using the plurality of 3D coordinates of the plurality of features instead of the plurality of 2D coordinates. The 3D body transformation matrix represents the current 3D position of the endoscopic camera between the plurality of consecutive 2D endoscopic images. A movement trajectory of the camera can be calculated based on the plurality of consecutive 3D transformation matrices. The movement trajectory of the camera can be presented in the colonogram, as described herein.

[0155] Optionally, the 3D position of the detected polyp and / or endoscope is iteratively tracked based on several calculated 3D positions for a plurality of consecutive images.

[0156] The 3D tracking process receives the output of the 2D tracking process described in reference 106 and / or the 3D reconstruction process described in reference 108. The feature extraction part and / or the matching part can implement the 2D tracking on the several 2D consecutive frames of the colonoscopy video as described in reference. However, it should be noted that the homography is calculated on the several 3D values ​​(x, y, z) of the several extracted features (key points), rather than on the several 2D values. The several 3D values ​​are reconstructed by the 3D reconstruction process described herein to find the best fitting 3D affine transformation. The 3D rigid body transformation that is closest to the calculated 3D affine transformation is calculated, for example using the process described in the following document: Yuan, Jie et al. "Application of Feature Point Detection and Matching in 3D Objects Reconstruction", PATTERNS 2011: The Third International Conferences on Pervasive Patterns and Applications, 19-24. The 3D rigid body transformation matrix describes the 3D movement of the camera from frame to frame in the colon.

[0157] Now refer to Figure 9 , Figure 9is a flow chart depicting an exemplary 3D tracking process for tracking 3D movement of the camera according to some embodiments of the present invention.

[0158] In 1102, the number of matching features between frame n and the next analyzed consecutive frame n+i is provided, for example, as a reference Figure 5 The output of the process of executing 510 is described.

[0159] At 1104, the calculated 3D coordinate values ​​of the matching features (keypoints) between frames n and n+i are optionally provided from the 3D reconstruction process (eg, output by the 3D reconstruction neural network), as described herein.

[0160] In 1106, the 3D holography is calculated based on the plurality of 3D coordinate values ​​(1104) of the plurality of matching features (1102).

[0161] In 1108, the closest 3D rigid body transformation matrix is ​​found.

[0162] In 1110, the affine 3D transformation matrix is ​​calculated.

[0163] In 1112, the 3D rigid body transformation matrix describing the movement of the camera from the nth frame to the n+ith frame is calculated.

[0164] Now refer to Figure 10 , Figure 10 is an example of a 3D rigid body transformation matrix 1202 for tracking 3D movement of a colonoscopy camera according to some embodiments of the present invention.

[0165] Now refer to Figure 1. The 3D tracking algorithm is capable of constructing a 3D trajectory of several movements of the camera (for example, in a colonoscopy process), which is calculated based on the 3D tracking of the position of the camera. The 3D trajectory can be defined according to a 3D coordinate system, for example, the origin is relative to the camera position when tracking has started (for example, when the scope has just entered the colon, such as the point [0,0,0] in the 3D coordinate system, in which several 3D values ​​of the first frame are reconstructed in the colon). The trajectory can be constructed by calculating the 3D position of the camera (for example, its principal point) for each new frame (and / or multiple new frames, for example between 1 and 25 new frames) relative to the last frame in which the 3D position is calculated. The calculation of the new 3D position can be derived from the 3D rigid body transformation matrix between the two frames (e.g., multiplying the newly calculated transformation matrix with the previously calculated 3D position of the camera) (e.g., immediately, without significant delay, such as during the time when the current frame is presented on the display until the next frame is presented).

[0166] Optionally, the 3D movement of the camera is tracked relative to the 3D position(s) of anatomical landmarks and / or the 3D position(s) of previously detected polyps. Anatomical landmarks may be predefined and / or set by the operator, such as the locations of hemorrhoids, the location of a cecum, anatomical abnormalities, and / or portions of the colon (e.g., transverse, ascending). Polyps may be automatically and / or manually detected during advancement of the camera for removal during retraction of the camera in the opposite direction.

[0167] At 112, the portions of the inner surface of the colon depicted in the images are calculated, for example, each portion being defined as one-third, one-quarter, one-eighth (or other equal division) of the circumference of the inner surface of the colon. The coverage of each portion can be calculated dynamically in real time, for example, to determine whether the plurality of currently captured images depicts the corresponding portion (e.g., a substantial portion thereof). The coverage of the entire (or portion of) the colon can be calculated as an aggregation of the coverage of a single portion, for example, to provide a result such as approximately 94% of the inner surface of the colon is covered and / or approximately 6% of the inner surface of the colon is not covered.

[0168] Optionally, an indication of an amount of the interior surface depicted in a plurality of images relative to an amount of interior surface not depicted is calculated by aggregating the portion covered during the helical scanning movement relative to the portion not covered during the helical scanning movement.

[0169] For example, the portion of the inner surface of the colon depicted in the endoscopic image(s) can be calculated based on an analysis of the 3D reconstruction of the image(s) and / or based on an analysis of the images themselves, e.g., based on an identification of the location of the lumen within the images. For example, the lumen can be identified as a region of pixels having intensity values ​​below a threshold indicating darkness (i.e., where the lumen does not reflect the light source back to the camera).

[0170] Optionally, multiple sequential images are aggregated to produce a single assembled image. The portion of the inner surface of the colon can be calculated for the single assembled image and / or for each image. The multiple sequential images can be taken at several different orientations of the camera, for example, as the camera is oriented clockwise, counterclockwise, using an x-pattern, or other movement. For example, as the camera is advanced and / or retracted, the camera can remain stationary in the same position along a long axis of the colon, or can be simultaneously moved and oriented, for example, in a spiral pattern.

[0171] Several cumulative portions of the inner surface of the colon depicted in several consecutive endoscopic images can be tracked by position. The portion of the inner surface of the colon can be calculated for each position in the colon for several different orientations of the camera without moving the camera forward or backward (for example, the displacement is zero, or below a predefined threshold). When the camera is moved (for example, when the colonoscope is removed from the colon), the portion of the inner surface of the colon depicted in the several captured images can be recalculated for each new position. For example, Q1, Q2, Q3, Q4 are one group for the current position and another group Q1, Q2, Q3, Q4 are the new position. Alternatively, several cumulative portions of the inner surface of the colon depicted in several consecutive endoscopic images are tracked during the spiral movement of the camera. As the camera is moved, the portion of the inner surface of the colon depicted in the several captured images can be recalculated in a spiral pattern, for example, individually iterating over the multiple quarters, for example, Q1, Q2, Q3, Q4, Q1, Q2, Q3, Q4, Q1, Q2, Q3, Q4, for the spiral movement.

[0172] Optionally, each portion corresponds to a time window, and the time window has an interval, and the interval corresponds to the amount of time to cover the entire inner circumference in the spiral scanning movement, that is, to return to the same arc position after a single spiral scan, that is, to complete about 360 degrees, or to return to the arc range that defines the same quarter block, that is, to complete Q2, Q3, Q4 and return to Q1 after Q1 is completed. An indication of the coverage of each portion can be updated to a correlation with the time window. For example, when all portions are continuously correlated with an indication of coverage (for example, all portions are painted red or other colors to obtain sufficient coverage), it represents that the operator is fully covering all portions. Each portion can be associated with the indication of sufficient coverage corresponding to a time interval of the time window, that is, the portion is fully covered until the end of the several current circumferential spirals and needs to be fully depicted in the new circumferential spiral. During the continuous spiral scanning, the operator can use the indication that all portions are fully covered as an indicator of correctly capturing the several intestinal images of the knot during the continuous spiral scanning. When one of the indication(s) of one of the portions changes to an indication of insufficient coverage, the operator may capture the corresponding portion in the image(s).

[0173] The time window can be an estimate of the speed of the helical scanning movement, for example, based on clinical guidelines and / or based on physician practices, such as approximately 2.5 seconds per quarter block, or approximately 5 seconds per quarter block, or several other values ​​per section, or approximately 10 seconds or approximately 20 seconds per circumferential helical scan, or other values. The time window can be dynamically calculated and adjusted based on a real-time measurement of the helical scanning movement performed by the operator. For example, when the operator stops the helical scan, for example, to focus on a polyp, the time window stops and resumes when the operator resumes the helical scan. When the operator decreases the helical scanning speed, for example, when the same operator or a student (e.g., a resident) performs the scan, the time window increases accordingly. The real-time speed of the helical scanning movement can be measured, for example, by an analysis of the captured images (e.g., tracking the distances between the matched extracted features given the frame capture rate) and / or by sensor(s) sensing the movement of the colonoscope. An indication of sufficient coverage is generated when one or more images predominantly depicting the respective portion are captured during the time window and / or another indication of insufficient coverage is generated when several images predominantly depicting the respective portion are not captured during the time window.

[0174] For example, an indication of sufficient coverage can be generated when at least 50%, or 60%, or 70%, or 80%, or 90%, or other intermediate or greater values, of the corresponding portion is depicted in the corresponding image. For example, the threshold for determining the amount of the desired portion in the corresponding image can be set based on the number of lenses and the area of ​​the inner surface of the colon depicted in the image.

[0175] Optionally, the 3D values ​​of the pixels in the tracked frames are integrated into a single 3D coordinate system. A 3D panoramic (assembled) image of the colon surface can be generated based on the single 3D coordinate system, for example based on the reference Morimoto, Carlos, and Rama Chellappa. "Fast 3D stabilization and mosaic construction." cvpr. IEEE, 1997. The assembled image can be constructed continuously (for example, as the endoscope is pulled out of the colon).

[0176] For each frame included in the assembled image, the luminal region (e.g., the center of the colon duct) can be detected, for example, by identifying dark pixels as having intensity values ​​below a threshold (e.g., below 20 or other value when the intensity range is 0 to 255), and the dark pixels have associated 3D positions (or the 3D positions associated with pixels in their close environment) that are farthest from the camera position when the frame is captured.

[0177] Optionally, the position around the inner surface of the colon covered by the current frame is calculated. Optionally, the inner surface of the colon is divided into, for example, four equal parts (a plurality of quarter blocks). The plurality of quarter blocks of the inner surface of the colon depicted by the current frame may be calculated, for example, the first, second, third or fourth quarter block of the colon. The plurality of captured images may be aggregated into the assembled image for incrementally covering the plurality of quarter blocks, optionally until the assembled image depicts all four quarter blocks, indicating that the entire circumference of the inner surface of the colon at the current position has been depicted in the plurality of images.

[0178] For example, the quarter-block(s) of the inner surface of the colon depicted in several single images and / or the assembled image can be calculated based on the detected lumen area. For each new frame (e.g., each rendered and / or added to the assembled image), the 3D position of the center pixel and / or the direction of the 3D position of the center pixel relative to the 3D position of the detected lumen is calculated. The 3D position of the lumen can be estimated based on the 3D positions of the several pixels in their surroundings. The quarter-block can be calculated for the current frame.

[0179] The estimated quarter blocks of the colon depicted by the currently presented frame and / or assembled image can be indicated to the user. An indication can be output when all four quarter blocks are covered, indicating that the camera can be moved to a new position, and / or when one or more quarter blocks are not covered and the camera is moved, another indication can be generated indicating a lack of adequate imaging of the local colon region. Using the indication, the physician performing the colonoscopy procedure can determine whether the four quarter blocks of the current local colon region are adequately covered by a number of consecutive frames. A quarter block that is adequately covered by a frame (e.g., for at least 5 seconds) can be indicated to the user so that the user can see the grouping of the four indicated quarter blocks on the screen as a sign that the inner surface of the colon is adequately covered during the colon scan. For example, a covered quarter block can be indicated by a number that appears and / or flashes on the screen.

[0180] Optionally, as the endoscope is advanced and / or retracted (e.g., on its way out of the colon, such as by being pulled out of the cecum until the endoscope is completely removed from the colon), the sequences of covered quarter-blocks are dynamically aggregated and / or labeled with the corresponding 3D positions. The calculated trajectory of the endoscope (e.g., as the endoscope is being advanced and / or retracted) can use a window of a predefined length (e.g., approximately 2 cm to 3 cm) and / or a predefined step length (e.g., approximately 0.5 cm to 1 cm). If a quarter-block is not covered at all in a number of consecutive scan windows (denoted as Ncs, having a value of, for example, 3), it is registered as a missed quarter-block. The percentage of the colon covered by the endoscopic camera when advancing and / or pulling can be calculated using the mathematical relationship: (1-Tms / ((Nsw / Ncs)*4)))*100, where Tms represents the total number of missed quarter blocks and Nsw represents the total number of scanning windows. For example, when the total number of missed quarter blocks (Tms) is 8, the total number of scanning windows (Nsw) is 100, and the number of consecutive scanning windows (Ncs) is 3, the percentage of the colon covered according to the mathematical relationship is 76%. The calculation of the percentage of the colon covered by the endoscopic camera can be performed dynamically in real time, for example based on an aggregation of the several images captured during the procedure, and / or offline after the procedure has been completed using a set of images captured during the procedure. Now referring to Figure 11 , Figure 11 13 is a schematic diagram of an assembled and / or panoramic image of the colon 1302 constructed for each pixel in the combined 2D images in which the 3D position in the single 3D coordinate system is calculated, according to some embodiments of the present invention. As shown, three 2D frames 1304 with a step of 3 frames between each consecutive frame are used to create panoramic image 1302. A luminal region 1306 is depicted as a dark area in the center of the colon tube. Panoramic image 1302 covers only a portion of the localized region of the colon. Region 1308 remains undocumented by the assembled image because several images of it have now been captured.

[0181] Now refer to Figure 12 , Figure 12 FIG2 is a schematic diagram illustrating a volumetric image 1402 depicted within a corresponding quarter of the inner surface of the colon according to some embodiments of the present invention. Section 1404A is depicted as quarter 1, section 1404B is depicted as quarter 2, section 1404C is depicted as quarter 3, and section 1404D is depicted as quarter 4.

[0182] Now refer to Figure 13 , Figure 13 is a flow chart of a method for calculating the quarter block depicted by a box according to some embodiments of the present invention. It should be noted that the selected quarter block represents the quarter block that is mostly covered by the frame, as the frame may overlap between two or more quarter blocks.

[0183] In 1502, the current frame (denoted as #n) is provided.

[0184] At 1504, frame #n is registered to the 3D assembled image being constructed.

[0185] At 1506, a lumen may be detected in the current frame. The 3D position of the lumen may be estimated.

[0186] In 1508 , when no lumen (eg, the dark area in the center of the colon duct) is detected in the last few frames of the current assembly (eg, in the last 10 seconds of the video), iterate R02 to R08 .

[0187] In 1510, when a lumen is detected, the 3D position of the central pixel of the image is calculated. When the plurality of central pixels are determined to be located in a lumen region, the 3D position is calculated based on a plurality of pixels in the close environment.

[0188] In 1512 , the relative direction (eg, 3D vector direction) between the 3D position of the center pixel of the current frame and the estimated 3D position of the last detected lumen region is calculated.

[0189] At 1514, based on the assumption that the vector for calculating the direction at 1512 starts from the center of the colon duct (i.e., the center of the lumen), the colon quarter block that covers most of the area of ​​the current frame is determined based on the direction of the vector. The selected quarter block is the quarter block that covers most of the current frame (e.g., if two or more quarter blocks are evenly covered, then none of the quarter blocks are covered).

[0190] The quarter block depicted in the current frame is provided at 1516. Instructions for presenting an indication of the determined quarter block in the GUI can be generated as described herein.

[0191] Now back Figure 1At 114, the dimension of the detected polyp is calculated. The dimension can be a 2D and / or 3D dimension, for example, the volume of the polyp, the radius of a sphere representing the polyp, the surface area of ​​the polyp (e.g., a flat polyp), and / or the radius of a 2D circle representing the polyp. The dimension can be calculated by performing a best fit of a 3D sphere and / or 2D circle to the number of pixels of the 3D image representing the polyp (created from the 2D image as described herein, optionally by the 3D CNN). The dimension is obtained from the best-fit 3D sphere and / or 2D circle, for example, the radius of the best-fit 3D sphere and / or the radius of the best-fit 2D circle.

[0192] As described in references 108 and / or 110, the dimension(s) of the polyp are calculated based on an ROI that delineates the 3D coordinates of the pixels of the polyp and / or the output 3D image (i.e., the 3D reconstruction of the 2D image) calculated by the 3D reconstruction neural network. The ROI delineating the polyp can be manually set by the operator (e.g., using the GUI) and / or automatically output by the detection network described in reference 104 when fed the 2D image(s).

[0193] The process described herein can calculate the dimensions of the 3D volume and / or 2D surface area of ​​the polyp using (optionally only using) the 3D coordinate values ​​(x, y, z) calculated for the pixels in the 2D images described herein (e.g., output by the 3D CNN). In contrast, other processes that use 2D images to calculate 3D and / or 2D dimensions require knowledge of the camera's characteristics (e.g., its pose relative to the polyp), which may be difficult to obtain.

[0194] The size of the polyp may be calculated based on a radius of a circle that best fits the polyp slice.

[0195] The volume of the polyp can be automatically calculated by taking the calculated 3D values ​​of the 2D pixels within the polyp that delineate an outline and / or bounding box, and finding a best-fit 3D sphere (or circle if the polyp is flat) to the exposed 3D surface created by interpolating between the 3D values ​​of the pixels. The radius of the sphere (or circle) is the polyp size.

[0196] It should be noted that the 3D volume can be calculated based on the calculated radius of a sphere associated with the polyp, as described herein.

[0197] For example, the estimated dimension (e.g., 3D volume) of the polyp(s) can be calculated by: calculating a best-fit 3D surface for the 3D coordinates of the pixels of the region of the at least one 2D image. Calculating a plurality of normal vectors about a 3D location relative to a centroid of the region delineating the polyp (i.e., ROI). Determining the side of the 3D surface based on the relative directions of the normal vectors. Calculating a plane that includes a normal vector of the normal vectors relative to the 3D location relative to the centroid. Calculating a vector at the 3D location relative to the centroid, the centroid being tangent to the best-fit 3D surface. Calculating a first curvature as a radius of a tangent parabola of a contour intersecting the best-fit 3D surface and the calculated plane. A second curvature is calculated as a radius of a tangent parabola of a contour intersecting between the best-fit 3D surface and an orthogonal plane orthogonal to the calculated plane, wherein the orthogonal plane includes the normal vector. A 3D radius of a 3D volume of the polyp is calculated as an average of the first curvature and the second curvature.

[0198] Now refer to Figure 14 , Figure 14 Included are several diagrams depicting the process for calculating the volume of a polyp from a 3D reconstructed surface calculated from a 2D image, according to some embodiments of the present invention. Diagram 1602 shows a 2D colonoscopy frame with a detected polyp 1604 obtained from the website sites(dot)google(dot)com(slash)site(slash)suryaiit(slash)research(slash)endoscopy(slash)sfs. Diagram 1606 shows the 3D reconstructed surface of frame 1602. Polyp 1608 is the 3D reconstruction of polyp 1604 from 2D image 1602. Image 1610 depicts the process of calculating the volume of the polyp by finding the best-fit (tangent) 3D sphere 1612 that is concave to the surface interpolated from the 3D values ​​of the pixels within the polyp region, as described herein. The volume is the radius (R) of the 3D sphere 1612 .

[0199] Now refer to Figure 15 , Figure 15 is a flow chart of an exemplary process for calculating a polyp volume from a 2D image according to some embodiments of the present invention.

[0200] At 1702, the 2D frame labeled #n is received, having a polyp detected. A plurality of frames before and after the current frame are received.

[0201] At 1704, the 3D reconstruction of frame #n is calculated as described herein.

[0202] At 1706, the pixels within the bounding box delimiting the contour and / or the polyp (as available) area are identified, and the surface that best fits their 3D values ​​is calculated.

[0203] At 1708, a plurality of normal vectors are calculated near the 3D point relative to the centroid of the bounding box of the polyp (when there is a delineated contour and the centroid is outside the delineated contour, the centroid is replaced by the closest pixel).

[0204] At 1710, the concavity of the surface is identified based on the calculated normal vectors and their relative directions.

[0205] At 1712, the infinite plane containing the normal vector relative to the centroid (or its alternative) is calculated, and the vector (tangent to the surface) at this 3D point (relative to the centroid) is calculated.

[0206] In 1714, the curvature of the contour (the radius of the tangent parabola) as the intersection between the polyp surface and the calculation plane is calculated, for example, based on the method described in the document Har'el, Zvi. "Curvature of curves and surfaces—aparabolic approach." Department of Mathematics, Technion—Israel Institute of Technology (1995).

[0207] At 1716, the last step is repeated for a plane that is orthogonal to the previous plane and includes the normal vectors from 1712 and 1714 (to obtain an additional estimate of the curvature).

[0208] At 1718, the average of the last two calculated curvatures is calculated to provide the 3D radius of the polyp.

[0209] At 1720, when the 3D radius is greater than a predetermined threshold (e.g., 7 mm, 7 cm, or other value), the polyp size is indicated as flat at 1722. The best-fit 2D circle is calculated. The size of the polyp is defined based on the size of the 2D circle. At 1726, when the 3D radius is less than the predefined threshold, the polyp size is defined based on the 3D radius.

[0210] Now refer to Figure 1 At 116, instructions are generated for updating a presentation of a GUI. The image may be enhanced. The instructions may be, for example, for injecting the GUI elements into the image, for presenting an overlay on the image, and / or for presenting the image in one portion of the GUI and presenting other graphical elements, such as the overlaid quadrants and / or the colonogram, in another portion of the GUI, as described herein.

[0211] The plurality of instructions are generated based on the plurality of outputs of one or more features described in references 104 to 114:

[0212] Generate a plurality of instructions to enhance the image using the location of the detected polyp as described in reference 104, such as marking the ROI (e.g., a bounding box) that depicts the detected polyp, and / or color-codes the polyp and / or bounding box, and / or an arrow pointing to the polyp.

[0213] Instructions are generated based on the indication of the vector, the vector pointing from the current image to the ROI, the ROI depicting the polyp located outside the current image, as described with reference to feature 106. The instructions are for creating an enhanced endoscopic image by enhancing the corresponding endoscopic image with an indication of the vector.

[0214] The vector may be represented as an arrow. The arrow points to the direction the camera should move to recapture the polyp in the image. The arrow and the plurality of images may be presented in 2D.

[0215] Optionally, when the polyp is recaptured in the image (e.g., after moving the camera in the direction of the arrow), the instructions are generated for augmenting the image with the ROI delineating the polyp. The ROI can be re-labeled on the image without having to perform feature 104, i.e., solely based on the tracking step.

[0216] Optionally, based on the output of the features described with reference to 106 and / or 110, instructions are generated for presenting a colonogram within the colon. The 2D and / or 3D location of the polyp is obtained (e.g., delineating the ROI of the polyp), e.g., output by the detection neural network and / or the 3D reconstruction neural network, and / or manually marked by the operator (e.g., using the GUI, e.g., by pressing a "polyp" icon). The location of the polyp on a schematic diagram of a colon of the patient is marked with an indicator, such as an X and / or a circle, optionally color-coded according to whether the polyp was identified during insertion and / or removal of the colonoscope. The colonogram is dynamically updated as newly detected polyps are detected.

[0217] Optionally, instructions are generated to mark the polyps displayed on the colonogram with a treatment indication (e.g., surgical resection, ablation). For example, polyps displayed as ovals in the colonogram have been removed and marked with an X. Ovals in the colonogram that have not been removed are not marked with an X. The polyps are marked as treated based on an indication of resection of the polyps, such as provided manually by the user (e.g., by pressing a "polyp removal" icon on the GUI) and / or automatically detected by code (e.g., based on detection of movement of surgical tools in the colonoscope).

[0218] Optionally, instructions are generated for drawing the tracked 3D positions of the endoscope on the colonogram presented in the GUI, for example as tracks and / or curves and / or dashed lines. The tracks and / or curves and / or dashed lines are dynamically updated as the endoscope moves (e.g., forward and / or backward) in the colon. Optionally, forward-direction tracked 3D positions are marked on the colonogram, wherein a forward direction indicating that the endoscopic camera is moving deeper into the colon (e.g., from the rectum to the cecum) is marked, for example, with arrows in the forward direction and / or using a color coding. Reverse-direction tracked 3D positions presented on the colonogram are marked with another marker indicating an opposite direction (e.g., from the cecum to the rectum) of the endoscopic camera being removed from the colon, for example, with arrows in the opposite direction and / or a different color.

[0219] Optionally, instructions are generated for presenting the calculated dimensions of the detected polyp. For example, the calculated dimensions are presented as a numerical value, optionally having units proximate to the ROI marked on the image, such as the numerical value of the calculated volume in cubic millimeters. In another example, the calculated dimensions are presented as a numerical value having units proximate to the indication of the polyp on the colonogram.

[0220] Optionally, when the dimension is above a threshold, instructions are generated for presenting an alert within the GUI indicating a recommendation to remove the polyp. For example, when the radius of a sphere defining the 3D volume of the polyp is greater than 2 mm. For example, the alert may be generated, for example, by coloring the polyp and / or using a unique color (e.g., red for removal and green for retention) indicating a removal recommendation for the boundary of the ROI of the polyp on the image, and / or coloring the polyp presentation on the colonogram, and / or an audio message, and / or a pop-up text message within the GUI.

[0221] Optionally, instructions are generated for presenting an estimate of the remaining portions of the inner surface area that have not yet been depicted in any previously captured endoscopic images. Alternatively or additionally, instructions are generated for presenting an estimate of the portions of the inner surface area that have already been depicted. For example, a GUI element is presented that is the shape of a circle divided into 4 quadrants (or other number of divisions, e.g., as slices). The quadrants that are depicted in the images are displayed using a marker, e.g., green. The quadrants that have not yet been depicted are displayed using a different marker, e.g., red. The operator can view the marked locations (e.g., color-coded circles) and orient the camera in the direction toward the quadrants that have not yet been imaged. Once the GUI element depicts that all quadrants have been imaged, the operator can move the colonoscope (e.g., forward or backward).

[0222] The quadrants may represent a majority of the imaged areas, for example, representing greater than 50%, 70%, or 80% of the surface of the quadrants in one or more images. Other divisions may be selected based on the imaging capabilities of the camera lens, for example, a greater number of divisions may be used for narrow-angle lenses.

[0223] Optionally, when the current position of the camera approaches the anatomical landmark and / or polyp, instructions for presenting an alarm are generated according to predefined system definitions (e.g., which can be set from time to time via a settings menu). For example, the definitions can be configured such that when the camera position approaches a location of the landmark, an alarm is generated when the distance is within a predefined threshold. The distance between the camera and the landmark and / or polyp can be calculated, for example, using the L2 metric and / or a proximity that is small enough to give an alarm (e.g., approximately 3 cm). The threshold distance value can take into account the aggregate calculation errors in constructing the trajectory and / or the fact that the colon itself is not completely stable in the stomach, i.e., not stable in the coordinate system used to construct the trajectory.

[0224] Optionally, the amount of time the endoscope spends in each defined portion of the colon can be calculated and presented. For example, each portion can be defined based on the number of transition points between a number of anatomical landmarks that divide the colon into multiple portions, such as the ascending colon and the transverse colon (e.g., between the transverse colon and the ascending colon). The several anatomical landmarks can be manually detected by the user (e.g., the user marks the landmarks using the GUI) and / or automatically detected by code (e.g., based on an analysis of the several images). The several 3D positions of the endoscopic camera relative to the (several) anatomical landmarks are tracked. The amount of time the endoscopic camera spends in each portion of the colon is calculated. Several instructions are generated for presenting the amount of time the endoscopic camera spends in each portion of the colon in the GUI. For example, the time can be presented as a mark on the corresponding portion corresponding to the colonogram of the portion of the patient's colon.

[0225] The generated instructions are executed at 118. The GUI is updated according to the instructions.

[0226] Now refer to Figure 16 , Figure 161 is a schematic diagram depicting a series of enhanced images for tracking an ROI of a polyp presented in the GUI based on instructions generated from the data according to some embodiments of the present invention. The sequence depicts 2D tracking of the polyp and generating an arrow in the direction of the polyp when the polyp is outside the image, as described herein. Image 1902 is a first image enhanced with an ROI 1904 depicting the automatically detected polyp. In the second image of the sequence 1906, ROI 1904 has moved to the left side of the image due to a reorientation of the camera. In a third image of the sequence 1908, ROI 1904 has moved further to the left side of the image. In a fourth image of the sequence 1910, the ROI is outside the image and, therefore, not depicted in image 1910. An arrow 1912 is presented pointing in the direction of the outside ROI. In a fifth image of the sequence 1914, ROI 1904 reappears after the operator redirects the camera in the direction indicated by arrow 1912 in image 1910. In a sixth image of the sequence 1916, ROI 1904 has moved further to the right due to the operator moving the camera.

[0227] Now refer to Figure 17 , Figure 17FIG2 is a schematic diagram of a colonogram 2002 showing movement trajectories of the endoscope, locations of detected polyps, removed polyps, and anatomical landmarks presented within the GUI, according to some embodiments of the present invention. Colonogram 2002 can be presented, for example, as an overlay presented in a corner of the images, or in a designated window of a GUI having another designated window for presenting the images. Trajectory 2004 depicts the tracked 3D movement of the endoscope during forward movement into the colon, e.g., color-coded. Trajectory 2006 depicts the tracked 3D movement of the endoscope during reverse movement out of the colon, e.g., color-coded using a different color. Ovals 2008 represent polyps automatically detected during forward movement of the colonoscope, e.g., colored using the same color as forward trajectory 2004. Ovals 2010 represent polyps automatically detected during reverse movement of the colonoscope, e.g., colored using the same color as reverse trajectory 2006. Circles 2012 represent a polyp manually discovered by the user and / or an automatically detected polyp manually marked by the user. An X mark 2014 represents a polyp removed by the operator. Arrow 2016 marks the current 3D position of the distal end (e.g., tip) of the endoscope, and optionally a camera. Box 2018 represents a manually designated landmark entered by the user (e.g., via the GUI), such as a hemorrhoid. Circle 2020 represents an automatically identified landmark, e.g., detected by an analysis of the 3D reconstructed images. The colonogram 2002 is dynamically updated as the colonoscope is moved, polyps are detected, and / or polyps are removed.

[0228] Now refer to Figure 18 , Figure 1821 is a diagram 2102 depicting a plurality of quadrants of the inner surface of the colon depicted in one or more images 2104A-2104C and / or a quadrant of the inner surface of the colon not yet depicted in image 2106, according to some embodiments of the present invention. As the camera moves forward and / or backward, diagram 2102 is created for each new position of the camera. Alternatively or additionally, diagram 2102 is calculated for a recently defined time window (e.g., 5 seconds, 10 seconds, or some other value). The time window may be short enough to exclude forward and / or backward movement of the camera, but long enough to fully image the circumference of the inner wall of the colon. The quadrants are updated as new images are acquired at the same position by reorienting the camera. The quadrants 2104A-2104C and 2106 may be color-coded. As the operator moves the camera to image the remaining portions of the inner surface of the colon, schematic diagram 2102 is dynamically updated.

[0229] At 120, one or more features described with reference to 100 through 118 are iterated. The iterations may dynamically update the GUI, for example, to dynamically track the polyps, dynamically generate arrows pointing to ROIs (depicting polyps located outside the current image), update the colon map with new 2D and / or 3D polyp locations, update the camera trajectory of the colon map as the camera moves, update the GUI to depict coverage of the inner surface of the colon (e.g., quadrilaterals), and / or update the calculated 2D and / or 3D sizes of the detected polyps.

[0230] The updated GUI can be used by the operator, for example, to manipulate the camera to capture images of polyps, to determine which polyps to remove, to manipulate the camera to ensure complete coverage of the inner surface of the colon, and / or to track the location of the camera and / or detected polyps within the colon.

[0231] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the described embodiments, the practical applications, or technical improvements to technologies found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0232] It is anticipated that many related endoscopes will be developed during the life of the patent from the application described, and the scope of the term endoscope is intended to include all such a priori novel technologies.

[0233] As used herein, the term "about" refers to ±10%.

[0234] The terms "comprising," "having," and their conjugations mean "including but not limited to." Such terms encompass the terms "consisting of" and "consisting essentially of.

[0235] The phrase "consisting essentially of" means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0236] As used herein, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

[0237] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude such combination of features of other embodiments.

[0238] The word “optionally” is used herein to mean “provided in some embodiments and not provided in several other embodiments.” Any particular embodiment of the present invention may include multiple “optional” features, unless such features are in conflict.

[0239] In this application, various embodiments of the present invention can be presented in a range format. It should be understood that the description of the range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Therefore, it should be considered that the description of the range has specifically disclosed all possible sub-ranges and each numerical value within the range. For example, the description of a range such as from 1 to 6 should be considered to have specifically disclosed sub-ranges (e.g., from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc.) and a single number (e.g., 1, 2, 3, 4, 5, and 6) within the range. This applies regardless of the breadth of the range.

[0240] Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integer) within the indicated range. The phrases "a range between" a first indicated numeral and a second indicated numeral and "a range from" a first indicated numeral to" a second indicated numeral are used interchangeably herein and are intended to include the first and second indicated numerals and all decimals and integers therebetween.

[0241] It should be understood that certain features of the invention described in the context of separate embodiments for the sake of clarity may also be provided in combination in a single embodiment. Conversely, various features of the invention described in the context of a single embodiment for the sake of brevity may also be provided individually or in any suitable subcombination or in any other described embodiment of the invention as appropriate. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperable without those elements.

[0242] Although the present invention has been described in conjunction with its specific embodiments, it is obvious that many alternatives, modifications and variations will be apparent to those skilled in the art. It is therefore intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

[0243] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety, to the same extent as if each individual publication, patent, or patent application were specifically and individually indicated as being incorporated herein by reference. In addition, the citation or identification of any reference in this application should not be construed as an admission that such reference can serve as prior art for the present invention. To the extent that several section headings are used, they should not be construed as necessarily limiting. In addition, any priority document of this application is incorporated herein by reference in its entirety.

Claims

1. A system for generating a plurality of instructions for presenting a graphical user interface (GUI) for dynamically tracking at least one polyp in a plurality of endoscopic images of a colon of a patient, characterized in that: The system comprises: At least one processor executes code for: Iterating over the plurality of endoscopic images: tracking a position of a region that delineates at least one polyp within a corresponding endoscopic image relative to at least one previous endoscopic image, wherein the corresponding endoscopic image represents at least one current endoscopic image; When the position of the region is outside the at least one current endoscopic image, wherein the region depicting the at least one polyp has disappeared from the at least one current endoscopic image, and the region depicting the at least one polyp is no longer depicted in the at least one current endoscopic image: calculating a vector from the region of the at least one polyp within the at least one current endoscopic image to the position of the region no longer appearing in the at least one current endoscopic image and located outside the at least one current endoscopic image; creating an enhanced endoscopic image by enhancing the corresponding endoscopic image using an indication of the vector; and A plurality of instructions are generated for presenting the enhanced endoscopic image within the GUI.

2. The system according to claim 1, wherein The indication of the vector depicts a direction and / or orientation for adjusting an endoscopic camera to capture at least one further endoscopic image depicting the region of at least one image.

3. The system according to claim 1, wherein: When the location of the area delineating at least one polyp appears in the corresponding endoscopic image, executing code by the at least one processor enhances the corresponding endoscopic image using the location of the area to create an enhanced image, wherein the indication of the vector is excluded from the enhanced endoscopic image.

4. The system according to claim 1, wherein: The system further includes code for: calculating a location of the region, the location of the region delineating at least one polyp within the colon of the patient; creating a colonogram by plotting a schematic diagram representing the colon of the patient, the schematic diagram showing an indication of the location of the region delineating at least one polyp; and Instructions are generated for presenting the colon map within the GUI, wherein the colon map is dynamically updated with locations of newly detected polyps.

5. The system according to claim 1, wherein: The system further includes code for: translating and / or rotating at least one endoscopic image of a continuous subset of the plurality of endoscopic images including the corresponding endoscopic image for creating a processed continuous subset of the plurality of endoscopic images, wherein the region depicting the at least one polyp is at a same position in all of the images of the continuous subset of the plurality of endoscopic images; inputting the processed continuous subset of the plurality of endoscopic images into a detection neural network; Outputting a current region by the detection neural network, wherein the current region depicts the at least one polyp in the corresponding endoscopic image; creating an enhanced image of the corresponding endoscopic image by enhancing the corresponding endoscopic image using the current region; and A plurality of instructions for presenting the augmented image in the GUI are generated.

6. The system according to claim 5, wherein: When a previous endoscopic image that is sequentially earlier than the corresponding endoscopic image provides the output of the neural network and the tracked position of the area delineating at least one polyp within the corresponding endoscopic image is at a different location than the area output by the neural network for the previous endoscopic image, the at least one processor executes code to create the enhanced image for the corresponding endoscopic image based on the tracked position.

Images