Ai-based imaging mode selection in endoscopy

Abstract

Systems, devices, and methods for determining or adjusting an imaging mode for use in inspecting tissue or foreign objects during an endoscopy procedure are disclosed. An exemplary endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging system to obtain images or video streams of a target anatomy during an endoscopy procedure. The controller circuit can generate from the obtained images or video streams endoscopic image or video features characterizing an anomaly in the target anatomy, and determine a personalized, pathology-specific imaging mode to enhance discernability of the anomaly from the images or video streams. The recommended imaging mode may be determined using a trained machine learning model. The recommended imaging mode may be provided to a user or a process to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

Description

AI-BASED IMAGING MODE SELECTION IN ENDOSCOPYPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 582,029, filed September 12, 2023, the contents of which are incorporated herein by reference.FIELD OF THE DISCLOSURE

[0002] The present document relates generally to endoscopic systems, and more particularly to systems and methods of determining or adjusting an imaging mode for use in inspecting tissue or foreign objects during an endoscopy procedure.BACKGROUND

[0003] Endoscopes have been used in a variety of clinical procedures, including, for example, illuminating, imaging, detecting and diagnosing one or more disease states, providing fluid delivery (e.g., saline or other preparations via a fluid channel) toward an anatomical region, providing passage (e.g., via a working channel) of one or more therapeutic devices or biological matter collection devices for sampling or treating an anatomical region, and providing suction passageways for collecting fluids (e.g., saline or other preparations), among other procedures. Examples of such anatomical region can include gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary duct, intestines, colon, and the like), renal area (e.g., kidney(s), ureter, bladder, urethra) and other internal organs (e.g., reproductive systems, sinus cavities, submucosal regions, respiratory tract), and the like.

[0004] Some endoscopes include a working channel through which an operator can perform suction, placement of diagnostic or therapeutic devices (e.g., a brush, a biopsy needle or forceps, a stent, a basket, or a balloon), or minimally invasive surgeries such as tissue sampling or removal of unwanted tissue (e.g., benign or malignant strictures) or foreign objects (e.g., calculi). Some endoscopes can be used with a laser or plasma system to deliver energy to an anatomical target (e.g., soft or hard tissue or calculi) to achieve desired treatment. For example, laser has been used in applications of tissue ablation,coagulation, vaporization, fragmentation, and lithotripsy to break down calculi in kidney, gallbladder, ureter, among other stone-forming regions, or to ablate large calculi into smaller fragments. One example of endoscopy is colonoscopy for used to reduce incidence and mortality rate of colorectal cancer. Colonoscopy is typically performed with rapid advance of a colonoscope to the cecum and then performing a thorough inspection to detect anomalies (e.g., polyps) and necessary treatment (e.g., polypectomy) during withdrawal.

[0005] An endoscope generally includes an imaging sensor (e.g., a camera) that can take live images or video streams during an endoscopy procedure. The imaging sensor can operate under a preset imaging mode to obtain the live images or video streams. The imaging mode may include lighting mode, optical magnification, viewing angle of the imaging sensor, among other settings of the imaging sensor and lighting system. Different imaging modes have been used in real-time endoscopic inspection and optical diagnosis, such as high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), etc. Optical magnification has been recommended for inspecting lesions or pathologies of distinct characteristics. Proper selection of an imaging mode can improve image or video quality and facilitate inspection and diagnosis of anomalous tissue or foreign objects of interest during the procedure.SUMMARY

[0006] Various specialized imaging modalities exist and have been used to enhance detection and / or diagnosis of certain types of pathologies or anomalies during an endoscopy procedure. For example, Image enhanced endoscopy (IEE) techniques have allowed for better detection and management of colorectal cancers and other pathologies. Although there are currently available specialized imaging modality recommendations depending on the type of pathology being investigated, toggling between such imaging modalities during an endoscopy procedure remains highly manual and dependent on a clinician both recognizing a current type of pathology being viewed and recalling which imaging modality is recommended for viewing such pathology. The high operator dependency on the operator preference and experience inreading these advanced endoscopy modes their usage may result in variability in diagnosis and treatment outcome. Although training on advanced endoscopic imaging can potentially improve overall anomaly detection and allow for more accurate optical diagnosis and effective treatment, the training can be costly. The adoption of IEE techniques in colonoscopy is still operator dependent and requires substantial specialized training.

[0007] The inventors of the present disclosure have identified an unmet need of real-time recommendations for personalized, pathology-specific imaging modality for use in inspecting tissue or foreign objects of interest during an endoscopy procedure. This document describes artificial intelligence (Al)-based systems and methods of determining and recommending an imaging modality, such as an IEE mode, based on the suspected anomaly recognized based on realtime computer-performed analysis of endoscopic images or video streams of the target anatomy. An exemplary endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging system to obtain images or video streams of a target anatomy during an endoscopy procedure. The controller circuit can generate from the obtained images or video streams endoscopic image or video features characterizing an anomaly in the target anatomy, and determine a personalized, pathology-specific imaging mode to enhance discernability of the anomaly from the images or video streams. The recommended imaging mode may be determined using machine learning (ML) techniques. The recommended imaging mode may be provided to a user or a process to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

[0008] The systems, devices, and methods described herein may be used in various endoscopy procedures to improve real-time inspection and detection and diagnosis of pathologies. It can also help reduce the inter-operator variability in experience and / or preference and produce more consistent and predictable diagnosis and treatment, while at the same time reduce the cost associated with intensive training. The systems and techniques described herein also promote adoption of advanced IEE technologies, thereby improving the overall endoscopy procedure success rate and patient outcome.

[0009] Example 1 is an endoscope system including: an endoscope, including an imaging system configured to obtain images or video streams of atarget anatomy in a patient during an endoscopy procedure; and a controller circuit configured to: analyze the obtained images or video streams to generate endoscopic image or video features characterizing an anomaly in the target anatomy; based at least in part on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, automatically determine a target or recommended imaging mode of the imaging system to enhance discemability of the anomaly from the images or video streams of the target anatomy in subsequent imaging of the target anatomy; and provide the target or recommended imaging mode to a user or a robotic system to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

[0010] In Example 2, the subject matter of Example 1 optionally includes the target or recommended imaging mode that can include one or more of a lighting modality, a zoom setting, or a viewing angle with respect to the anomaly.

[0011] In Example 3, the subject matter of any one or more of Examples 1-2 optionally includes the target or recommended imaging mode that can include one of a narrow-band imaging (NBI) mode, a red dichromatic imaging (RDI) mode, a white-light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

[0012] In Example 4, the subject matter of any one or more of Examples 1-3 optionally includes a colonoscope, wherein the imaging system is configured to obtain images or video streams of each of distinct colon segments during an colonoscopy procedure, wherein the controller circuit is configured to determine respective target or recommended imaging modes for use in subsequent imaging of the distinct colon segments to enhance anomaly discernability from the images or video streams.

[0013] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes the controller circuit that can be configured to: detect the anomaly in the target anatomy using the endoscopic image or video features; and determine the target or recommended imaging mode based at least in part on a result of the anomaly detection.

[0014] In Example 6, the subject matter of Example 5 optionally includes, wherein to detect the anomaly includes to detect a presence or absenceof the anomaly and one or more of a type, a size, a shape, a location, or an amount of pathological tissue or obstructed mucosa in the target anatomy.

[0015] In Example 7, the subject matter of any one or more of Examples 5-6 optionally includes the controller circuit that can be configured to detect the anomaly using a first trained machine-learning (ML) model.

[0016] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes the controller circuit that can be configured to: determine in substantially real time a location of the endoscope in the target anatomy based at least in part on the endoscopic image or video features; register the location of the endoscope to a pre-generated template of the target anatomy; and display the location of the endoscope in the target anatomy on a user interface.

[0017] In Example 9, the subject matter of Example 8 optionally includes the controller circuit that can be configured to recognize an anatomical landmark using the endoscopic image or video features, and to determine the location of the endoscope based on the recognized anatomical landmark.

[0018] In Example 10, the subject matter of any one or more of Examples 8-9 optionally includes the controller circuit that can be configured to determine the target or recommended imaging mode further using the determined location of the endoscope in the target anatomy.

[0019] In Example 11, the subject matter of any one or more of Examples 1-10 optionally includes, wherein to determine the target or recommended imaging mode, the controller circuit is configured to apply the endoscopic image or video features to a second trained machine-learning (ML) model, the second trained ML model being trained to established a correspondence between the endoscopic image or video features to one of a group of candidate imaging modes.

[0020] In Example 12, the subject matter of Example 11 optionally includes the second trained ML model that can be trained further using the one or more auxiliary information features including one or more of: a profile of an endoscopist performing the endoscopy procedure; patient information and medical history data of the patient; endoscope system setup information; or preprocedure imaging study data, wherein to determine the target or recommended imaging mode, the controller circuit is configured to apply an augmented inputcomprising the endoscopic image or video features and an auxiliary input to the second trained ML model.

[0021] In Example 13, the subject matter of any one or more of Examples 11-12 optionally includes the second trained ML model that can be trained to predict, for each of a plurality of candidate imaging modes, a probability representing a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode, wherein the controller circuit is configured to determine the target or recommended imaging mode based at least in part on the probabilities corresponding to the candidate imaging modes.

[0022] In Example 14, the subject matter of any one or more of Examples 1-13 optionally includes the controller circuit that can be configured to provide a recommendation to the user to switch to the target or recommended imaging mode to re-capture images or video streams of the target anatomy in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy.

[0023] In Example 15, the subject matter of any one or more of Examples 1-14 optionally includes the controller circuit that can be configured to generate a control signal to the imaging system to automatically switch to the target or recommended imaging mode and to re-capture images or video streams of the target anatomy in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy.

[0024] Example 16 is a method of determining or adjusting an endoscopic imaging mode during an endoscopy procedure, the method including steps of: obtaining images or video streams of a target anatomy during an endoscopy procedure using an imaging system associated with an the endoscope; analyzing the obtained images or video streams to generate endoscopic image or video features characterizing an anomaly in the target anatomy; based at least in part on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, automatically determining a target or recommended imaging mode of the imaging system to enhance discernability of the anomaly from the images or video streams of thetarget anatomy in subsequent imaging of the target anatomy; and providing the target or recommended imaging mode to a user or a robotic system to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

[0025] In Example 17, the subject matter of Example 16 optionally includes the target or recommended imaging mode that can include one or more of a lighting modality, a zoom setting, or a viewing angle with respect to the anomaly.

[0026] In Example 18, the subject matter of any one or more of Examples 16-17 optionally includes the target or recommended imaging mode that can include one of a narrow-band imaging (NBI) mode, a red dichromatic imaging (RDI) mode, a white-light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

[0027] In Example 19, the subject matter of any one or more of Examples 16-18 optionally includes detecting the anomaly in the target anatomy using the endoscopic image or video features; and determining the target or recommended imaging mode based at least in part on a result of the anomaly detection.

[0028] In Example 20, the subject matter of any one or more of Examples 16-19 optionally includes recognize an anatomical landmark using the endoscopic image or video features; determine in substantially real time a location of the endoscope in the target anatomy based at least in part on the recognized anatomical landmark; registering the location of the endoscope to a pre-generated template of the target anatomy; and displaying the location of the endoscope in the target anatomy on a user interface, wherein determining the target or recommended imaging mode is further based on the determined location of the endoscope in the target anatomy.

[0029] In Example 21, the subject matter of any one or more of Examples 16-20 optionally includes determining the target or recommended imaging mode that can include applying the endoscopic image or video features to a second trained machine-learning (ML) model, the second trained ML model being trained to established a correspondence between the endoscopic image or video features to one of a group of candidate imaging modes.

[0030] In Example 22, the subject matter of Example 21 optionally includes the second trained ML model that can be trained further using the one or more auxiliary features information including one or more of: a profile of an endoscopist performing the endoscopy procedure; patient information and medical history data of the patient; endoscope system setup information; or preprocedure imaging study data, wherein determining the target or recommended imaging mode includes applying an augmented input comprising the endoscopic image or video features and an auxiliary input to the second trained ML model.

[0031] In Example 23, the subject matter of Example 22 optionally includes the second trained ML model that can be trained to predict, for each of a plurality of candidate imaging modes, a probability representing a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode, wherein determining the target or recommended imaging mode includes identifying one of the plurality of candidate imaging modes with the corresponding probability higher than any other of the candidate imaging modes.

[0032] In Example 24, the subject matter of any one or more of Examples 16-23 optionally includes, in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy, providing a recommendation to the user to switch to the target or recommended imaging mode to re-capture images or video streams of the target anatomy.

[0033] In Example 25, the subject matter of any one or more of Examples 16-24 optionally includes, in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy, generating a control signal to the imaging system to automatically switch to the target or recommended imaging mode and to re-capture images or video streams of the target anatomy.

[0034] The presented techniques are described in terms of controlled withdrawal of endoscope in colonoscopy but are not so limited. The systems, devices, and techniques as described in accordance with various embodiments in this document, may additionally or alternatively be used in other procedures involving different types of endoscopes, including, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, colposcopy, cystoscopy,esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy etc.

[0035] This summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. Other aspects of the disclosure will be apparent to persons skilled in the art upon reading and understanding the following detailed description and viewing the drawings that form a part thereof, each of which are not to be taken in a limiting sense. The scope of the present disclosure is defined by the appended claims and their legal equivalents.BRIEF DESCRIPTION OF THE DRAWING

[0036] FIGS. 1-2 are schematic diagrams illustrating an example of an endoscope system for use in endoscopy procedures, procedures, such as a colonoscopy procedure.

[0037] FIG. 3 illustrates an example of an endoscope system configured to determine a target or recommended imaging mode to be used for inspecting tissue or foreign objects of interest during an endoscopy procedure.

[0038] FIG. 4 illustrates examples of auxiliary input that may be used in determining a target or recommended imaging mode.

[0039] FIG. 5 illustrates by way of example different stages of an image- guided colonoscopy and automatic imaging modality switching based on characteristics of the anomaly.

[0040] FIG. 6 illustrates example images of a portion of colon segment under different imaging modes.

[0041] FIGS. 7A-7B illustrate examples of training a machine learning (ML) model and using the trained ML model to determine a recommended pathology-specific imaging modality for anomaly inspection and diagnosis.

[0042] FIG. 8 is a flow chart illustrating an example method of determining or adjusting an endoscopic imaging mode during an endoscopy procedure.

[0043] FIG. 9 is a block diagram illustrating an example machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform.DETAILED DESCRIPTION

[0044] This document describes systems, devices, and methods for determining or adjusting an endoscopic imaging modality in an endoscopy procedure. An exemplary endoscope system includes an endoscope and a controller circuit. The endoscope includes at least one imaging sensor to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure. The controller circuit can perform real-time analysis of endoscopic images or video streams and generate endoscopic image or video features characterizing an anomaly in the target anatomy. Based on the endoscopic image or video features, the controller can determine a target or recommended pathology-specific imaging mode to enhance discemability of the anomaly from the images or video streams of the target anatomy. A machine learning (ML) model may be trained and used for determining the target or recommended imaging mode. The target or recommended imaging mode may be provided to a user or a process to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

[0045] FIG. 1 is a schematic diagram of an endoscope system 10 for use in an endoscopy procedure, such as colonoscopy. The system 10 can include an imaging and control system 12 and an endoscope 14. The system 10 is an illustrative example of an endoscope system suitable for use with the systems, devices, and methods described herein, such as a colonoscopy system for use in image-guided colonoscopy with auto-adjusted imaging modality as described in this document.

[0046] The endoscope 14 can be insertable into an anatomical region for imaging or to provide passage of or attachment to (e.g., via tethering) one or more sampling devices for biopsies or therapeutic devices for treatment of a disease state associated with the anatomical region. The endoscope 14 can interface with and connect to imaging and control system 12. The endoscope 14 may be a colonoscope, though other types of endoscopes can be used with the features and teachings of the present disclosure. The imaging and control system12 can include a control unit 16, an output unit 18, an input unit 20, a light source unit 22, a fluid source 24, and a suction pump 26.

[0047] The imaging and control system 12 can include various ports for coupling with the endoscope system 10. For example, the control unit 16 can include a data input / output port for receiving data from and communicating data to the endoscope 14. The light source unit 22 can include an output port for transmitting light to the endoscope 14, such as via a fiber optic link. The fluid source 24 can include a port for transmitting fluid to the endoscope 14. The fluid source 24 can include, for example, a pump and a tank of fluid or can be connected to an external tank, vessel, or storage unit. The suction pump 26 can include a port to draw a vacuum from the endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and the input unit 20 can be used by an operator of the endoscope system 10 to control functions of the endoscope system 10 and view the output of the endoscope 14. The control unit 16 can also generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical output, acoustic output, fluid output, and the like for treating the anatomical region with, for example, cauterizing, cutting, freezing, and the like.

[0048] The fluid source 24 can be in communication with control unit 16 and can include one or more sources of air, saline, or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels, or the like) and connectors (barb fittings, fluid seals, valves, or the like). The fluid source 24 can be utilized as an activation energy for a biasing device or a pressure-applying device of the present disclosure. The imaging and control system 12 can also include the drive unit 46, which can include a motorized drive for advancing a distal section of endoscope 14.

[0049] The endoscope 14 can include an insertion section 28, a functional section 30, and a handle section 32, which can be coupled to a cable section 34 and a coupler section 36. The insertion section 28 can extend distally from the handle section 32, and the cable section 34 can extend proximally from the handle section 32. The insertion section 28 can be elongated and can include a bending section and a distal end to which the functional section 30 can be attached. The bending section can be controllable (e.g., by a control knob 38 onthe handle section 32) to maneuver the distal end through tortuous anatomical passageways (e.g., stomach, duodenum, kidney, ureter, etc.). The insertion section 28 can also include one or more working channels (e.g., an internal lumen) that can be elongated and can support the insertion of one or more therapeutic tools of the functional section 30. The working channel can extend between the handle section 32 and the functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires, can also be provided by the insertion section 28 (e.g., via suction or irrigation passageways or the like).

[0050] A coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the input unit 20, the light source unit 22, the fluid source 24, and the suction pump 26.

[0051] The handle section 32 can include the knob 38 and the port 40A. The knob 38 can be connected to a pull wire or other actuation mechanisms that can extend through the insertion section 28. The port 40 A, as well as other ports, such as a port 40B (FIG. 2), can be configured to couple various electrical cables, guide wires, auxiliary scopes, tissue collection devices, fluid tubes, and the like to the handle section 32, such as for coupling with the insertion section 28.

[0052] According to examples, the imaging and control system 12 can be provided on a mobile platform (e.g., a cart 41) with shelves for housing the light source unit 22, the suction pump 26, an image processing unit 42 (FIG. 2), etc. Alternatively, several components of the imaging and the control system 12 (shown in FIGS. 1 and 2) can be provided directly on the endoscope 14 to make the endoscope “self-contained.”

[0053] The functional section 30 can include components for treating and diagnosing anatomy of a patient. The functional section 30 can include an imaging device, an illumination device, and an elevator. The functional section 30 can further include optically enhanced biological matter and tissue collection and retrieval devices. For example, the functional section 30 can include one or more electrodes conductively connected to the handle section 32 and functionally connected to the imaging and control system 12 to analyze biological matter in contact with the electrodes based on comparative biologicaldata stored in the imaging and control system 12. In other examples, the functional section 30 can directly incorporate tissue collectors.

[0054] In some examples, the endoscope 14 can be robotically controlled, such as by a robot arm attached thereto. The robot arm can automatically, or semi-automatically (e.g., with certain user manual control or commands), via an actuator, position and navigate the endoscope 14 (e.g., the functional section 30 and / or the insertion section 28) in the target anatomy or position a device at a desired location with desired posture to facilitate an operation of an anatomical target. In accordance with various examples discussed in this document, a controller can generate a control signal to the actuator of the robot arm to facilitate anomaly inspection and diagnosis under the target or recommended imaging mode in a robotically assisted endoscopy procedure.

[0055] FIG. 2 is a schematic diagram of the endoscope system 10 of FIG. 1 including the imaging and control system 12 and the endoscope 14. FIG. 2 schematically illustrates components of the imaging and the control system 12 coupled to the endoscope 14, which in the illustrated example includes a colonoscope. The imaging and control system 12 can include the control unit 16, which can include or be coupled to an image processing unit 42, a treatment generator 44, and a drive unit 46, as well as the light source unit 22, the input unit 20, and the output unit 18. The control unit 16 can include, or can be in communication with, an endoscope, a surgical instrument 48, and an endoscope system, which can include a device configured to engage tissue and collect and store a portion of that tissue and through which imaging equipment (e.g., a camera) can view target tissue via inclusion of optically enhanced materials and components. The control unit 16 can be configured to activate a camera to view target tissue distal of the endoscope system. Likewise, the control unit 16 can be configured to activate the light source unit 22 to shine light on the surgical instrument 48, which can include select components configured to reflect light in a particular manner, such as enhanced tissue cutters with reflective particles.

[0056] The coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the image processing unit 42 and the treatment generator 44. In examples, the port 40A can be used to insert another surgical instrument 48 or device, such as adaughter scope or auxiliary scope, into the endoscope 14. Such instruments and devices can be independently connected to the control unit 16 via the cable 47. In examples, the port 40B can be used to connect coupler section 36 to various inputs and outputs, such as video, air, light, and electric.

[0057] The image processing unit 42 and light source unit 22 can each interface with the endoscope 14 (e.g., at the functional section 30) by wired or wireless electrical connections. The imaging and control system 12 can accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 can include the light source unit 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). The imaging and control system 12 can connect (e.g., via an endoscope connector) to the endoscope 14 for signal transmission (e.g., light output from light source, video signals from imaging system in the distal end, diagnostic and sensor signals from a diagnostic device, and the like).

[0058] The treatment generator 44 can generate a treatment plan, which can be used by the control unit 16 to control the operation of the endoscope 14, or to provide with the operating physician a guidance for maneuvering the endoscope 14, during an endoscopy procedure. In an example, the treatment generator 44 can generate an endoscope navigation plan, including estimated values for one or more cannulation or navigation parameters (e.g., an angle, a force, etc.) for maneuvering the steerable elongate instrument, using patient information including an image of the target anatomy. The endoscope navigation plan can help guide the operating physician to cannulate and navigate the endoscope in the patient anatomy. The endoscope navigation plan may additionally or alternatively be used to robotically adjust the position, angle, force, and / or navigation of the endoscope or other instrument.

[0059] FIG. 3 is a block diagram illustrating an example of an endoscope system 300 that can determine a target or recommended imaging mode to be used for inspecting tissue or foreign objects of interest during an endoscopy procedure. The target or recommended imaging mode can help improve realtime inspection, recognition, and optical diagnosis of various pathologies. Byway of example and not limitation, the system 300 may be used in a colonoscopy procedure to better detect and manage anomalies such as polyps or colorectal cancers. The system 300 may be implemented as a part of the control unit 16 in FIG. 1.

[0060] The system 300 may include one or more of an endoscope 310, an auxiliary input 315, a controller circuit 320, a user interface 330, and a storage device 340. In some examples, the system 300 may further include or be communicatively coupled to a robotic system 350 in a robotically assisted endoscopy procedure.

[0061] The endoscope 310 can be an example of the endoscope 14 as described above and shown in FIGS. 1-2. The endoscope 310 may include, among other things, an imaging system 312 and a lighting system 314. The imaging system 312 can include at least one imaging sensor or device (e.g., a camera) configured to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure. The imaging sensor or device may be located at a distal portion or a distal end of the endoscope 310. The lighting system 314 may include one or more light sources to produce illumination on the target anatomy via one or more lighting lenses.

[0062] The imaging system 312 may be controllably adjusted to operate on different settings including zoom settings, contrast settings, exposure levels, or viewing angles toward the target anatomy. The lighting system 314 may be controllably adjusted to provide different lighting or illumination conditions. The imaging system 312 and the lighting system 314 together can define an imaging mode or modality for capturing endoscopic images or video streams of the target anatomy. In this document, the imaging mode or modality may include one or more of a lighting modality, an optical magnification, or a viewing angle of the imaging device. Examples of imaging or lighting modality may include high- definition white light imaging (WLI), chromoendoscopy techniques like dyebased, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others. The optical magnification defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy). The viewing angle of the imaging device, also referred to as a field of view, describes the angular extent of a given scene that is imaged by the imaging device.

[0063] The controller circuit 320 may include circuit sets comprising one or more other circuits or sub-circuits that may, alone or in combination, perform the functions, methods, or techniques described herein. In an example, the controller circuit 320 and the circuits sets therein may be implemented as a part of a microprocessor circuit, which may be a dedicated processor such as a digital signal processor, application specific integrated circuit (ASIC), microprocessor, or other type of processor for processing information including physical activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that may receive and execute a set of instructions of performing the functions, methods, or techniques described herein. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0064] The controller circuit 320 can determine a personalized, pathology-specific imaging modality in substantially real time during an endoscopy procedure using at least the endoscopic images or video streams or features extracted therefrom. As described above, imaging modality plays an important role in determining image or video quality during endoscopy and therefore the accuracy and efficiency of optical detection, diagnosis, andtreatment of pathologies of various types. Conventionally, determining a desired or “optimal” imaging modality best suited to detect or diagnose certain pathologies is a highly manual process. It generally requires the user (e.g., endoscopist) to manually toggle among the available imaging modalities, recognize a current type of pathology being viewed, and recall which imaging modality was used for viewing such pathology. Based on the visual inspection, the user manually adjusts the imaging system or the lighting system to achieve desired lighting modality, optical magnification, or viewing angle. Such manual process takes time and can be laborious and may introduce unpredictable interuser variations. The system 300 as described herein can automate the imaging modality selection and optimization process using artificial intelligence (Al) or machine learning (ML) based techniques, as described in further detail below.

[0065] The controller circuit 320 may include one or more of an image processor 321, an anomaly detector circuit 322, an endoscope localization circuit 323, and an image mode selector circuit 324. The image processor 321 can analyze the images or video streams obtained from the imaging system 312 and generate endoscopic image or video features. Examples of the image or video features include statistical features of pixel values or morphological features, such as comers, edges, blobs, curvatures, speeded up robust features (SURF), or scale-invariant feature transform (SIFT) features, among others. In some examples, the image processor 321 may pre-process the images or video streams, such as filtering, resizing, orienting, or color or grayscale correction, and the endoscopic features may be extracted from the pre-processed images or video streams. In some examples, the image processor 321 may post-process the image features to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0066] The anomaly detector circuit 322 may detect anomaly at the target anatomy based at least in part on the endoscopic mage or video features. The anomaly detection includes detecting and / or recognizing one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the anatomy. In an example of colonoscopy, the anomaly being detected may include pathological tissue segments, such as mucosal abnormalities (polyps, inflammatory bowel diseases, Meckel’s diverticulum,lipoma, bleeding, vascularized mucosa etc.), or obstructed mucosa (e.g., segments with bad bowel preparation, distended colon etc.).

[0067] In an example, the anomaly detector circuit 322 may detect the anomaly using a template matching technique, in which an anomaly may be recognized based on a comparison of the endoscopic features (e.g., features characterizing shapes or contours of a structure) to one or more pre-generated templates of known anomalous structure. In another example, the anomaly detector circuit 322 may detect the anomaly using Al or ML based techniques, where the endoscopic images or video streams or features extracted therefrom may be applied to a trained ML model to automatically recognize presence or absence, type, size, location, and / or other characteristics of the anomaly. The ML model may be trained to establish a correspondence between an endoscopic image or video stream or features extracted therefrom and one or more anomaly characteristics. Examples of the ML model used for recognizing anomaly from endoscopic images or video streams include Convolutional Neural Networks, bidirectional LSTM, Recurrent Neural Networks, Conditional Random Fields, Dictionary Learning, or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering), among other ML techniques. The trained ML model may be stored in the storage device 340.

[0068] The endoscope localization circuit 323 may determine in substantially real time a location of the endoscope in a particular segment of the anatomy. With such real-time endoscope location information, the anomaly detector circuit 322 may associate a detected anomaly with the segment of the anatomy. The endoscope location may be determined using electromagnetic tracking, or using anatomical landmark detected from the images or video streams such as obtained during the insertion phase of endoscopy, or image or video stream features extracted therefrom by the image processor 321. In an example of colonoscopy, the endoscope localization circuit 323 may recognize from the images or video streams colon landmarks such as anus, left ascending colon, splenic flexure, transverse colon, hepatic flexure, right descending colon, cecum, appendix and terminal ileum etc. In an example, the endoscope localization circuit 323 may recognize the landmarks using a template matching algorithm. In some examples, the endoscope localization circuit 323 may recognize the landmarks using an Al or ML approach, such as a trained MLmodel that has been trained to establish a correspondence between an endoscopic image or video stream or features extracted therefrom and a landmark recognition. Examples of the ML model used for recognizing an anatomical landmark include Deep Belief Network, ResNet, DenseNet, Autoencoders, capsule networks, generative adversarial networks, Siamese networks, Convolutional Neural Networks (CNN), deep reinforcement learning, support vector machine (SVM), Bayesian models, decision trees, k-means clustering, among other ML models. The trained ML model may be stored in a storage device 340. Once the endoscope location is determined, the endoscope localization circuit 323 may register the substantially real-time endoscope location to a pre-generated template of the anatomy. Information of the endoscope location in the segments of an anatomy may be presented to the user on the user interface 330.

[0069] The imaging mode selector circuit 324 may determine a personalized, pathology-specific imaging modality using at least the information about the detected anomaly (as produced by the anomaly detector circuit 322) and the substantially real-time location of the endoscope when the anomaly is detected (as produced by the endoscope localization circuit 323). The personalized pathology-specific imaging modality may include a lighting modality, an optical magnification (i.e., a zoom setting of the imaging system 312), or a viewing angle of the imaging system 312. Examples of lighting modality may include high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others. The optical magnification defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy). The viewing angle of the imaging device, also referred to as a field of view, describes the angular extent of a given scene that is imaged by the imaging device.

[0070] In an example, the personalized, pathology-specific imaging modality may be determined using Al or ML based techiques. In an example, information about the detected anomaly, the landmark, and the substantially realtime endoscope location may be applied to trained ML model(s) 360. The ML model(s) 360 may be trained to establish a correspondence between an anomaly characteristic and a target or recommended imaging modality. The trained MLmodel(s) 360, once trained, may be stored in a storage device 340. Examples of training an ML model and using the trained ML model to determine a personalized pathology-specific imaging modality are discussed below with respect to FIG. 6.

[0071] In some examples, the imaging mode selector circuit 324 may compute an anomaly score based on one or more anomaly characteristics such as a type, a size, a shape, a location, or an amount of the anomaly. The anomaly score may have numerical values, such as within a range of 0 to 10. In some examples, a composite anomaly score may be generated such as using a linear combination or a non-linear combination of multiple anomaly scores each quantifying an anomaly characteristic. The imaging mode selector circuit 324 may map the anomaly score to an imaging mode (e.g., one or more of a lighting modality, an optical magnification, or a viewing angle of the imaging system 312) based on a comparison to one or more threshold values or value ranges, such that a higher anomaly score, which typically indicates a more severe anomalous condition, can be mapped to a more enhanced lighting modality or a higher magnification. An established correspondence between anomaly scores or ranges of anomaly scores and corresponding imaging mode may be determined by a lookup table to a normative database, or via a rule-based system. The established mapping may be stored in the storage device 340. In another example, the imaging mode selector circuit 324 may apply the anomaly score to the trained ML model(s) 360 to determine the personalized pathology-specific imaging modality. In some examples, the anomaly score calculation may be included into the ML model, such as in a layer of the neural network model.

[0072] In some examples, the imaging mode selector circuit 324 may determine the personalized pathology-specific imaging modality further using an auxiliary input 315. Referring to FIG. 4, the auxiliary input 315 may include, by way of example and not limitation, image or video streams and clinical data from previous endoscopy procedures 410 performed on the patient, patient information and medical history 420, or pre-procedure imaging study data 430 (e.g., X-ray or fluoroscopy images, electrical potential map or an electrical impedance map, computer tomography (CT) images, magnetic resonance imaging (MRI) images, among other imaging modalities). In an example of colonoscopy, the auxiliary input may include previous colonoscopies insurveillance cases, including those polyps that the endoscopist left in situ or areas of the colon which were operated on. The patient information and medical history 420 may include, in a typical endoscopy scenario, clinical demographic information, past and current indications and treatment received, etc.

[0073] In some examples, the auxiliary input 315 may additionally or alternatively include user (e.g., endoscopist) profile 440 including user’s experience, working environment (e.g., a hospital setting or an ambulatory screening centre), affinity to new technology, preference of a certain endoscopy protocol or certain imaging modes. The user profile 440 may be provided by the user via the user interface 330. Alternatively, the user profile 440 may be automatically generated or updated through learning from the user’s past choices and training.

[0074] In examples, the auxiliary input 315 may additionally or alternatively include endoscope and equipment information 450. This may include, for example, specification of the endoscope 310 including type, size, dimension, shape, and structures of the endoscope or other steerable instruments such as a cannular, a catheter, or a guidewire supporting imaging modes and lighting modes (e.g., NBI, RDI, WLI, TXI, etc.); specification of size, dimension, shape, and structures of tissue section, sampling, or treatment tools; current state of the equipment, including which light mode is on, which endo buttons are engaged like waterjet, insufflation, what light modes are supported, or whether the magnification is turned on and the current magnification selection, etc.

[0075] If there are additional Al or ML algorithms running in the background, the states of these Al or ML algorithms, collectively referred to hereinafter as “Al finding” 460, can be additional elements of the auxiliary input 315 that can be passed to the imaging mode selector circuit 324. Examples of the Al finding 460 may include Al algorithms for detecting anomaly, landmark, or other features or structural elements of interested in the target anatomy.

[0076] Information about the personalized pathology-specific imaging modality as determined by the imaging mode selector circuit 324 may be displayed on the user interface 330 to assist the endoscopist during an endoscopy procedure, such as during image-guided colonoscopy. If the determined imaging modality is not the same as the current imaging modality being used forinspecting and detecting the anomaly, the user (e.g., endoscopist) may be notified or warned of such difference via the user interface 330. The notification may be delivered through optical means on the diagnostic monitor, or via auditory means like a warning alarm. In an example, highlighting, flash alerts, audible or haptic feedback may be provided to the user. Examples of automatically switching the pathology-specific imaging modality are discussed below with reference to FIG. 5. Other information, including the endoscopic images and image features, information about the detected anomaly and landmarks in each segment of the anatomy, or substantially real-time location of the endoscope during the endoscopy procedure, may also be displayed on the user interface 330. At the completion of withdrawal (e.g., rectum detected), information about the anomaly detected and treated, and the total net withdrawal time, among other information, may be generated in a post-procedure summary. Such post-procedure analytics may be used for quality assurance and for determining if the colonoscopy procedure was successful.

[0077] In some example, the personalized pathology-specific imaging modality may be stored in the storage device 340. Information about the anomaly detected may also be stored in the storage device 340. The storage device 340 can be local to the endoscope system 300. Alternatively, the storage device 340 can be a remote storage device, such as a part of a cloud comprising one or more storage and computing devices (e.g., servers) that provides secure access to cloud-based services including, for example, data storage, computing services, and provisioning of customer services, among others. In some examples, at least some of the data processing and computation with regard to anomaly detection, landmark recognition, endoscope localization, and imaging mode selection may be performed in a cloud. For example, images or video streams or features extracted therefrom may be streamed to the cloud, get processed therein, and the computation results (e.g., the personalized pathologyspecific imaging modality) may be relayed back to local endoscope system.

[0078] In some examples, the system 300 may be operated in a closed- loop fashion with a feedback loop for continuous learning of the pathologyspecific imaging modality, such as updating the target or recommended imaging modality. The continuous learning may be via explicit endoscopist feedback (e.g., satisfaction with recommendations, a “like” button etc.). Alternatively, thesystem 300 can run in a shadow mode to experienced endoscopists and monitor their withdrawal actions and correct the segment-wise withdrawal plan through reinforcement feedback.

[0079] In some examples, image-guided endoscopy, or a portion therefore such as endoscope withdrawal, may be performed using the robotic system 350. The robotic system 350 can automatically adjust the imaging modality in accordance the recommended pathology-specific imaging modality. The robot system 350 may include a robot arm detachably attached to the endoscope 310. The robot arm can automatically, or semi -automatically (e.g., with certain user manual control or commands), via an actuator, position and manipulate the endoscope 310 in an anatomical target, or position a device at a desired location with desired posture to facilitate an operation on the anatomical target.

[0080] FIG. 5 is a diagram illustrating by way of example different stages of an image-guided colonoscopy, and automatic imaging modality switching based on characteristics of the anomaly detected during an image- guided endoscopy procedure 500, such as a colonoscopy as shown in this example. The procedure includes an endoscope insertion stage, followed by a withdrawal stage. The image-guided endoscopy procedure may be carried out using the system 300. During the insertion stage, endoscopic images or video streams can be obtained respectively for each of a plurality of colon segments, including the rectosigmoid segment, sigmoid segment, descending segment, transverse segment, ascending segment, and cecum, such as using the imaging system 312. Anomaly may be detected automatically using the anomaly detector circuit 322 during the insertion phase, or alternatively, as shown in FIG. 5, detected or confirmed during the withdrawal phase. In an example, at the end of the insertion stage where the endoscope distal tip reaches cecum, the imaging mode selection may be triggered manually or automatically by automatic cecum detection. During withdrawal, a default imaging mode, including a default lighting mode such as white light imaging (WLI) as shown in FIG. 5, may be used. When the endoscope is withdrawn to a first colon segment, an endoscopic image 510 is obtained under the WLI mode, and a first anomaly is detected from the endoscopic image 510. Based at least on image features extracted from the image 510 (optionally along with the auxiliary input 315), the imaging modeselector circuit 324 recommends in substantially real time a pathology-specific imaging modality, including a recommended lighting mode of narrow-band imaging (NBI), such as using the trained ML model 360. Compared to default WLI mode, NBI mode can enhance discernability of the first anomaly from the background of the image 510. Imaging mode can then be automatically switched from WLI to NBI, and the first anomaly is re-inspected and diagnosed under the NBI mode. Alternatively, the user is prompted to switch to NBI mode to reinspect the first anomaly. Image 512 of the first anomaly under NBI mode can be displayed to the user in substantially real time. The imaging mode can then be switched back to the default WLI for inspecting other colon segments. A subsequent inspection reveals a second anomaly detected from endoscopic image 520 of another segment. Based at least on image features extracted from the image 520 (optionally along with the auxiliary input 315), the imaging mode selector circuit 324 recommends in substantially real time a pathology-specific imaging modality, including a recommended lighting mode of red dichromatic imaging (RDI), such as using the trained ML model 360. Compared to default WLI mode, RDI mode can enhance discernability of the second anomaly from the background of the image 520. Imaging mode is then automatically switched from WLI to RDI, and the first anomaly is re-inspected and diagnosed under the RDI mode. Alternatively, the user is prompted to switch to RDI mode to reinspect the second anomaly. Image 522 of the second anomaly under RDI mode can be displayed to the user in substantially real time. After the diagnosis, the imaging mode can be switched back to the default WLI for inspecting other colon segments.

[0081] FIG. 6 illustrates example images of a portion of colon segment under different imaging modes. Image 610 was obtained under a “default” WLI mode. A lesion 615 is detected from the endoscopic image 610. Based at least on the image features extracted from the image 610 (optionally along with the auxiliary input 315), a recommendation to switch to narrow-band imaging (NBI) is generated. Imaging mode can be automatically or manually switched to the NBI model. The lesion 615 is then re-inspected from the image 620 obtained under NBI mode. As illustrated in FIG. 6, compared to the default WLI mode, NBI mode can enhance discemability of the lesion 615 from the background ofthe image 620, allowing for more accurate and efficient real-time inspection and optical diagnosis of the lesion 615.

[0082] FIGS. 7A-7B are diagrams illustrating examples of training an ML model and using the trained ML model to determine a recommended pathology-specific imaging modality for anomaly inspection and diagnosis. FIG. 7A illustrates an ML model training (or learning) phase during which an ML model 720 may be trained to determine a target or recommended pathologyspecific imaging modality based at least in part on endoscopic images of the anomaly and surrounding environment. A training dataset may include a plurality of endoscopic images 710 of the same type of anomaly (e.g., a polyp or lesion in the lining of a colon segment) obtained from colonoscopy procedures performed on a plurality of patients. In some examples, the training data may also include auxiliary input 315 as described above with respect to FIG. 3. Examples of the auxiliary input may include supporting imaging modes and lighting modes 732 (an example or a portion of the endoscope and equipment information 450), endoscopist preference 734 (an example of the user profile 440), clinical information 736 (which may include one or more of the image or video streams and clinical data from previous endoscopy procedures 410, the patient information and medical history 420, or the pre-procedure imaging study data 430), or Al findings 738 (an example of the Al finding 460). Endoscope location information for each of the plurality of endoscopic images 710 may also be included in the training dataset. The ML model 720 may have a neural network structure comprising an input layer, one or more hidden layers, and an output layer. The plurality of endoscopic images 710 or features generated therefrom, along with one or more of the auxiliary inputs 732, 734, 736, or 738, may be fed into the input layer of the ML model 720, which propagates the input data or data features through one or more hidden layers to the output layer that outputs a recommended pathology-specific imaging modality. The ML model 720 is able to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The ML model 720 explores the study and construction of algorithms (e.g., ML algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building the ML model 720 from training data in order to make data- driven predictions or decisions expressed as outputs or assessments.

[0083] The ML model 720 may be trained using supervised learning or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model can implement the same relationships when given inputs to generate the corresponding outputs. Unsupervised learning is the training of an ML algorithm using information that is neither classified nor labelled and allowing the algorithm to act on that information without guidance. Unsupervised learning is useful in exploratory analysis because it can automatically identify structure in data.

[0084] Common tasks for supervised learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values. Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM). Examples of DNN include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), or a hybrid neural network comprising two or more neural network models of different types or different model configurations.Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0085] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machine-learning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing toaddress critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0086] The training of the ML model 720 may be performed continuously or periodically, or in near real time as additional procedure data are made available. The training process involves algorithmically adjusting one or more ML model parameters (e.g., weights or bias at any particular layer of a neural network model), until the ML model being trained satisfies a specified training convergence criterion. By way of example and not limitation, the ML model 720 may be trained with weighted square loss (for explicit feedback) or with binary cross-entropy loss (for implicit feedback). Other training techniques, such as deep factorization machine, wide and deep learning, deep structured semantic models, or autoencoder based recommender systems, may be used. The trained ML mode 720 can establish a correspondence between the endoscopic images 710 (or features extracted therefrom) and the recommended imaging modality and associated probability indicating the likelihood of the recommended imaging modality being selected.

[0087] FIG. 7B illustrates an inference phase during, for example, withdrawal of colonoscope to determine a target or recommended imaging modality to inspect and diagnosis an anomaly suspected under an existing imaging modality. A live endoscopic image 750 of a suspected anomaly (e.g., a polyp or lesion), along with auxiliary input 760 (including, for example, information about the endoscopist profile, patient clinical information, Al findings, equipment setup features, and endoscope location features) may be applied to the trained ML mode 720 to determine output 770 including a recommended imaging modality and associated probability indicating the likelihood of the target or recommended imaging modality being selected. The output 770 may be provided to the endoscopist or a robotic system for manual or robotically assisted adjusting of imaging modality for enhanced anomaly inspection and diagnosis.

[0088] In some examples, the training of the ML model 720 as shown in FIG. 7A may include predicting a probability associated with each of a plurality of supporting imaging modes, such as various lighting modes (e.g., NBI, RDI, WLI, TXI, etc.). Such probability represents a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode. Theprobability may be predicted based on the current system state (including current imaging mode being used to produce the endoscopic image being analyzed) conditioned on various inputs including the image features and various auxiliary input features. For example, for endoscopist v, clinical information w , Al findings i, equipment setup features e, and endoscope location features s may be collectively represented by a composite feature vector x = [v, w, i, e, s]. Let the features of the endoscopic image be represented as u and the imaging or lighting mode as I. The training of ML model 720 involves predicting an “affinity” based on current video stream u and for the given feature vector x, what is the likelihood the endoscopist chooses light source I. Such “affinity” can be represented by a conditional probability rut\xtaking value between 0 and 1. A higher affinity value indicates a higher probability that the imaging or lighting mode I would be selected as the recommended imaging mode. The conditional probability rut\xmay be learnt from prior implicit or explicit user feedback to predict fui\xby learning a recommender system f defined as: fui\x= f(u, I, x, G) where 6 are learnt parameters of the recommender system. In some examples, the recommended system f may be learned using the multi-input collaborative learning process as illustrated in FIG. 5. In the inference phase as shown in FIG. 7B, the trained ML model 720 may predict the conditional probabilities fui\xfor each of a plurality of candidate imaging or lighting modes, and determine the target or recommended imaging mode based at least in part on the probabilities corresponding to the candidate imaging modes. In an example, the target or recommended imaging mode can be selected to be the candidate imaging mode associated with the highest conditional probability rui\x.

[0089] FIG. 8 is a flow chart illustrating an example method 800 of determining or adjusting an endoscopic imaging mode during an endoscopy procedure in a patient, such as a colonoscopy procedure. A target or recommended pathology-specific imaging modality can be determined using at least endoscopic images or video streams of the anatomy. The imaging mode can help improve real-time inspection, recognition, and diagnosis of various pathologies. The method 800 may be implemented in the endoscope system 300. Although the processes of the method 800 are drawn in one flow chart, they arenot required to be performed in a particular order. In various examples, some of the processes can be performed in a different order than that illustrated herein.

[0090] At 810, images or video streams of distinct segments of the target anatomy may be obtained using an imaging system associated with an endoscope. The images or video streams may be acquired when the imaging system is set to one of a plurality of available imaging modes. An imaging mode or modality refers to one or more of a lighting modality, an optical magnification, or a viewing angle of the imaging device. Examples of lighting modality may include high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others. The optical magnification defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy). The viewing angle of the imaging device, also referred to as a field of view, describes the angular extent of a given scene that is imaged by the imaging device.

[0091] At 820, the obtained images or video streams may be analyzed to generate endoscopic image or video features for each of the distinct segments of the anatomy. Examples of the image or video features include statistical features of pixel values or morphological features, such as corners, edges, blobs, curvatures, speeded up robust features (SURF), or scale-invariant feature transform (SIFT) features, among others. In some examples, the segmentspecific images or video streams may be pre-processed, such as filtering, resizing, orienting, or color or grayscale correction, and the endoscopic image features may be extracted from the pre-processed images or video streams. In some examples, the endoscopic image or video features may be post-processed to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0092] At 830, based at least in part on the endoscopic image or video features, a personalized, target or recommended pathology-specific imaging modality may be automatically determined, such as using the imaging mode selector circuit 324 as shown in FIG. 3. In an example, the target or recommended pathology-specific imaging modality may be estimated using information about anomaly detected from any particular segments of the anatomy. Anomaly may be detected from the endoscopic mage or video featuresusing a template matching technique, or an artificial intelligence (Al) or machine learning (ML) based technique. The anomaly detection includes detecting and / or recognizing one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the anatomy. In an example of colonoscopy, the anomaly being detected may include pathological tissue segments, such as mucosal abnormalities (polyps, inflammatory bowel diseases, Meckel’s diverticulum, lipoma, bleeding, vascularized mucosa etc.), or obstructed mucosa (for e.g., segments with bad bowel preparation, distended colon etc.). In some examples, the target or recommended pathology-specific imaging modality may be determined further using substantially real-time location of the endoscope when the anomaly is detected. The endoscope location may be determined using electromagnetic tracking, or anatomical landmark recognized from the images or video streams such as obtained during the insertion phase of endoscopy, or image or video stream features extracted therefrom. In an example of landmarkbased endoscope localization, the landmark may be recognized using an Al or ML approach, such as a trained ML model that has been trained to establish a correspondence between an endoscopic image or video stream or features extracted therefrom and a landmark recognition, as described above with respect to FIG. 3.

[0093] In an example, the target or recommended pathology-specific imaging modality may be estimated further using auxiliary input, including, by way of example and not limitation, image or video streams and clinical data from previous endoscopy procedures performed on the patient, patient information and medical history, or pre-procedure imaging study data, user (e.g., endoscopist) profile (including user’s experience, working environment, affinity to new technology, preference of a certain endoscopy protocol etc.), endoscope and equipment information, or “Al findings” including states of the Al or ML algorithms for detecting anomaly, landmark, or other features or structural elements of interested in the target anatomy, as described above with respect to FIG. 4.

[0094] The target or recommended pathology-specific imaging modality may be estimated using an Al or ML approach. In an example, the result of anomaly detection, the landmark detection and real-time endoscope locationinformation, and the auxiliary input may be applied to a trained ML model. The ML model may have been trained to establish a correspondence between the composite input and a target or recommended value of a target or recommended pathology-specific imaging modality, as described above with respect to FIGS. 7A-7B. In some examples, the training of the ML model may include predicting a probability associated with each of a plurality of supporting imaging modes, such as various lighting modes (e.g., NBI, RDI, WLI, TXI, etc.). Such probability represents a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode. The probability may be predicted based on the current system state (including current imaging mode being used to produce the endoscopic image being analyzed) conditioned on various inputs including the image features and various auxiliary input features. The training of ML model involves predicting an “affinity” based on current video stream u and for the given composite input feature vector (which may include, for example, endoscopist information, patient clinical information, Al findings, equipment setup features, and endoscope location features, etc.), what is the likelihood the endoscopist chooses a particular light source. Such “affinity” can be represented by a conditional probability that may be learnt from prior implicit or explicit user feedback through a multi-input collaborative learning process as illustrated in FIG. 5. In the inference phase, the trained ML model may predict the conditional probabilities for each of a plurality of candidate imaging or lighting modes. The target or recommended imaging mode may be selected to be the candidate imaging mode associated with the highest conditional probability.

[0095] The target or recommended pathology-specific imaging modality determined at 830 may be provided to a user or a process to facilitate manual or robotic withdrawal of the endoscope. At 842, the target or recommended pathology-specific imaging modality may be displayed to the user to assist the endoscopist during an endoscopy procedure, such as during image-guided colonoscopy. If the target or recommended imaging modality is not the same as the current imaging modality being used for inspecting and detecting the anomaly, at 844, an alert may be generated to notify or warn the endoscopist such an effect. The notification or alert may be delivered through optical means on the diagnostic monitor, or via auditory means like a warning alarm. Arecommendation may be provided to the user to switch to the target or recommended imaging modality. Additionally or alternatively, in some examples, at 846, a control signal may be generated to control the imaging system to automatically switch to the target or recommended imaging mode and to re-capture images or video streams of the target anatomy for further diagnosis.

[0096] FIG. 9 illustrates generally a block diagram of an example machine 900 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Portions of this description may apply to the computing framework of various portions of the endoscope system 300.

[0097] In alternative embodiments, the machine 900 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 900 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 900 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 900 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0098] Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership may be flexible over time and underlying hardware variability. Circuit sets include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units,transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively connected to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0099] Machine (e.g., computer system) 900 may include a hardware processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 904 and a static memory 906, some or all of which may communicate with each other via an interlink (e.g., bus) 908. The machine 900 may further include a display unit 910 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) navigation device 914 (e.g., a mouse). In an example, the display unit 910, input device 912 and UI navigation device 914 may be a touch screen display. The machine 900 may additionally include a storage device (e.g., drive unit) 916, a signal generation device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensors. The machine 900 may include an output controller 928, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0100] The storage device 916 may include a machine readable medium 922 on which is stored one or more sets of data structures or instructions 924(e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904, within static memory 906, or within the hardware processor 902 during execution thereof by the machine 900. In an example, one or any combination of the hardware processor 902, the main memory 904, the static memory 906, or the storage device 916 may constitute machine readable media.

[0101] While the machine-readable medium 922 is illustrated as a single medium, the term "machine readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 924.

[0102] The term “machine readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 900 and that cause the machine 900 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Nonlimiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. In an example, a massed machine-readable medium comprises a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine- readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0103] The instructions 924 may further be transmitted or received over a communication network 926 using a transmission medium via the network interface device 920 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, andwireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as WiFi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or phonejacks) or one or more antennas to connect to the communication network 926. In an example, the network interface device 920 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 900, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.Additional Notes

[0104] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0105] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the followingclaims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0106] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

What is claimed is:

1. An endoscope system, comprising: an endoscope, including an imaging system configured to obtain images or video streams of a target anatomy in a patient during an endoscopy procedure; and a controller circuit configured to: analyze the obtained images or video streams to generate endoscopic image or video features characterizing an anomaly in the target anatomy; based at least in part on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, automatically determine a target or recommended imaging mode of the imaging system to enhance discernability of the anomaly from the images or video streams of the target anatomy in subsequent imaging of the target anatomy; and provide the target or recommended imaging mode to a user or a robotic system to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

2. The endoscope system of claim 1, wherein the target or recommended imaging mode includes one or more of a lighting modality, a zoom setting, or a viewing angle with respect to the anomaly.

3. The endoscope system of any of claims 1-2, wherein the target or recommended imaging mode includes one of a narrow-band imaging (NBI) mode, a red dichromatic imaging (RDI) mode, a white-light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

4. The endoscope system of any of claims 1-3, wherein the endoscope is a colonoscope, wherein the imaging system is configured to obtain images or video streams of each of distinct colon segments during an colonoscopy procedure, wherein the controller circuit is configured to determine respective target or recommended imaging modes for use in subsequent imaging of the distinctcolon segments to enhance anomaly discemability from the images or video streams.

5. The endoscope system of any of claims 1-4, wherein the controller circuit is configured to: detect the anomaly in the target anatomy using the endoscopic image or video features; and determine the target or recommended imaging mode based at least in part on a result of the anomaly detection.

6. The endoscope system of claim 5, wherein to detect the anomaly includes to detect a presence or absence of the anomaly and one or more of a type, a size, a shape, a location, or an amount of pathological tissue or obstructed mucosa in the target anatomy.

7. The endoscope system of any of claims 5-6, wherein the controller circuit is configured to detect the anomaly using a first trained machine-learning (ML) model.

8. The endoscope system of any of claims 1-7, wherein the controller circuit is configured to: determine in substantially real time a location of the endoscope in the target anatomy based at least in part on the endoscopic image or video features; register the location of the endoscope to a pre-generated template of the target anatomy; and display the location of the endoscope in the target anatomy on a user interface.

9. The endoscope system of claim 8, wherein the controller circuit is configured to recognize an anatomical landmark using the endoscopic image or video features, and to determine the location of the endoscope based on the recognized anatomical landmark.

10. The endoscope system of any of claims 8-9, wherein the controller circuit is configured to determine the target or recommended imaging mode further using the determined location of the endoscope in the target anatomy.

11. The endoscope system of any of claims 1-10, wherein to determine the target or recommended imaging mode, the controller circuit is configured to apply the endoscopic image or video features to a second trained machinelearning (ML) model, the second trained ML model being trained to established a correspondence between the endoscopic image or video features to one of a group of candidate imaging modes.

12. The endoscope system of claim 11, wherein the second trained ML model is trained further using the one or more auxiliary features including one or more of: a profile of an endoscopist performing the endoscopy procedure; patient information and medical history data of the patient; endoscope system setup information; or pre-procedure imaging study data, wherein to determine the target or recommended imaging mode, the controller circuit is configured to apply an augmented input comprising the endoscopic image or video features and an auxiliary input to the second trained ML model.

13. The endoscope system of any of claims 11-12, wherein the second trained ML model is trained to predict, for each of a plurality of candidate imaging modes, a probability representing a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode, wherein the controller circuit is configured to determine the target or recommended imaging mode based at least in part on the probabilities corresponding to the candidate imaging modes.

14. The endoscope system of any of claims 1-13, wherein, in response to the target or recommended imaging mode being different from an existing imagingmode being used to obtain the images or video streams of the target anatomy, the controller circuit is configured to provide a recommendation to the user to switch to the target or recommended imaging mode to re-capture images or video streams of the target anatomy.

15. The endoscope system of any of claims 1-14, wherein, in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy, the controller circuit is configured to generate a control signal to the imaging system to automatically switch to the target or recommended imaging mode and to recapture images or video streams of the target anatomy.

16. A method of determining or adjusting an endoscopic imaging mode during an endoscopy procedure, the method comprising: obtaining images or video streams of a target anatomy during an endoscopy procedure using an imaging system associated with an the endoscope; analyzing the obtained images or video streams to generate endoscopic image or video features characterizing an anomaly in the target anatomy; based at least in part on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, automatically determining a target or recommended imaging mode of the imaging system to enhance discemability of the anomaly from the images or video streams of the target anatomy in subsequent imaging of the target anatomy; and providing the target or recommended imaging mode to a user or a robotic system to facilitate manual or automatic endoscopic imaging mode adjustment during the endoscopy procedure.

17. The method of claim 16, wherein the target or recommended imaging mode includes one or more of a lighting modality, a zoom setting, or a viewing angle with respect to the anomaly.

18. The method of any of claims 16-17, wherein the target or recommended imaging mode includes one of a narrow-band imaging (NBI) mode, a reddichromatic imaging (RD I) mode, a white-light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

19. The method of any of claims 16-18, further comprising: detecting the anomaly in the target anatomy using the endoscopic image or video features; and determining the target or recommended imaging mode based at least in part on a result of the anomaly detection.

20. The method of any of claims 16-19, further comprising: recognize an anatomical landmark using the endoscopic image or video features; determine in substantially real time a location of the endoscope in the target anatomy based at least in part on the recognized anatomical landmark; registering the location of the endoscope to a pre-generated template of the target anatomy; and displaying the location of the endoscope in the target anatomy on a user interface, wherein determining the target or recommended imaging mode is further based on the determined location of the endoscope in the target anatomy.

21. The method of any of claims 16-20, wherein determining the target or recommended imaging mode includes applying the endoscopic image or video features to a second trained machine-learning (ML) model, the second trained ML model being trained to established a correspondence between the endoscopic image or video features to one of a group of candidate imaging modes.

22. The method of claim 21, wherein the second trained ML model is trained further using the one or more auxiliary features including one or more of: a profile of an endoscopist performing the endoscopy procedure; patient information and medical history data of the patient; endoscope system setup information; or pre-procedure imaging study data,wherein determining the target or recommended imaging mode includes applying an augmented input comprising the endoscopic image or video features and an auxiliary input to the second trained ML model.

23. The method of claim 22, wherein the second trained ML model is trained to predict, for each of a plurality of candidate imaging modes, a probability representing a likelihood of a corresponding imaging mode being selected as the target or recommended imaging mode, wherein determining the target or recommended imaging mode includes identifying one of the plurality of candidate imaging modes with the corresponding probability higher than any other of the candidate imaging modes.

24. The method of any of claims 16-23, wherein, in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy, providing a recommendation to the user to switch to the target or recommended imaging mode to re-capture images or video streams of the target anatomy.

25. The method of any of claims 16-24, wherein, in response to the target or recommended imaging mode being different from an existing imaging mode being used to obtain the images or video streams of the target anatomy, generating a control signal to the imaging system to automatically switch to the target or recommended imaging mode and to re-capture images or video streams of the target anatomy.

Images