Real-time endoscopy scene analysis and photo-documentation

Abstract

Systems, devices, and methods for real-time endoscopy scene analysis and photo-documentation during an endoscopy procedure are disclosed. An exemplary endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging system to obtain images or video streams of a target anatomy during an endoscopy procedure. The controller circuit analyzes the images or video streams to recognize an area of interest (AO I), and generate an endoscope maneuvering plan including an target or recommended imaging setting of the imaging device. Real-time guidance can be provided to a user or a robotic system to assist in maneuvering the endoscope and imaging and photo¬ documenting the AOI in accordance with the endoscope maneuvering plan.

Description

REAL-TIME ENDOSCOPY SCENE ANALYSIS AND PHOTO-DOCUMENTATIONPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 660,283, filed lune 14, 2024, the contents of which are hereby incorporated by reference.FIELD OF THE DISCLOSURE

[0002] The present document relates generally to endoscopy systems, and more particularly to systems and methods of guided maneuverer of an endoscope and imaging setting optimization during an endoscopy procedure based on real-time endoscopy scene analysis.BACKGROUND

[0003] Endoscopes have been used in a variety of clinical procedures, including, for example, illuminating, imaging, detecting and diagnosing one or more disease states, providing fluid delivery (e.g., saline or other preparations via a fluid channel) toward an anatomical region, providing passage (e.g., via a working channel) of one or more therapeutic devices or biological matter collection devices for sampling or treating an anatomical region, and providing suction passageways for collecting fluids (e.g., saline or other preparations), among other procedures. Examples of such anatomical region can include gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary duct, intestines, colon, and the like), renal area (e.g., kidney(s), ureter, bladder, urethra) and other internal organs (e.g., reproductive systems, sinus cavities, submucosal regions, respiratory tract), and the like.

[0004] Some endoscopes include a working channel through which an operator can perform suction, placement of diagnostic or therapeutic devices (e.g., a brush, a biopsy needle or forceps, a stent, a basket, or a balloon), or minimally invasive surgeries such as tissue sampling or removal of unwanted tissue (e.g., benign or malignant strictures) or foreign objects (e.g., calculi). Some endoscopes can be used with a laser or plasma system to deliver energy to an anatomical target (e.g., soft or hard tissue or calculi) to achieve desired treatment. For example, laser has been used in applications of tissue ablation,coagulation, vaporization, fragmentation, and lithotripsy to break down calculi in kidney, gallbladder, ureter, among other stone-forming regions, or to ablate large calculi into smaller fragments. One example of endoscopy is colonoscopy which is typically performed with rapid advance of a colonoscope to the cecum, and an endoscopist can perform thorough inspection to identify any anomalies (e.g., polyps) and to perform necessary treatment (e.g., polypectomy) during the withdrawal of the colonoscope. Colonoscopy has been shown a potential to reduce incidence and mortality rate of colorectal cancer.

[0005] Endoscope systems generally have image or video recording and still photography capabilities which allow manual image capture by the endoscopist. An imaging sensor (e.g., a camera) can be incorporated into an endoscope to take live images or video streams during an endoscopy procedure. The imaging sensor can operate under a preset imaging mode to obtain live images or video streams. The endoscope systems may also allow endoscopists to manually photo-document the scenes or other findings during an endoscopy procedure, which is an important and valuable feature for diagnosis, treatment planning, patient education, and record-keeping. Some endoscope systems can intergrade with Electronic Medical Record (EMR) systems, providing instant storage directly into patient records.SUMMARY

[0006] Advances in endoscopy have allowed for better detection and management of certain types of pathologies or anomalies during an endoscopy procedure. For example, image enhanced endoscopy (IEE) techniques have allowed for better detection and management of colorectal cancers and other pathologies. However, adoption of such advanced endoscopy technologies is still operator dependent and requires specialized training. In an example of IEE- based detection and / or diagnosis of colorectal cancers and other colorectal pathologies, different imaging modes have been used in real-time endoscopic inspection and optical diagnosis, including, for example, high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, and red dichromatic imaging (RDI), etc. These imaging mode may differ in one or more aspects of lighting mode, optical magnification, viewing angle of theimaging sensor, among other settings of the imaging sensor and lighting system. Although it is generalized that proper selection of an imaging mode can improve image or video quality and facilitate inspection and diagnosis of anomalous tissue or foreign objects of interest, toggling between various specialized imaging modalities during an endoscopy procedure remains manual and highly dependent on the operator preference and experience. The high operatordependency may result in variability in image interpretation, anomaly diagnosis, and treatment outcomes. Advanced training on such topics can be costly.

[0007] Modem endoscope systems have incorporated augmented navigation features to guide endoscopists through anatomical structures using visual markers or electromagnetic (EM) field-based endoscope positioning devices. However, existing endoscope systems generally lack capabilities of real-time guidance to assist the user in maneuvering the endoscope and tuning the imaging system to obtain optimal images of a target of interest, and prompting the user to make proper photo-documentation of the target.

[0008] The present disclosure describes endoscope systems with realtime endoscopy scene analysis and photo-documentation during an endoscopy procedure. Based on the scene analysis, the system can identify critical anatomical and pathologic findings of an area of interest (AO I, e.g., an anomaly), generate an endoscope maneuvering plan to guide the user towards capturing an ideal or optimal view with critical details of the AOI, and prompt the user to make proper photo-documentation of the AOI. An exemplary endoscope system includes an endoscope having an imaging device to take images or video streams of a target anatomy during an endoscopy procedure, and a controller circuit to analyze the obtained images or video streams and identify an AOI such as an anatomical landmark or an anomalous structure. Based on the identification of the AOI, the controller circuit an generate an endoscope maneuvering plan including imaging settings of the imaging device for subsequent imaging of the AOI using, for example, artificial intelligence (Al) or machine learning (ML) techniques. Real-time guidance can be provided to a user or a robotic system to assist in maneuvering the endoscope and imaging and photo-documenting the AOI in accordance with the endoscope maneuvering plan.

[0009] The systems, devices, and methods described herein may be used in various endoscopy procedures to improve real-time inspection and detection and diagnosis of pathologies, such as mucosal inspection and documentation in a colonoscopy procedure. Various embodiments as described in this document may help reduce inter-operator variability in experience and / or preference and produce more consistent and predictable diagnosis and treatment, while at the same time reduce the cost of advanced training. The systems and techniques described herein also promote adoption of advanced endoscopy technologies, thereby improving the overall endoscopy procedure success rate and patient outcome.

[0010] Example 1 is an endoscope system, comprising: an endoscope, including an imaging device to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure; and a controller circuit configured to: analyze the obtained images or video streams to identify an area of interest (AOI) in the target anatomy during the endoscopy procedure; based on the identification of the AOI, generate an endoscope maneuvering plan including a target or recommended imaging setting of the imaging device for subsequent imaging of the AOI; and generate real-time guidance, to a user or a robotic system, in maneuvering of the endoscope and imaging and photodocumenting the AOI in accordance with the generated endoscope maneuvering plan.

[0011] In Example 2, the subject matter of Example 1 optionally includes the target or recommended imaging setting of the imaging device that can include at least one of: a position, a posture, or a viewing angle of the imaging device with respect to the identified AOI; a zoom setting; a focus setting; a contrast setting; a viewing angle; a lighting parameter; or a centering setting to center the identified AOI or a portion thereof in a field of view (FOV) of the imaging device.

[0012] In Example 3, the subject matter of any one or more of Examples 1-2 optionally includes the endoscope maneuvering plan that can include guided navigation instructions or paths towards the AOI, wherein the real-time guidance is to guide the user or the robotic system to maneuver the endoscope in accordance with the guided navigation instructions or path.

[0013] In Example 4, the subject matter of Example 3 optionally includes the guided navigation instructions or path defined in a coordinate system.

[0014] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes, wherein to generate the target or recommended imaging setting, the controller circuit is configured to generate an image quality indicator of the images or video streams produced under an existing imaging setting of the imaging device, and to determine whether to keep or adjust the existing imaging setting based at least in part on the image quality indicator.

[0015] In Example 6, the subject matter of any one or more of Examples 1-5 optionally includes the AOI that can include a pre-determined anatomical landmark in the target anatomy.

[0016] In Example 7, the subject matter of Example 6 optionally includes the endoscope that can include a colonoscope, and the imaging device that can be configured to obtain images or video streams of one or more colon segments during withdrawal of the colonoscope in a colonoscopy procedure; and the controller circuit is configured to analyze the obtained images or video streams of the one or more colon segments to identify one or more predetermined colon landmarks during the colonoscopy procedure.

[0017] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes the AOI that can include an anomalous structure in the target anatomy, wherein to identify the AOI includes to identify a presence or absence, a type, a size, a shape, a location, or an amount of lesion or obstructed mucosa in the target anatomy.

[0018] In Example 9, the subject matter of any one or more of Examples 1-8 optionally includes the AOI that can include a blind spot region of the target anatomy, wherein the controller circuit is further configured to generate a three- dimensional (3D) reconstruction of the blind spot region using images or video streams produced in accordance with the target or recommended imaging setting.

[0019] In Example 10, the subject matter of any one or more of Examples 1-9 optionally includes a device tracking circuit configured to track location and orientation of the imaging device or a distal portion of the endoscope during the endoscopy procedure, wherein the controller circuit isconfigured to identify the AOI or to generate the endoscope maneuvering plan further based on the tracked location or orientation.

[0020] In Example 11, the subject matter of Example 10 optionally includes the device tracking circuit that can be configured to track the location and orientation of the imaging device, or the distal portion of the endoscope based on an electromagnetic property thereof detected during the endoscopy procedure.

[0021] In Example 12, the subject matter of any one or more of Examples 10-11 optionally includes, wherein to perform location tracking, the device tracking circuit is configured to register the tracked location and orientation to a pre-generated template of the target anatomy during the endoscopy procedure.

[0022] In Example 13, the subject matter of any one or more of Examples 1-12 optionally includes the controller circuit that can be configured to identify the AOI using a first trained machine-learning (ML) model, the first trained ML model trained to establish a correspondence between (i) an endoscopic image or video stream of the target anatomy or features extracted therefrom and (ii) one or more characteristics of an AOI.

[0023] In Example 14, the subject matter of any one or more of Examples 1-13 optionally include the controller circuit that can be configured to generate the endoscope maneuvering plan including the target or recommended imaging setting using a second trained machine-learning (ML) model, the second trained ML model trained to establish a correspondence between (i) an endoscopic image or video stream or the identified AOI or features extracted therefrom and (ii) an imaging setting of the imaging device.

[0024] In Example 15, the subject matter of any one or more of Examples 1-14 optionally includes the controller circuit that can be further configured to generate an alert to the user about the identified AOI, and to receive a user feedback about the alert, wherein the controller circuit is configured to generate the endoscope maneuvering plan based at least in part on the user feedback about the alert.

[0025] In Example 16, the subject matter of any one or more of Examples 1-15 optionally includes the robotic system configured to, in response to a control signal from the controller circuit, robotically maneuver theendoscope in accordance with the endoscope maneuvering plan, and to produce guided imaging and documentation of the AOI in accordance with the target or recommended imaging setting.

[0026] Example 17 is a method of real-time endoscopy scene analysis and photo-documentation during an endoscopy procedure. The method comprises steps of: obtaining images or video streams of a target anatomy during an endoscopy procedure using an imaging device associated with an the endoscope; analyzing the obtained images or video streams to identify an area of interest (AOI) in the target anatomy during the endoscopy procedure; based on the identification of the AOI, generating an endoscope maneuvering plan including a target or recommended imaging setting of the imaging device for subsequent imaging of the AOI; and generating real-time guidance, to a user or a robotic system, in maneuvering of the endoscope and imaging and photodocumentation of the AOI in accordance with the generated endoscope maneuvering plan.

[0027] In Example 18, the subject matter of Example 17 optionally includes the endoscope maneuvering plan that can include guided navigation instructions or paths towards the AOI, wherein the real-time guidance is to guide the user or the robotic system to maneuver the endoscope in accordance with the guided navigation instructions or path.

[0028] In Example 19, the subject matter of Example 18 optionally includes thee guided navigation instructions or path defined in a coordinate system.

[0029] In Example 20, the subject matter of any one or more of Examples 17-19 optionally includes generating an image quality indicator of the images or video streams produced under an existing imaging setting of the imaging device, and determining whether to keep or adjust the existing imaging setting based at least in part on the image quality indicator.

[0030] In Example 21, the subject matter of any one or more of Examples 17-20 optionally includes the obtained images or video streams of the target anatomy that can include images or video streams of one or more colon segments during withdrawal of an colonoscope during a colonoscopy procedure, wherein the AOI includes a pre-determined colon landmark or an anomaly in a colon segment.

[0031] In Example 22, the subject matter of any one or more of Examples 17-21 optionally includes the AOI that can include a blind spot region of the target anatomy, the method further comprising generating a three- dimensional (3D) reconstruction of the blind spot region using images or video streams produced in accordance with the target or recommended imaging setting.

[0032] In Example 23, the subject matter of any one or more of Examples 17-22 optionally includes tracking location and orientation of the imaging device or a distal portion of the endoscope during the endoscopy procedure, wherein the identification of the AOI or the generation of the endoscope maneuvering plan are further based on the tracked location or orientation.

[0033] In Example 24, the subject matter of any one or more of Examples 17-23 optionally includes identifying the AOI using a first trained machine-learning (ML) model trained to establish a correspondence between (i) an endoscopic image or video stream of the target anatomy or features extracted therefrom and (ii) one or more characteristics of an AOI.

[0034] In Example 25, the subject matter of any one or more of Examples 17-24 optionally includes generating the endoscope maneuvering plan using a second trained machine-learning (ML) model trained to establish a correspondence between (i) an endoscopic image or video stream or the identified AOI or features extracted therefrom and (ii) an imaging setting of the imaging device.

[0035] In Example 26, the subject matter of any one or more of Examples 17-25 optionally includes generating an alert to the user about the identified AOI, and receiving a user feedback about the alert, wherein generating the endoscope maneuvering plan is based at least in part on the user feedback about the alert.

[0036] In Example 27, the subject matter of any one or more of Examples 17-26 optionally includes robotically maneuvering the endoscope and imaging and photo-documenting of the AOI in accordance with the endoscope maneuvering plan.

[0037] The presented techniques are described in terms of controlled withdrawal of endoscope in colonoscopy but are not so limited. The systems,devices, and techniques as described in accordance with various embodiments in this document, may additionally or alternatively be used in other procedures involving different types of endoscopes, including, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, colposcopy, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy etc.

[0038] This summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. Other aspects of the disclosure will be apparent to persons skilled in the art upon reading and understanding the following detailed description and viewing the drawings that form a part thereof, each of which are not to be taken in a limiting sense. The scope of the present disclosure is defined by the appended claims and their legal equivalents.BRIEF DESCRIPTION OF THE DRAWING

[0039] FIGS. 1-2 are schematic diagrams illustrating an example of an endoscope system for use in endoscopy procedures, procedures, such as a colonoscopy procedure.

[0040] FIG. 3 illustrates an example of an endoscope system for realtime endoscopy scene analysis and photo-documentation.

[0041] FIGS. 4A-4B illustrates exemplary anatomical landmarks that may be identified during upper and lower gastrointestinal (GI) endoscopy.

[0042] FIG. 5 illustrates by way of example alert of anomaly detection and real-time guidance in endoscope navigation and imaging and photodocumentation of the detected anomaly in image-guided colonoscopy.

[0043] FIG. 6 illustrates portions of a real-time alert and recommendation system for generating an alert of anomaly detection and realtime guidance in endoscope navigation and imaging and photo-documentation of the detected anomaly.

[0044] FIGS. 7A-7B are diagrams illustrating examples of training an ML model and using the trained ML model to determine a personalized endoscope maneuvering plan for anomaly inspection and diagnosis.

[0045] FIG. 8 is a flow chart illustrating an example method of real-time endoscopy scene analysis and photo-documentation during an endoscopy procedure.

[0046] FIG. 9 is a block diagram illustrating an example machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform.DETAILED DESCRIPTION

[0047] This document describes systems, devices, and methods for realtime endoscopy scene analysis and photo-documentation during an endoscopy procedure. An exemplary endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging device to obtain images or video streams of a target anatomy during an endoscopy procedure. The controller circuit can analyze the obtained images or video streams to identify an area of interest (AOI) such as anatomical landmarks or anomalous or pathological structure, and based on the identification of the AOI, generate an endoscope maneuvering plan including a target or recommended imaging setting of the imaging device. Real-time guidance can be provided to a user or a robotic system to assist in maneuvering the endoscope and imaging and photodocumenting the AOI in accordance with the endoscope maneuvering plan.

[0048] FIG. 1 is a schematic diagram of an endoscope system 10 for use in an endoscopy procedure, such as colonoscopy. The system 10 can include an imaging and control system 12 and an endoscope 14. The system 10 is an illustrative example of an endoscope system suitable for use with the systems, devices, and methods described herein, such as a colonoscopy system for use in image-guided colonoscopy with auto-adjusted imaging modality as described in this document.

[0049] The endoscope 14 can be insertable into an anatomical region for imaging or to provide passage of or attachment to (e.g., via tethering) one or more sampling devices for biopsies or therapeutic devices for treatment of a disease state associated with the anatomical region. The endoscope 14 can interface with and connect to imaging and control system 12. The endoscope 14 may be a colonoscope, though other types of endoscopes can be used with the features and teachings of the present disclosure. The imaging and control system12 can include a control unit 16, an output unit 18, an input unit 20, a light source unit 22, a fluid source 24, and a suction pump 26.

[0050] The imaging and control system 12 can include various ports for coupling with the endoscope system 10. For example, the control unit 16 can include a data input / output port for receiving data from and communicating data to the endoscope 14. The light source unit 22 can include an output port for transmitting light to the endoscope 14, such as via a fiber optic link. The fluid source 24 can include a port for transmitting fluid to the endoscope 14. The fluid source 24 can include, for example, a pump and a tank of fluid or can be connected to an external tank, vessel, or storage unit. The suction pump 26 can include a port to draw a vacuum from the endoscope 14 to generate suction, such as for withdrawing fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and the input unit 20 can be used by an operator of the endoscope system 10 to control functions of the endoscope system 10 and view the output of the endoscope 14. The control unit 16 can also generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical output, acoustic output, fluid output, and the like for treating the anatomical region with, for example, cauterizing, cutting, freezing, and the like.

[0051] The fluid source 24 can be in communication with control unit 16 and can include one or more sources of air, saline, or other fluids, as well as associated fluid pathways (e.g., air channels, irrigation channels, suction channels, or the like) and connectors (barb fittings, fluid seals, valves, or the like). The fluid source 24 can be utilized as an activation energy for a biasing device or a pressure-applying device of the present disclosure. The imaging and control system 12 can also include the drive unit 46, which can include a motorized drive for advancing a distal section of endoscope 14.

[0052] The endoscope 14 can include an insertion section 28, a functional section 30, and a handle section 32, which can be coupled to a cable section 34 and a coupler section 36. The insertion section 28 can extend distally from the handle section 32, and the cable section 34 can extend proximally from the handle section 32. The insertion section 28 can be elongated and can include a bending section and a distal end to which the functional section 30 can be attached. The bending section can be controllable (e.g., by a control knob 38 onthe handle section 32) to maneuver the distal end through tortuous anatomical passageways (e.g., stomach, duodenum, kidney, ureter, etc.). The insertion section 28 can also include one or more working channels (e.g., an internal lumen) that can be elongated and can support the insertion of one or more therapeutic tools of the functional section 30. The working channel can extend between the handle section 32 and the functional section 30. Additional functionalities, such as fluid passages, guide wires, and pull wires, can also be provided by the insertion section 28 (e.g., via suction or irrigation passageways or the like).

[0053] A coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the input unit 20, the light source unit 22, the fluid source 24, and the suction pump 26.

[0054] The handle section 32 can include the knob 38 and the port 40A. The knob 38 can be connected to a pull wire or other actuation mechanisms that can extend through the insertion section 28. The port 40 A, as well as other ports, such as a port 40B (FIG. 2), can be configured to couple various electrical cables, guide wires, auxiliary scopes, tissue collection devices, fluid tubes, and the like to the handle section 32, such as for coupling with the insertion section 28.

[0055] According to examples, the imaging and control system 12 can be provided on a mobile platform (e.g., a cart 41) with shelves for housing the light source unit 22, the suction pump 26, an image processing unit 42 (FIG. 2), etc. Alternatively, several components of the imaging and the control system 12 (shown in FIGS. 1 and 2) can be provided directly on the endoscope 14 to make the endoscope “self-contained.”

[0056] The functional section 30 can include components for treating and diagnosing anatomy of a patient. The functional section 30 can include an imaging device, an illumination device, and an elevator. The functional section 30 can further include optically enhanced biological matter and tissue collection and retrieval devices. For example, the functional section 30 can include one or more electrodes conductively connected to the handle section 32 and functionally connected to the imaging and control system 12 to analyze biological matter in contact with the electrodes based on comparative biologicaldata stored in the imaging and control system 12. In other examples, the functional section 30 can directly incorporate tissue collectors.

[0057] In some examples, the endoscope 14 can be robotically controlled, such as by a robot arm attached thereto. The robot arm can automatically, or semi-automatically (e.g., with certain user manual control or commands), via an actuator, position and navigate the endoscope 14 (e.g., the functional section 30 and / or the insertion section 28) in the target anatomy or position a device at a desired location with desired posture to facilitate an operation of an anatomical target. In accordance with various examples discussed in this document, a controller can generate a control signal to the actuator of the robot arm to facilitate anomaly inspection and diagnosis under the target or recommended imaging mode in a robotically assisted endoscopy procedure.

[0058] FIG. 2 is a schematic diagram of the endoscope system 10 of FIG. 1 including the imaging and control system 12 and the endoscope 14. FIG. 2 schematically illustrates components of the imaging and the control system 12 coupled to the endoscope 14, which in the illustrated example includes a colonoscope. The imaging and control system 12 can include the control unit 16, which can include or be coupled to an image processing unit 42, a treatment generator 44, and a drive unit 46, as well as the light source unit 22, the input unit 20, and the output unit 18. The control unit 16 can include, or can be in communication with, an endoscope, a surgical instrument 48, and an endoscope system, which can include a device configured to engage tissue and collect and store a portion of that tissue and through which imaging equipment (e.g., a camera) can view target tissue via inclusion of optically enhanced materials and components. The control unit 16 can be configured to activate a camera to view target tissue distal of the endoscope system. Likewise, the control unit 16 can be configured to activate the light source unit 22 to shine light on the surgical instrument 48, which can include select components configured to reflect light in a particular manner, such as enhanced tissue cutters with reflective particles.

[0059] The coupler section 36 can be connected to the control unit 16 to connect to the endoscope 14 to multiple features of the control unit 16, such as the image processing unit 42 and the treatment generator 44. In examples, the port 40A can be used to insert another surgical instrument 48 or device, such as adaughter scope or auxiliary scope, into the endoscope 14. Such instruments and devices can be independently connected to the control unit 16 via the cable 47. In examples, the port 40B can be used to connect coupler section 36 to various inputs and outputs, such as video, air, light, and electric.

[0060] The image processing unit 42 and light source unit 22 can each interface with the endoscope 14 (e.g., at the functional section 30) by wired or wireless electrical connections. The imaging and control system 12 can accordingly illuminate an anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display images representing the anatomical region on the display unit 18. The imaging and control system 12 can include the light source unit 22 to illuminate the anatomical region using light of desired spectrum (e.g., broadband white light, narrow-band imaging using preferred electromagnetic wavelengths, and the like). The imaging and control system 12 can connect (e.g., via an endoscope connector) to the endoscope 14 for signal transmission (e.g., light output from light source, video signals from imaging system in the distal end, diagnostic and sensor signals from a diagnostic device, and the like).

[0061] The treatment generator 44 can generate a treatment plan, which can be used by the control unit 16 to control the operation of the endoscope 14, or to provide with the operating physician a guidance for maneuvering the endoscope 14, during an endoscopy procedure. In an example, the treatment generator 44 can generate an endoscope navigation plan, including estimated values for one or more cannulation or navigation parameters (e.g., an angle, a force, etc.) for maneuvering the steerable elongate instrument, using patient information including an image of the target anatomy. The endoscope navigation plan can help guide the operating physician to cannulate and navigate the endoscope in the patient anatomy. The endoscope navigation plan may additionally or alternatively be used to robotically adjust the position, angle, force, and / or navigation of the endoscope or other instrument.

[0062] FIG. 3 is a block diagram illustrating an example of an endoscope system 300 for real-time endoscopy scene analysis and photo-documentation. The endoscopy system may be used during an endoscopy procedure to identify critical anatomical and pathologic findings based on scene analysis, prompt a user (e.g., an endoscopist) to capture and photo-document relevant images orvideo streams, and generate an endoscope maneuvering plan to guide the user towards capturing an ideal or optimal view with critical details of the tissue or foreign objects of interest. By way of example and not limitation, the endoscope system 300 may be used in a colonoscopy procedure to better detect and manage anomalies such as polyps or colorectal cancers. The endoscope system 300 may be implemented as a part of the control unit 16 in FIG. 1.

[0063] The endoscope system 300 may include one or more of an endoscope 310, an auxiliary input 315, a controller circuit 320, a user interface 330, and a storage device 340. In some examples, the endoscope system 300 may further include or be communicatively coupled to a robotic system 350 in a robotically assisted endoscopy procedure.

[0064] The endoscope 310 can be an example of the endoscope 14 as described above and shown in FIGS. 1-2. The endoscope 310 may include, among other things, an imaging system 312 and a lighting system 314. The imaging system 312 can include at least one imaging sensor or device (e.g., a camera) configured to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure. The imaging sensor or device may be located at a distal portion or a distal end of the endoscope 310. The lighting system 314 may include one or more light sources to produce illumination on the target anatomy via one or more lighting lenses.

[0065] The imaging system 312 may be controllably adjusted to operate on different settings including zoom settings, contrast settings, exposure levels, or viewing angles toward and around the target anatomy. The lighting system 314 may be controllably adjusted to provide different lighting or illumination conditions. The imaging system 312 and the lighting system 314 together can define an imaging mode for capturing endoscopic images or video streams of the target anatomy. In this document, the imaging mode may include one or more of a lighting modality, an optical magnification, or a viewing angle of the imaging device. Examples of imaging or lighting modality may include high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others. The optical magnification defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy). The viewing angle of the imaging device, alsoreferred to as a field of view, describes the angular extent of a given scene that is imaged by the imaging device.

[0066] The controller circuit 320 may include circuit sets comprising one or more other circuits or sub-circuits that may, alone or in combination, perform the functions, methods, or techniques described herein. In an example, the controller circuit 320 and the circuits sets therein may be implemented as a part of a microprocessor circuit, which may be a dedicated processor such as a digital signal processor, application specific integrated circuit (ASIC), microprocessor, or other type of processor for processing information including physical activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that may receive and execute a set of instructions of performing the functions, methods, or techniques described herein. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0067] The controller circuit 320 can determine a personalized, pathology-specific imaging modality in substantially real time during an endoscopy procedure using at least the endoscopic images or video streams or features extracted therefrom. As described above, imaging modality plays animportant role in determining image or video quality during endoscopy and therefore the accuracy and efficiency of optical detection, diagnosis, and treatment of pathologies of various types. Conventionally, determining a desired or “optimal” imaging modality best suited to detect or diagnose certain pathologies is a highly manual process. It generally requires the user (e.g., endoscopist) to manually toggle among the available imaging modalities, recognize a current type of pathology being viewed, and recall which imaging modality was used for viewing such pathology. Based on the visual inspection, the user manually adjusts the imaging system or the lighting system to achieve desired lighting modality, optical magnification, or viewing angle. Such manual process takes time and can be laborious and may introduce unpredictable interuser variations. The endoscope system 300 as described herein can automate the imaging modality selection and optimization process using artificial intelligence (Al) or machine learning (ML) based techniques, as described in further detail below.

[0068] The controller circuit 320 may include an image processor 321, an area of interest (AOI) detector 322, a device tracking circuit 323, an endoscope maneuver planning circuit 324, and a real-time alert and recommendation circuit 328. The image processor 321 can analyze the images or video streams obtained from the imaging system 312, and generate endoscopic image or video features. Examples of the image or video features include statistical features of pixel values or morphological features, such as corners, edges, blobs, curvatures, speeded up robust features (SURF), or scale-invariant feature transform (SIFT) features, among others. In some examples, the image processor 321 may pre-process the images or video streams, such as filtering, resizing, orienting, or color or grayscale correction, and the endoscopic features may be extracted from the pre-processed images or video streams. In some examples, the image processor 321 may post-process the image features to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0069] The AOI detector 322 may detect an AOI in at the target anatomy based at least in part on the endoscopic mage or video features. The AOI may include one or more pre-determined anatomical landmarks in the target anatomy. Systemic endoscopic image documentation of such anatomical landmarks servesthe purposes of showing crucial anatomic structures at these landmark regions, documenting the extent of the examination, and reflecting the quality of cleansing and mucosal visualization. Referring now to FIGS. 4A-4B, the diagrams, adapted from “ESGE Recommendations for Quality Control in Gastrointestinal Endoscopy: Guidelines for Image Documentation in Upper and Lower GI Endoscopy” (Endoscopy 2001; 33: 901-903), illustrate upper GI and lower GI endoscopy landmarks recommended by the European Society of Gastrointestinal Endoscopy (ESGE). The recommended upper GI landmarks, as illustrated in FIG. 4A, include proximal esophagus 410, distal esophagus 411, Z- line and diaphragm indentation 412, cardia and fundus on retroflexed view 413, body (including lesser curvature) 414, body on retroflexed view 415, Angulus on partial retroflexion 416, antrum 417, duodenal bulb 418, and second part of the duodenum (including the ampulla) 419. The recommended lower GI (colon, in particular) landmarks, as illustrated in FIG. 4B, include lower part of the rectum in retroflexed view 421, lower part of the rectum 422 (taken 2 centimeter above the anal line), middle part of the sigmoid 423, descending colon just distal to the splenic flexure 424, transverse colon just proximal to the splenic flexure 425, transverse colon just distal to the hepatic flexure 426, ascending colon just proximal to the hepatic flexure 427, cecum and ileocecal valve 428, and cecum and appendiceal orifice 429. In an example when the endoscope system 300 is used in a colonoscopy procedure, one or more pre-determined colon landmarks such as selected from the recommended lower GI landmarks as shown in FIG.4B may be identified from images or video streams of one or more colon segments such as captured by the imaging system 312 during withdrawal of the colonoscope.

[0070] In addition or alternative to the pre-determined anatomical landmarks, the AOI to be identified from the endoscopic images or video streams may include an anomalous structure in the target anatomy. This may include one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the target anatomy. In an example of colonoscopy, the anomalous structure may include pathological tissue segments, such as mucosal abnormalities (polyps, inflammatory bowel diseases, Meckel’sdiverticulum, lipoma, bleeding, vascularized mucosa etc.), or obstructed mucosa (e.g., segments with bad bowel preparation, distended colon etc.).

[0071] The AOI detector 322 may detect the AOI (e.g., a pre-determined landmark or an anomaly) using various techniques. In an example, a template matching technique can be used to identify AOI based on a comparison of the endoscopic image features (e.g., features characterizing shapes or contours of a structure) to one or more pre-generated templates of known anomalous structure. In another example, the AOI detector 322 may detect an AOI using Al or ML based techniques, where the endoscopic images or video streams or features extracted therefrom may be applied to a trained ML model to automatically identify the AOI. The ML model may be trained to establish a correspondence between an endoscopic image or video stream (or features extracted therefrom) and one or more recognizable anatomical landmarks. Examples of ML models for recognizing an anatomical landmark include Deep Belief Network, ResNet, DenseNet, Autoencoders, capsule networks, generative adversarial networks, Siamese networks, Convolutional Neural Networks (CNN), deep reinforcement learning, support vector machine (SVM), Bayesian models, decision trees, k- means clustering, among other ML models. In another example, an ML model may be trained to establish a correspondence between an endoscopic image or video stream, or features extracted therefrom and one or more anomaly characteristics. Examples of ML models for recognizing anomaly from endoscopic images or video streams include Convolutional Neural Networks, bidirectional LSTM, Recurrent Neural Networks, Conditional Random Fields, Dictionary Learning, or other machine learning techniques (support vector machine, Bayesian models, decision trees, k-means clustering), among other ML techniques. The trained ML model may be stored in the storage device 340.

[0072] In some examples, the AOI may include a “blind spot” region of the target anatomy. In colonoscopy for example, because colon has innumerable folds and two flexures, such anatomical characteristics may cause certain hidden places inadvertently missed for lesion screening, such as polyps or cancers. Blind spots commonly occur at acute bends where the camera’s view is blocked. In some examples, the controller circuit 320 may generate a three-dimensional (3D) reconstruction of the blind spot region using images or video streams produced in accordance with the target or recommended imaging setting.

[0073] The device tracking circuit 323 may track in substantially real time location and orientation of the endoscope or a portion thereof in a particular segment of the anatomy during an endoscopy procedure. In an example of colonoscopy, the device tracking circuit 323 can continuously monitor and record the location and orientation of the colonoscope, and align the position and orientation information with a standardized colon location template. In an example, the device tracking circuit 323 can track the location and orientation of an imaging device (e.g., a camera) at a distal portion of the endoscope during the procedure. With such real-time tracking information, the AOI detector 322 may associate a detected AOI (e.g., an anatomical landmark as shown in FIGS. 4A or 4B, or an anomaly) with the segment of the anatomy. In an example, the device tracking can be based on an electromagnetic property of the imaging device, or the distal portion of the endoscope detected during the endoscopy procedure. The device tracking circuit 323 may register a tracked location of the imaging device to a pre-generated template of the target anatomy during the endoscopy procedure, and the location of the imaging device can be determined based on a degree of template matching. In another example, the device tracking can be based on one or more anatomical landmarks identified from the images or video streams (or features extracted therefrom) such as obtained during the insertion phase of endoscopy. Once the endoscope location and orientation are determined, the device tracking circuit 323 may register the substantially realtime endoscope location and orientation to a pre-generated template of the anatomy. Information of the endoscope location and orientation may be presented to the user on the user interface 330.

[0074] The endoscope maneuver planning circuit 324 may generate a personalized endoscope maneuvering plan for operating the endoscope 310 and imaging the identified AOI with improved quality during the procedure using at least the information about the identified AOI (such as produced by the AOI detector 322) and the real-time location and orientation of the endoscope or the imaging device associated therewith (such as produced by the device tracking circuit 323). The endoscope maneuvering plan may include a guided navigation 325 of the endoscope, and an imaging mode or setting 326. The guided navigation 325 can include navigation instructions or navigation paths towards and around the AOI that are presented to the user (e.g., endoscopist) insubstantially real-time. The navigation instructions or paths may be presented in the for textual (e.g., tables or lists), graphical, animated, or other visual or audio formats easily interpreted and followed by the user to advance, retreat, rotate, or steering the distal tip of the endoscope. In an example, the guided navigation instructions or path can be displayed on a display of the user interface 330. In various example, the guided navigation instructions or path can be defined in a coordinate system, such as a two-dimensional (2D) or a three-dimensional (3D) coordinate system. Positions, postures, orientations, and headings of the distal tip of the endoscope may be defined as coordinates (Cartesian, polar, or spherical coordinates).

[0075] The imaging mode or setting 326 can be represented by a set of imaging and lighting parameters that provide ideal or optimal view(s) with critical details of the AOI such as tissue or foreign objects of interest. Imaging parameters of the imaging system 312 may include a position, a posture, or a viewing angle of the imaging device with respect to the identified AOI, an optical magnification effect that defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy), a focus setting, a contrast setting, a viewing angle (e.g., an angular extent of a given scene as captured by the imaging device), or a centering setting to center the identified AOI or a portion thereof in a field of view (FOV) of the imaging device. Lighting parameters of the lighting system 314 may include a lighting mode or lighting intensity of the lighting system 314. Examples of lighting modality may include high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others.

[0076] In an example, the personalized endoscope maneuvering plan may be determined further based on the quality of the images or video streams of the AOI as produced by the imaging system 312. The image processor 321 may generate an image quality indicator of the images or video streams produced under an existing imaging setting of the imaging device. The endoscope maneuver planning circuit 324 can determine whether to keep or modify the existing imaging setting (such as by adjusting imaging parameter values) based at least in part on the image quality indicator.

[0077] In an example, the personalized endoscope maneuvering plan may be determined using Al or ML based techniques. In an example, information about the identified AOI (including anomalous structures and / or anatomical landmarks of interest) and substantially real-time endoscope locations may be applied to trained ML model(s) 360. The ML model(s) 360 may be trained to establish a correspondence between images or video streams of the AOI or features thereof and an ideal or recommended endoscope maneuvering plan (including one or more of ideal or recommended guided navigation 325 or imaging mode or setting 326). The trained ML model(s) 360 may be stored in a storage device 340. Examples of training an ML model and using the trained ML model to determine a personalized endoscope maneuvering plan are discussed below with respect to FIGS. 7A-7B.

[0078] In some examples, the endoscope maneuver planning circuit 324 may determine the personalized endoscope maneuvering plan further using an auxiliary input 315. By way of example and not limitation, the auxiliary input may include image or video streams and clinical data from previous endoscopy procedures performed on the patient, patient information and medical history, or pre-procedure imaging study data (e.g., X-ray or fluoroscopy images, electrical potential map or an electrical impedance map, computer tomography (CT) images, magnetic resonance imaging (MRI) images, among other imaging modalities). In an example of colonoscopy, the auxiliary input may include previous colonoscopies in surveillance cases, including those polyps that the endoscopist left in situ or areas of the colon which were operated on. The patient information and medical history may include, in a typical endoscopy scenario, clinical demographic information, past and current indications and treatment received, etc. In some examples, the auxiliary input may include user (e.g., endoscopist) profile including user’s experience, working environment (e.g., a hospital setting or an ambulatory screening centre), affinity to new technology, preference of a certain endoscopy protocol or certain imaging modes. The user profile may be provided by the user via the user interface 330. Alternatively, the user profile may be automatically generated or updated through learning from the user’s past choices and training. In some examples, the auxiliary input may additionally or alternatively include endoscope and equipment information, such as specification of the endoscope 310 including type, size, dimension, shape, andstructures of the endoscope or other steerable instruments such as a cannular, a catheter, or a guidewire supporting imaging modes and lighting modes (e.g., NBI, RDI, WLI, TXI, etc.); specification of size, dimension, shape, and structures of tissue section, sampling, or treatment tools; current state of the equipment, including which light mode is on, which endo buttons are engaged like waterjet, insufflation, what light modes are supported, or whether the magnification is turned on and the current magnification selection, etc.

[0079] The real-time alert and recommendation circuit 328 can generate an alert of the identified AOI (e.g., an anatomical landmark or an anomaly), provide real-time guidance in maneuvering the endoscope and imaging and photo-documenting the AOI using a recommended imaging mode or setting. Images or video streams of AOI (e.g., anomalies) identified by the AOI detector 322, and personalized endoscope maneuvering plan determined by the endoscope maneuver planning circuit 324, may be presented to the user on the user interface 330. The user may review the images of the identified AOIs, and either take the recommendation to make photo-documentation of the images as presented, or reject the recommendation and ignore the identified anomaly (e.g., a false alarm). Guided navigation or imaging setting optimization may be initiated per user’s request, or alternatively automatically triggered based on an image quality analysis. For example, a low image quality may automatically trigger imaging setting optimization. In an example, if the determined imaging mode or setting is not the same as the current imaging mode or setting being used for inspecting and detecting the anomaly, the user may be notified or warned of such difference via the user interface 330. The notification may be delivered through optical means on the diagnostic monitor, or via auditory means like a warning alarm. In an example, highlighting, flash alerts, audible or haptic feedback may be provided to the user. Examples of real-time alert of anomalies detected, guided endoscope navigation, and imaging mode / setting recommendation for photo-documentation are discussed below with reference to FIGS. 5 and 6. Other information, including the endoscopic images and image features, information about the detected anomaly and landmarks in each segment of the anatomy, or substantially real-time location of the endoscope during the endoscopy procedure, may also be displayed on the user interface 330. At the completion of the procedure (e.g., withdrawal of colonoscope duringcolonoscopy), information about the anomaly detected and treated, and the total net withdrawal time, among other information, may be generated in a postprocedure summary. Such post-procedure analytics may be used for quality assurance and other purposes.

[0080] In some example, the personalized endoscope maneuvering plan may be stored in the storage device 340. Information about the anomaly detected may also be stored in the storage device 340. The storage device 340 can be local to the endoscope system 300. Alternatively, the storage device 340 can be a remote storage device, such as a part of a cloud comprising one or more storage and computing devices (e.g., servers) that provides secure access to cloud-based services including, for example, data storage, computing services, and provisioning of customer services, among others. In some examples, at least some of the data processing and computation with regard to anomaly detection, landmark identification, endoscope localization, and imaging mode selection may be performed in a cloud. For example, images or video streams or features extracted therefrom may be streamed to the cloud, get processed therein, and the computation results (e.g., the personalized pathology-specific imaging modality) may be relayed back to local endoscope system.

[0081] In some examples, the endoscope system 300 may be operated in a closed-loop fashion with a feedback loop for continuous learning and of personalized endoscope maneuvering plan, such as updating a previously generated personalized endoscope maneuvering plan. The continuous learning may be achieved via explicit endoscopist feedback (e.g., satisfaction with recommendations, a “like” button etc.). Alternatively, the endoscope system 300 can run in a shadow mode to monitor actions of experienced endoscopists and correct the personalized endoscope maneuvering plan through reinforcement feedback.

[0082] In some examples, the image-guided endoscopy (or a portion therefore such as endoscope withdrawal), may be performed using the robotic system 350. The robotic system 350 can manipulate the endoscope, adjust the imaging mode or setting, and produce guided imaging and photo-documentation of the AOI, in accordance the recommended endoscope maneuvering plan. The robotic system 350 may include a robot arm detachably attached to the endoscope 310. The robot arm can automatically, or semi -automatically (e.g.,with certain user manual control or commands), via an actuator, position and manipulate the endoscope 310 in an anatomical target, or position a device at a desired location with desired posture to facilitate an operation on the anatomical target.

[0083] FIG. 5 is a diagram illustrating by way of example real-time alerts of anomalies and automatic guidance in navigating an endoscope and imaging and photo-documenting the detected anomalies in image-guided colonoscopy such as performed using the endoscope system 300. The colonoscopy procedure, as illustrated, includes colonoscope insertion stage and a subsequent colonoscope withdrawal stage. During the insertion stage, endoscopic images or video streams can be obtained respectively for each of a plurality of colon segments, including the rectosigmoid segment, sigmoid segment, descending segment, transverse segment, ascending segment, and cecum, such as using the imaging system 312. Various AOIs, including pre-determined anatomical landmarks and anomalies, may be identified automatically using the AOI detector 322 during the insertion phase or alternatively, as shown in FIG. 5, during the withdrawal phase. In an example, at the end of the insertion stage when the distal tip of the colonoscope is determined to be reaching the cecum (such as by the device tracking circuit 323), anomaly detection may be manually or automatically initiated. As described above with respect to FIG. 3, the AOI detector 322 may analyze the scans (i.e., images or video streams) of the endoscopic feed using template-matching or AI / ML based algorithm, and identify unusual or suspicious regions indicative of pathological abnormalities, such as anomalies 510A-510D in one or more colon segments during the withdrawal phase.

[0084] Each image-based anomaly detection may trigger an alert, such as alerts 520A-520D corresponding to imaged anomalies 510A-510D. The alert may be presented to the user, such as on a display of the user interface 330. The user may be prompted to make proper photo-documentation of the identified anomalies. The user may review the images of the identified anomalies, and choose to take the recommendation to make photo-documentation of the images as presented, or reject the recommendation and ignore the identified anomaly (e.g., a false alarm). In some examples, on the real-time feed, visual cues ormarkers may be superimposed to underline the identified landmarks or anomalous areas, thereby guiding the user’s focus on those areas of interest.

[0085] The endoscope maneuver planning circuit 324 can perform guided navigation and generate target or recommended imaging setting to assist the user in securing the high-quality images for record-keeping. Guided navigation or imaging setting optimization may be initiated per user’s request, or alternatively automatically triggered based on an image quality analysis (e.g., a low image quality automatically triggers imaging setting optimization). As described above with respect to FIG. 3, the image processor 321 may evaluate the current images or video stream, and generate an image quality indicator indicating, for example, presence and degree of issues like blur, immoderate illumination, or other quality-compromising factors. In some examples, to guarantee a symmetrically framed snapshot, the endoscope maneuver planning circuit 324 may generate on-screen guides aiding the user in manipulating the endoscope and positioning the imaging device’s focus on a particular location on the identified AOI. In an example, should the present viewing angle of the imaging device be less than ideal for capturing the scene (e.g., the AOI is not positioned at the center of the FOV of the imaging device), a recommendation to adjust the viewing angle (e.g., a tilt or turn of the camera) may be presented to the user for an optimal viewpoint. In another example, to provide sharp depiction of the AOI, the endoscope maneuver planning circuit 324 may provide recommendations on the ideal zoom level, making certain that the AOI is neither overly enlarged nor too re- mote. Adjustment of other imaging parameters may be similarly recommended. The imaging system 312 may be activated manually or automatically to retake images or video streams under the adjusted imaging setting. The retaken images or video streams may be presented to the user who may perform a final examination before committing the images or video streams to photo-documentation.

[0086] In the example as illustrated, in response to the alert 520A of identified anomaly shown in image 510A, the user reviews the image, and takes the recommendation to proceed with photo-recommendation. In response, guided navigation or imaging setting optimization 530A may be initiated, either manually by the user or automatically triggered based on image quality analysis, as described above. Referring now to FIG. 6, the diagram therein illustratesportions of the real-time alert and recommendation system 600 for generating an alert of anomaly detection and real-time guidance in endoscope navigation and imaging and photo-documentation of the detected anomaly. An alert module 610 may involve colonoscope position tracking (as enabled by the device tracking circuit 323 of FIG. 3) and image-based anomaly detection (as enabled by the AOI detector 322 of FIG. 3). The colonoscope position tracking may include colonoscope location registration, electromagnetic tracking of an imaging device associated with the colonoscope, and visual landmark detection. The image or video stream-based anomaly detection may include segment analysis (e.g., colon segments), anomaly detection, and anomaly analysis (e.g., to identify characteristics of the detected anomaly). An imaging mode or setting module 620 may involve one or more of image clarity analysis, landmark centering assistance, optimal viewing angle prediction, zoom level guidance, and final review of the image before photo-documentation. In some examples, one or more of the imaging setting parameters may be determined using Al or ML techniques, as will be discussed further below with respect to FIGS. 7A-7B. In response to manually requested or automatically generated imaging setting optimization 530A, a target or recommended imaging mode or setting can be determined, and an updated image 540A of the same identified anomaly may be generated in accordance with the target or recommended imaging mode or setting. If the updated image 540A is satisfactory, the user may take action 550A to free and document the updated image 540A.

[0087] Referring back to FIG. 5, the user may similarly review other alerted anomalies, and take actions such as accepting or rejecting the recommendation for photo-documenting the respectively detected anomalies. For example, in response to alerts 520B and 520D, the user inspects the images of the detected anomalies 510B and 510D, and takes respective actions 530B and 530D to freeze and document said images without further optimization of the imaging mode or setting. In response to alert 520C, the user inspects the image of the detected anomaly 510C, recognizes it as a normal or nonsignificant structure, and takes an action 530C to ignore and not to photodocument the image of the detected anomaly 510C.

[0088] FIGS. 7A-7B are diagrams illustrating examples of training an ML model and using the trained ML model to determine a personalizedendoscope maneuvering plan for anomaly inspection and diagnosis. FIG. 7A illustrates an ML model training (or learning) phase during which an ML model 730 may be trained to determine a target or recommended endoscope maneuvering plan based at least in part on endoscopic images of the AOI and surrounding environment. A training dataset may include a plurality of endoscopic images 710 of the same type of anatomical landmarks or the same type of anomalies (e.g., a polyp or lesion in the lining of a colon segment) obtained from colonoscopy procedures performed on a plurality of patients. In some examples, the training data may also include auxiliary input 315 as described above with respect to FIG. 3, including, for example, supporting imaging modes and lighting modes, endoscopist preference, clinical information 736 which may include one or more of the image or video streams and clinical data from previous endoscopy procedures, patient information and medical history, or the pre-procedure imaging study data, etc. Endoscope location information for each of the plurality of endoscopic images may also be included in the training dataset.

[0089] The ML model 730 may have a neural network structure comprising an input layer, one or more hidden layers, and an output layer. The plurality of endoscopic images 710 or features generated therefrom, along with one or more of the auxiliary training data 720, may be fed into the input layer of the ML model 730, which propagates the input data or data features through one or more hidden layers to the output layer that outputs a recommended pathologyspecific imaging modality. The ML model 730 can perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The ML model 730 explores the study and construction of algorithms (e.g., ML algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building the ML model 730 from training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0090] The ML model 730 may be trained using supervised learning or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between thetraining inputs and outputs so that the ML model can implement the same relationships when given inputs to generate the corresponding outputs. Unsupervised learning is the training of an ML algorithm using information that is neither classified nor labelled and allowing the algorithm to act on that information without guidance. Unsupervised learning is useful in exploratory analysis because it can automatically identify structure in data.

[0091] Common tasks for supervised learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values. Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of commonly used supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM). Examples of DNN include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), or a hybrid neural network compri sing two or more neural network models of different types or different model configurations.Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0092] Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machine-learning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0093] The training of the ML model 730 may be performed continuously or periodically, or in near real time as additional procedure data are made available. The training process involves algorithmically adjusting one ormore ML model parameters (e.g., weights or bias at any particular layer of a neural network model), until the ML model being trained satisfies a specified training convergence criterion. By way of example and not limitation, the ML model 730 may be trained with weighted square loss (for explicit feedback) or with binary cross-entropy loss (for implicit feedback). Other training techniques, such as deep factorization machine, wide and deep learning, deep structured semantic models, or autoencoder based recommender systems, may be used. The trained ML mode 730 can establish a correspondence between the endoscopic images 710 (or features extracted therefrom) and an ideal or recommended endoscope maneuvering plan, including a guided endoscope navigation path and / or a recommended imaging mode or setting.

[0094] In an example, the trained ML model 730 can establish a correspondence between the endoscopic images 710 (or features extracted therefrom) and optimal or recommended imaging parameters. In various examples, different ML algorithms may be used to learn different imaging parameters. For example, image clarity may be learned through a Deep Learning algorithm, where the trained model can learn to filter out blurs and lighting inconsistencies using, for example, autoencoders and generative adversarial networks. In another example, to produce optimal cantering effect of the anatomical landmark or the identified anomaly on the FOV of the imaging device, Siamese networks and deep reinforcement learning may be used to ensure that the landmark is properly centered and provide on-screen cues. In another example, decision trees and Bayesian models may be used to predict an optimal viewing angle for capturing the landmark or anomaly, offering real-time guidance to the user. In yet another example, support vector machines and k- means clustering techniques may be used to determine an ideal zoom level for detailed yet proportionate imaging. Before finalizing the images or video streams for photo-documentation, Dictionary Learning techniques may be used to compare the image against a repository of optimal shots, and based on the comparison, a feedback may be provided to the user such as to retake the images.

[0095] FIG. 7B illustrates an inference phase during, for example, withdrawal of colonoscope to determine a target or recommended endoscope maneuvering plan. A live endoscopic image 750 of a suspected anomaly (e.g., a polyp or lesion), optionally along with auxiliary training data 760 (including oneor more of, for example, information about the endoscopist profile, patient clinical information, equipment setup features, and endoscope location features) may be applied to the trained ML model 730 to determine an output of a target or recommended endoscope maneuvering plan 770, including target or recommended guided navigation instructions or paths and / or target or recommended imaging mode or settings. The endoscope maneuvering plan 770 may be provided to the endoscopist or a robotic system to assist in anomaly inspection and diagnosis.

[0096] FIG. 8 is a flow chart illustrating an example method 800 of realtime endoscopy scene analysis and photo-documentation during an endoscopy procedure, such as a colonoscopy procedure. A target or recommended endoscope maneuvering plan, including a target or recommended imaging mode or setting of an imaging device, can be determined using at least endoscopic images or video streams of the anatomy. The endoscope maneuvering plan thus determined can help improve real-time inspection, recognition, and diagnosis of various anomalies or pathologies. The method 800 may be implemented in the endoscope system 300. Although the processes of the method 800 are drawn in one flow chart, they are not required to be performed in a particular order. In various examples, some of the processes can be performed in a different order than that illustrated herein.

[0097] At step 810, images or video streams of distinct segments of the target anatomy may be obtained using an imaging system associated with an endoscope. The images or video streams may be acquired when the imaging system is set to one of a plurality of available imaging modes or settings. An imaging mode refers to one or more of a lighting modality, an optical magnification, or a viewing angle of the imaging device. Examples of lighting modality may include high-definition white light imaging (WLI), chromoendoscopy techniques like dye-based, or virtual CE like narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichromatic imaging (RDI), among others. The optical magnification defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy). The viewing angle of the imaging device, also referred to as a field of view, describes the angular extent of a given scene that is imaged by the imaging device.

[0098] At step 820, the obtained images or video streams may be analyzed to identify an area of interest (AOI) in the target anatomy, such as using the AOI detector 322. One example of the AOI includes one or more predetermined anatomical landmarks in the target anatomy, such as those landmarks of the upper GI or of the lower GI duct as illustrated in FIGS. 4A-4B. The AOI may additionally or alternatively include an anomalous structure in the target anatomy, such as one or more of a presence or absence, a type, a size, a shape, a location, or an amount of pathological tissue or anatomical structures, among other objects in an environment of the target anatomy. In some examples, the AOI may include a “blind spot” region of the target anatomy, such as hidden places inadvertently missed for lesion screening during colonoscopy. In colonoscopy for example, because colon has innumerable folds and two flexures, such anatomical characteristics may cause certain. The AOI (e.g., a predetermined landmark or an anomaly) may be identified using techniques such as a template matching technique, or Al or ML based techniques, as described above with respect to FIG. 3.

[0099] At step 830, an endoscope maneuvering plan can be generated based on the identification of the AOI, such as using the endoscope maneuver planning circuit 324. As described above with respect to FIGS. 7A-7B, the endoscope maneuvering plan may be determined using Al or ML based techniques. The endoscopy maneuvering plan may include a target or recommended imaging mode or setting of an imaging device associated with the endoscope, such as a camera for capturing images or video streams of the target anatomy during the procedure. The imaging mode or setting can be represented by a set of imaging and lighting parameters that provide ideal or optimal view(s) with critical details of the AOI such as tissue or foreign objects of interest. Examples of the imaging parameters may include a position, a posture, or a viewing angle of the imaging device with respect to the identified AOI, an optical magnification effect that defines a zoom setting (e.g., zoom in or zoom out of a suspicious anomaly of the target anatomy), a focus setting, a contrast setting, a viewing angle (e.g., an angular extent of a given scene as captured by the imaging device), or a centering setting to center the identified AOI or a portion thereof in a field of view (FOV) of the imaging device. Lightingparameters of the lighting system may include a lighting mode or lighting intensity of a lighting system such as the lighting system 314.

[0100] The endoscopy maneuvering plan may also include guided navigation of the endoscope, represented by navigation instructions or navigation paths towards and around the AOI that are presented to the user (e.g., endoscopist) in substantially real-time. In various example, the guided navigation instructions or path can be defined in a coordinate system, such as a two-dimensional (2D) or a three-dimensional (3D) coordinate system.

[0101] In an example, the endoscope maneuvering plan may be determined further using location and orientation of the endoscope or a portion thereof in a particular segment of the anatomy during an endoscopy procedure. Such endoscope location and orientation information may be tracked using the device tracking circuit 323 as described above in FIG. 3. In an example, the device tracking can be based on an electromagnetic property of the imaging device, or the distal portion of the endoscope detected during the endoscopy procedure.

[0102] In an example, the endoscope maneuvering plan may be determined further based on quality of the images or video streams of the AOI, such as produced by the imaging system 312. For example, a low quality of the images or video streams may automatically trigger guided navigation optimization and / or imaging mode or setting optimization.

[0103] At step 840, a real-time guidance in maneuvering the endoscope and imaging and photo-documenting the AOI may be generated and provided to a user (e.g., an endoscopist) or a robotic system (e.g., the robotic system 350). More specifically, in one example, at step 852, the identified AOI and the generated endoscope maneuvering plan may be displayed to a user to assist in manual maneuver of the endoscope, as well as imaging and photodocumentation of the AOI (e.g., an anomaly) as a target or recommended imaging setting. In another example, at step 854, an alert may be generated to notify or warn the user about the identified AOI, and receiving user’s feedback. The user may review the images of the identified AOI, and choose to take the recommendation to make photo-documentation of the images as presented, or reject the recommendation and ignore the identified anomaly (e.g., a false alarm), as described above with respect to FIGS. 5 and 6. In yet anotherexample, at step 856, a control signal may be generated to control a robotic system to automatically maneuver the endoscope for automatic imaging and photo-documenting of the AOI in accordance with the endoscope maneuvering plan previously generated.

[0104] FIG. 9 illustrates generally a block diagram of an example machine 900 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. Portions of this description may apply to the computing framework of various portions of the endoscope system 300.

[0105] In alternative embodiments, the machine 900 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 900 may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine 900 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. The machine 900 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0106] Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership may be flexible over time and underlying hardware variability. Circuit sets include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable mediumphysically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively connected to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0107] Machine (e.g., computer system) 900 may include a hardware processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 904 and a static memory 906, some or all of which may communicate with each other via an interlink (e.g., bus) 908. The machine 900 may further include a display unit 910 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) navigation device 914 (e.g., a mouse). In an example, the display unit 910, input device 912 and UI navigation device 914 may be a touch screen display. The machine 900 may additionally include a storage device (e.g., drive unit) 916, a signal generation device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensors. The machine 900 may include an output controller 928, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0108] The storage device 916 may include a machine readable medium 922 on which is stored one or more sets of data structures or instructions 924 (e.g., software) embodying or utilized by any one or more of the techniques orfunctions described herein. The instructions 924 may also reside, completely or at least partially, within the main memory 904, within static memory 906, or within the hardware processor 902 during execution thereof by the machine 900. In an example, one or any combination of the hardware processor 902, the main memory 904, the static memory 906, or the storage device 916 may constitute machine readable media.

[0109] While the machine-readable medium 922 is illustrated as a single medium, the term "machine readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 924.

[0110] The term “machine readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 900 and that cause the machine 900 to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions. Nonlimiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. In an example, a massed machine-readable medium comprises a machine-readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine- readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0111] The instructions 924 may further be transmitted or received over a communication network 926 using a transmission medium via the network interface device 920 utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone (POTS) networks, and wireless data networks (e.g., Institute of Electrical and Electronics Engineers(IEEE) 802.11 family of standards known as WiFi®, IEEE 802.16 family of standards known as WiMax®), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or phonejacks) or one or more antennas to connect to the communication network 926. In an example, the network interface device 920 may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine 900, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.Additional Notes

[0112] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0113] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, asystem, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0114] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

What is claimed is:

1. An endoscope system, comprising: an endoscope, including an imaging device to obtain images or video streams of a target anatomy of a patient during an endoscopy procedure; and a controller circuit configured to: analyze the obtained images or video streams to identify an area of interest (AOI) in the target anatomy during the endoscopy procedure; based on the identification of the AOI, generate an endoscope maneuvering plan including a target or recommended imaging setting of the imaging device for subsequent imaging of the AOI; and generate real-time guidance, to a user or a robotic system, in maneuvering of the endoscope and imaging and photo-documenting the AOI in accordance with the generated endoscope maneuvering plan.

2. The endoscope system of claim 1, wherein the target or recommended imaging setting of the imaging device includes at least one of: a position, a posture, or a viewing angle of the imaging device with respect to the identified AOI; a zoom setting; a focus setting; a contrast setting; a viewing angle; a lighting parameter; or a centering setting to center the identified AOI or a portion thereof in a field of view (FOV) of the imaging device.

3. The endoscope system of claim 1, wherein the endoscope maneuvering plan includes guided navigation instructions or paths towards the AOI, wherein the real-time guidance is to guide the user or the robotic system to maneuver the endoscope in accordance with the guided navigation instructions or path.

4. The endoscope system of claim 3, wherein the guided navigation instructions or path are defined in a coordinate system.

5. The endoscope system of claim 1, wherein to generate the target or recommended imaging setting, the controller circuit is configured to generate an image quality indicator of the images or video streams produced under an existing imaging setting of the imaging device, and to determine whether to keep or adjust the existing imaging setting based at least in part on the image quality indicator.

6. The endoscope system of claim 1, wherein the AOI includes a predetermined anatomical landmark in the target anatomy.

7. The endoscope system of claim 6, wherein: the endoscope is a colonoscope, and the imaging device is configured to obtain images or video streams of one or more colon segments during withdrawal of the colonoscope in a colonoscopy procedure; and the controller circuit is configured to analyze the obtained images or video streams of the one or more colon segments to identify one or more predetermined colon landmarks during the colonoscopy procedure.

8. The endoscope system of claim 1, wherein the AOI includes an anomalous structure in the target anatomy, wherein to identify the AOI includes to identify a presence or absence, a type, a size, a shape, a location, or an amount of lesion or obstructed mucosa in the target anatomy.

9. The endoscope system of claim 1, wherein the AOI includes a blind spot region of the target anatomy, wherein the controller circuit is further configured to generate a three- dimensional (3D) reconstruction of the blind spot region using images or video streams produced in accordance with the target or recommended imaging setting.

10. The endoscope system of claim 1, further comprising a device tracking circuit configured to track location and orientation of the imaging device or a distal portion of the endoscope during the endoscopy procedure, wherein the controller circuit is configured to identify the AOI or to generate the endoscope maneuvering plan further based on the tracked location or orientation.

11. The endoscope system of claim 10, wherein the device tracking circuit is configured to track the location and orientation of the imaging device or the distal portion of the endoscope based on an electromagnetic property thereof detected during the endoscopy procedure.

12. The endoscope system of claim 10, wherein to perform location tracking, the device tracking circuit is configured to register the tracked location and orientation to a pre-generated template of the target anatomy during the endoscopy procedure.

13. The endoscope system of claim 1, wherein the controller circuit is configured to identify the AOI using a first trained machine-learning (ML) model, the first trained ML model trained to establish a correspondence between (i) an endoscopic image or video stream of the target anatomy or features extracted therefrom and (ii) one or more characteristics of an AOI.

14. The endoscope system of claim 1, wherein the controller circuit is configured to generate the endoscope maneuvering plan including the target or recommended imaging setting using a second trained machine-learning (ML) model, the second trained ML model trained to establish a correspondence between (i) an endoscopic image or video stream or the identified AOI or features extracted therefrom and (ii) an imaging setting of the imaging device.

15. The endoscope system of claim 1, wherein the controller circuit is further configured to generate an alert to the user about the identified AOI, and to receive a user feedback about the alert,wherein the controller circuit is configured to generate the endoscope maneuvering plan based at least in part on the user feedback about the alert.

16. The endoscope system of claim 1, further comprising the robotic system configured to, in response to a control signal from the controller circuit, robotically maneuver the endoscope in accordance with the endoscope maneuvering plan, and to produce guided imaging and documentation of the AOI in accordance with the target or recommended imaging setting.

17. A method of real-time endoscopy scene analysis and photodocumentation during an endoscopy procedure, the method comprising: obtaining images or video streams of a target anatomy during an endoscopy procedure using an imaging device associated with an the endoscope; analyzing the obtained images or video streams to identify an area of interest (AOI) in the target anatomy during the endoscopy procedure; based on the identification of the AOI, generating an endoscope maneuvering plan including a target or recommended imaging setting of the imaging device for subsequent imaging of the AOI; and generating real-time guidance, to a user or a robotic system, in maneuvering of the endoscope and imaging and photo-documentation of the AOI in accordance with the generated endoscope maneuvering plan.

18. The method of claim 17, wherein the endoscope maneuvering plan includes guided navigation instructions or paths towards the AOI, wherein the real-time guidance is to guide the user or the robotic system to maneuver the endoscope in accordance with the guided navigation instructions or path.

19. The method of claim 18, wherein the guided navigation instructions or path are defined in a coordinate system.

20. The method of claim 17, further comprising generating an image quality indicator of the images or video streams produced under an existing imagingsetting of the imaging device, and determining whether to keep or adjust the existing imaging setting based at least in part on the image quality indicator.

21. The method of claim 17, wherein the obtained images or video streams of the target anatomy includes images or video streams of one or more colon segments during withdrawal of an colonoscope during a colonoscopy procedure, wherein the AOI includes a pre-determined colon landmark or an anomaly in a colon segment.

22. The method of claim 17, wherein the AOI includes a blind spot region of the target anatomy, the method further comprising generating a three- dimensional (3D) reconstruction of the blind spot region using images or video streams produced in accordance with the target or recommended imaging setting.

23. The method of claim 17, further comprising tracking location and orientation of the imaging device or a distal portion of the endoscope during the endoscopy procedure, wherein the identification of the AOI or the generation of the endoscope maneuvering plan are further based on the tracked location or orientation.

24. The method of claim 17, wherein identifying the AOI includes using a first trained machine-learning (ML) model trained to establish a correspondence between (i) an endoscopic image or video stream of the target anatomy or features extracted therefrom and (ii) one or more characteristics of an AOI.

25. The method of claim 17, wherein generating the endoscope maneuvering plan includes using a second trained machine-learning (ML) model trained to establish a correspondence between (i) an endoscopic image or video stream or the identified AOI or features extracted therefrom and (ii) an imaging setting of the imaging device.

26. The method of claim 17, further comprising generating an alert to the user about the identified AOI, and receiving a user feedback about the alert,wherein generating the endoscope maneuvering plan is based at least in part on the user feedback about the alert.

27. The method of claim 17, further comprising robotically maneuvering the endoscope and imaging and photo-documenting of the AOI in accordance with the endoscope maneuvering plan.

Images