AI-based imaging mode selection in endoscopy

Abstract

Systems, devices, and methods are disclosed for determining or adjusting an imaging mode for examining tissue or foreign matter during an endoscopic procedure. An example endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging system to obtain an image or video stream of a target anatomical structure during an endoscopic procedure. The controller circuit may generate endoscopic image or video features characterizing anomalies in the target anatomy from the obtained image or video stream and determine personalized pathology-specific imaging modalities to enhance discernability of anomalies from the image or video stream. A recommended imaging mode may be determined using a trained machine learning model. The recommended imaging mode may be provided to a user or a processing device to facilitate manual or automatic endoscopic imaging mode adjustment during an endoscopic procedure.

Description

Priority Statement

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 582,029, filed on September 12, 2023, the contents of which are incorporated herein by reference. Technical Field

[0002] This document relates generally to endoscopic systems, and more specifically to systems and methods for determining or adjusting imaging patterns for examining tissues or foreign bodies during endoscopic examinations. Background Technology

[0003] Endoscopy has been used in a variety of clinical procedures, including, for example: illuminating, imaging, detecting, and diagnosing one or more disease states; providing fluid delivery toward anatomical regions (e.g., delivering saline or other preparations via a fluid channel); providing access for sampling or treating anatomical regions for one or more therapeutic devices or biomaterial collection devices (e.g., via a working channel); and providing aspiration access for collecting fluids (e.g., saline or other preparations), among other procedures. Examples of such anatomical regions may include the gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary ducts, intestines, colon, etc.), the renal region (e.g., kidneys, ureters, bladder, urethra), and other internal organs (e.g., the reproductive system, sinuses, submucosal regions, respiratory tract), etc.

[0004] Some endoscopes include a working channel through which the operator can perform aspiration, placement of diagnostic or therapeutic devices (e.g., brushes, biopsy needles or forceps, stents, baskets, or balloons), or minimally invasive procedures such as tissue sampling or removal of unwanted tissue (e.g., benign or malignant strictures) or foreign bodies (e.g., stones). Some endoscopes can be used with laser or plasma systems to deliver energy to anatomical targets (e.g., soft or hard tissue or stones) to achieve the desired treatment. For example, lasers have been used in tissue ablation, coagulation, vaporization, fragmentation, and lithotripsy to break down stones in the kidneys, gallbladder, ureters, and other areas of stone formation, or to ablate larger stones into smaller fragments. An example of endoscopy is colonoscopy, used to reduce the incidence and mortality of colorectal cancer. Colonoscopy is typically performed by rapidly advancing the colonoscope to the cecum and then thoroughly examining it during withdrawal to detect abnormalities (e.g., polyps) and necessary treatments (e.g., polypectomy).

[0005] Endoscopes typically include imaging sensors (e.g., camera devices) that acquire real-time images or video streams during endoscopic procedures. Imaging sensors can operate in preset imaging modes to obtain real-time images or video streams. Imaging modes can include illumination modes, optical magnification, the field of view of the imaging sensor, and other settings for the imaging sensor and illumination system. Different imaging modes have been used for real-time endoscopy and optical diagnosis, such as high-definition white light imaging (WLI), dye-based chromoendoscopy, or virtual CE such as narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichroism imaging (RDI), etc. Recommended optical magnification is used to examine lesions or pathologies with different characteristics. Appropriate selection of imaging modes can improve image or video quality and facilitate the examination and diagnosis of foreign bodies or abnormal tissues of interest during the procedure. Summary of the Invention

[0006] Various specialized imaging modalities exist and have been used to enhance the detection and / or diagnosis of certain types of pathology or abnormalities during endoscopic procedures. For example, image-enhanced endoscopy (IEE) has enabled better detection and management of colorectal cancer and other pathologies. Although specialized imaging modalities are currently recommended based on the type of pathology being examined, switching between such modalities during endoscopy remains highly manual and relies on the clinician to both identify the current type of pathology being examined and recall which imaging modality was recommended for viewing such pathology. The high dependence on operator preference and experience in interpreting these advanced endoscopic modalities can lead to variability in diagnostic and treatment outcomes. While training in advanced endoscopic imaging could potentially improve overall abnormality detection and lead to more accurate optical diagnoses and effective treatments, training can be costly. The use of IEE in colonoscopy remains operator-dependent and requires extensive specialized training.

[0007] The inventors of this disclosure have identified an unmet need for real-time recommendations of personalized, pathology-specific imaging modalities for examining tissues or foreign bodies of interest during endoscopic procedures. This document describes artificial intelligence (AI)-based systems and methods for determining and recommending imaging modalities, such as IEE patterns, based on suspicious abnormalities identified through real-time computer-performed analysis of endoscopic images or video streams of a target anatomical structure. An exemplary endoscopic system includes an endoscope and controller circuitry. The endoscope includes an imaging system to acquire images or video streams of a target anatomical structure during an endoscopic procedure. The controller circuitry can generate endoscopic image or video features characterizing abnormalities in the target anatomical structure based on the acquired image or video stream and determine personalized, pathology-specific imaging modalities to enhance the discernibility of abnormalities from the image or video stream. Machine learning (ML) techniques can be used to determine the recommended imaging modalities. The recommended imaging modalities can be provided to the user or processing device to facilitate manual or automatic adjustment of the endoscopic imaging modalities during the endoscopic procedure.

[0008] The systems, devices, and methods described herein can be used in a variety of endoscopic procedures to improve the real-time examination and detection of pathology and diagnosis. These systems, devices, and methods can also help reduce variability in operator experience and / or preferences, resulting in more consistent and predictable diagnoses and treatments, while reducing the costs associated with intensive training. The systems and techniques described herein also facilitate the adoption of advanced IEE technologies, thereby improving overall endoscopic procedure success rates and patient outcomes.

[0009] Example 1 is an endoscopy system comprising: an endoscope including an imaging system configured to acquire images or video streams of a target anatomical structure in a patient during an endoscopic examination; and controller circuitry configured to: analyze the acquired image or video streams to generate endoscopic image or video features characterizing abnormalities in the target anatomical structure; automatically determine a target or recommended imaging mode of the imaging system, at least in part based on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, to enhance the discriminability of abnormalities from the image or video stream of the target anatomical structure in subsequent imaging of the target anatomical structure; and provide the target or recommended imaging mode to a user or robotic system to facilitate manual or automatic adjustment of the endoscopic imaging mode during the endoscopic examination.

[0010] In Example 2, the subject of Example 1 may optionally include a target or recommended imaging mode, which may include one or more of an illumination mode, a zoom setting, or a perspective relative to an anomalous viewpoint.

[0011] In Example 3, the subject of any one or more of Examples 1 to 2 may optionally include a target or recommended imaging mode, which may include one of a narrow-band imaging (NBI) mode, a red dichroic imaging (RDI) mode, a white light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

[0012] In Example 4, the subject matter of any one or more of Examples 1 to 3 may optionally include a colonoscopy, wherein the imaging system is configured to acquire an image or video stream of each of the different colonic segments during a colonoscopy procedure, wherein the controller circuitry is configured to determine a corresponding target or recommended imaging mode for use in subsequent imaging of the different colonic segments to enhance the discernibility of abnormalities from the image or video stream.

[0013] In Example 5, the subject matter of any one or more of Examples 1 to 4 may optionally include controller circuitry that can be configured to: detect abnormalities in the target anatomical structure using endoscopic image or video features; and determine the target or recommended imaging mode based at least in part on the results of the abnormality detection.

[0014] In Example 6, the subject matter of Example 5 may optionally include, wherein the detection of abnormalities includes the presence or absence of the detection of abnormalities and one or more of the type, size, shape, location, or number of pathological tissue or obstructing mucosa in the target anatomical structure.

[0015] In Example 7, the subject of any one or more of Examples 5 to 6 may optionally include controller circuitry that can be configured to detect anomalies using a first trained machine learning (ML) model.

[0016] In Example 8, the subject matter of any one or more of Examples 1 to 7 may optionally include controller circuitry that can be configured to: determine the position of the endoscope in the target anatomical structure substantially in real time, at least in part based on endoscopic image or video features; register the position of the endoscope to a pre-generated template of the target anatomical structure; and display the position of the endoscope in the target anatomical structure on a user interface.

[0017] In Example 9, the subject of Example 8 may optionally include controller circuitry that can be configured to identify anatomical landmarks using endoscopic images or video features and determine the position of the endoscope based on the identified anatomical landmarks.

[0018] In Example 10, the subject matter of any one or more of Examples 8 to 9 may optionally include controller circuitry that can be configured to further determine the target or recommended imaging mode using the determined position of the endoscope in the target anatomical structure.

[0019] In Example 11, the subject matter of any one or more of Examples 1 to 10 may optionally include, in order to determine a target or recommended imaging mode, the controller circuitry is configured to apply endoscopic image or video features to a second trained machine learning (ML) model, which is trained to establish a correspondence between endoscopic image or video features and one of a set of candidate imaging modes.

[0020] In Example 12, the subject of Example 11 may optionally include a second trained ML model, which may also be trained using one or more auxiliary information features, said one or more of the following: a profile of the endoscopist performing the endoscopic procedure; patient information and patient medical history data; endoscopic system setup information; or pre-procedure imaging study data, wherein, in order to determine the target or recommended imaging mode, the controller circuitry is configured to apply auxiliary inputs and enhanced inputs, including endoscopic image or video features, to the second trained ML model.

[0021] In Example 13, the subject of any one or more of Examples 11 to 12 may optionally include a second trained ML model, which can be trained to predict for each of a plurality of candidate imaging modes: a probability representing the likelihood that the corresponding imaging mode is selected as the target or recommended imaging mode, wherein the controller circuitry is configured to determine the target or recommended imaging mode based at least in part on the probability corresponding to the candidate imaging modes.

[0022] In Example 14, the subject matter of any one or more of Examples 1 to 13 may optionally include controller circuitry that, in response to a target or recommended imaging mode being different from an existing imaging mode used to acquire an image or video stream of the target anatomical structure, the controller circuitry may be configured to provide a user with a recommendation to switch to the target or recommended imaging mode to recapture an image or video stream of the target anatomical structure.

[0023] In Example 15, the subject matter of any one or more of Examples 1 to 14 may optionally include controller circuitry that, in response to a target or recommended imaging mode being different from an existing imaging mode for acquiring an image or video stream of the target anatomical structure, the controller circuitry may be configured to generate a control signal to cause the imaging system to automatically switch to the target or recommended imaging mode and recapture the recommended image or video stream of the target anatomical structure.

[0024] Example 16 is a method for determining or adjusting an endoscopic imaging modality during an endoscopic examination, the method comprising the steps of: acquiring an image or video stream of a target anatomical structure during the endoscopic examination using an imaging system associated with an endoscope; analyzing the acquired image or video stream to generate endoscopic image or video features characterizing abnormalities in the target anatomical structure; automatically determining a target or recommended imaging modality of the imaging system, at least in part based on the endoscopic image or video features and one or more auxiliary features different from the endoscopic image or video features, to enhance the discriminability of abnormalities from the image or video stream of the target anatomical structure in subsequent imaging of the target anatomical structure; and providing the target or recommended imaging modality to a user or robotic system to facilitate manual or automatic adjustment of the endoscopic imaging modality during the endoscopic examination.

[0025] In Example 17, the subject of Example 16 may optionally include a target or recommended imaging mode, which may include one or more of an illumination mode, a zoom setting, or a viewpoint relative to the anomaly.

[0026] In Example 18, the subject matter of any one or more of Examples 16 to 17 may optionally include a target or recommended imaging mode, which may include one of a narrow-band imaging (NBI) mode, a red dichroic imaging (RDI) mode, a white light imaging (WLI) mode, or a texture and image enhancement (TXI) mode.

[0027] In Example 19, the subject matter of any one or more of Examples 16 to 18 may optionally include: detecting abnormalities in a target anatomical structure using endoscopic image or video features; and determining a target or recommended imaging modality based at least in part on the results of abnormality detection.

[0028] In Example 20, the subject matter of any one or more of Examples 16 to 19 may optionally include: identifying anatomical landmarks using endoscopic image or video features; determining the position of an endoscope in a target anatomical structure substantially in real time, at least in part based on the identified anatomical landmarks; registering the position of the endoscope to a pre-generated template of the target anatomical structure; and displaying the position of the endoscope in the target anatomical structure on a user interface, wherein the determination of the target or recommended imaging pattern is also based on the determined position of the endoscope in the target anatomical structure.

[0029] In Example 21, the subject matter of any one or more of Examples 16 to 20 may optionally include determining a target or recommended imaging pattern, which may include applying endoscopic image or video features to a second trained machine learning (ML) model trained to establish a correspondence between endoscopic image or video features and a candidate imaging pattern from a set of candidate imaging patterns.

[0030] In Example 22, the subject of Example 21 may optionally include a second trained ML model, which may also be trained using one or more auxiliary feature information, said one or more of the following: a profile of the endoscopist performing the endoscopic procedure; patient information and patient medical history data; endoscopic system setup information; or pre-procedure imaging study data, wherein determining the target or recommended imaging mode includes applying auxiliary inputs and enhanced inputs including endoscopic image or video features to the second trained ML model.

[0031] In Example 23, the subject of Example 22 may optionally include a second trained ML model, which can be trained to predict for each of a plurality of candidate imaging modes: a probability representing the likelihood that the corresponding imaging mode is selected as the target or recommended imaging mode, wherein determining the target or recommended imaging mode includes identifying a candidate imaging mode among the plurality of candidate imaging modes whose corresponding probability is higher than any other candidate imaging mode among the candidate imaging modes.

[0032] In Example 24, the subject matter of any one or more of Examples 16 to 23 may optionally include providing a recommendation to the user to switch to the target or recommended imaging mode to recapture the image or video stream of the target anatomical structure in response to a target or recommended imaging mode that differs from an existing imaging mode used to obtain an image or video stream of the target anatomical structure.

[0033] In Example 25, the subject matter of any one or more of Examples 16 to 24 may optionally include generating a control signal in response to a target or recommended imaging mode that differs from an existing imaging mode used to acquire images or video streams of the target anatomical structure, so that the imaging system automatically switches to the target or recommended imaging mode and recaptures images or video streams of the target anatomical structure.

[0034] The techniques presented are described based on the controlled withdrawal of the endoscope during colonoscopy, but are not limited thereto. The systems, apparatus, and techniques described according to various embodiments in this document can be additionally or alternatively used in other procedures involving different types of endoscopes, including, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, colposcopy, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy, etc.

[0035] This disclosure is an overview of some of the teachings in this application and is not intended as an exclusive or exhaustive treatment of the subject matter. Further details regarding the subject matter are found in the detailed description and the appended claims. Other aspects of this disclosure will become apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the accompanying drawings, which form a part of this disclosure, and each of these aspects should not be considered limiting. The scope of this disclosure is defined by the appended claims and their legal equivalents. Attached Figure Description

[0036] Figures 1 to 2 This is a schematic diagram illustrating an example of an endoscopic system used in endoscopic procedures, such as colonoscopy.

[0037] Figure 3 An example of an endoscope system is shown, which is configured to determine a target or recommended imaging pattern for examining tissue or foreign body of interest during an endoscopic examination.

[0038] Figure 4 An example of auxiliary input that can be used to determine a target or a recommended imaging pattern is shown.

[0039] Figure 5 The different stages of image-guided colonoscopy and automatic imaging modality switching based on abnormal characteristics are illustrated by example.

[0040] Figure 6 Example images of a portion of the colon are shown in different imaging modes.

[0041] Figures 7A to 7B An example is shown of training a machine learning (ML) model and using the trained ML model to determine recommended pathology-specific imaging modalities for anomaly examination and diagnosis.

[0042] Figure 8 This is a flowchart illustrating an example method for determining or adjusting the endoscopic imaging mode during an endoscopic examination.

[0043] Figure 9This is a block diagram illustrating an example machine on which any or more of the techniques (e.g., methods) discussed herein can be performed. Detailed Implementation

[0044] This document describes systems, apparatus, and methods for determining or adjusting endoscopic imaging modalities during endoscopic examinations. An exemplary endoscopic system includes an endoscope and controller circuitry. The endoscope includes at least one imaging sensor to acquire images or video streams of a target anatomical structure of a patient during the endoscopic examination. The controller circuitry can perform real-time analysis of the endoscopic images or video streams and generate endoscopic image or video features characterizing abnormalities in the target anatomical structure. Based on the endoscopic image or video features, the controller can determine a target or recommended pathology-specific imaging modality to enhance the discernibility of abnormalities from the image or video stream of the target anatomical structure. A machine learning (ML) model can be trained and used to determine the target or recommended imaging modality. The target or recommended imaging modality can be provided to the user or processing device to facilitate manual or automatic adjustment of the endoscopic imaging modality during the endoscopic examination.

[0045] Figure 1 This is a schematic diagram of an endoscope system 10 used in endoscopic procedures such as colonoscopy. System 10 may include an imaging and control system 12 and an endoscope 14. System 10 is an illustrative example of an endoscope system suitable for use with the systems, apparatus, and methods described herein, such as a colonoscopy system for image-guided colonoscopy in an automatically adjusted imaging modality as described in this document.

[0046] Endoscope 14 may be inserted into an anatomical region to provide access for imaging or for one or more sampling devices for biopsy or for treatment devices for treating disease conditions associated with the anatomical region, or may be attached (e.g., via tethering) to said one or more sampling devices or said treatment devices. Endoscope 14 may interface with and be connected to imaging and control system 12. Endoscope 14 may be a colonoscope; however, other types of endoscopes may be used in conjunction with the features and teachings of this disclosure. Imaging and control system 12 may include control unit 16, output unit 18, input unit 20, light source unit 22, fluid source 24, and suction pump 26.

[0047] The imaging and control system 12 may include various ports for coupling with the endoscope system 10. For example, the control unit 16 may include a data input port for receiving data from the endoscope 14 and a data output port for transmitting data to the endoscope 14. The light source unit 22 may include an output port for transmitting light to the endoscope 14, for example, via an optical fiber link. The fluid source 24 may include a port for transmitting fluid to the endoscope 14. The fluid source 24 may include, for example, a pump and a fluid tank, or may be connected to an external tank, container, or storage unit. The suction pump 26 may include a port for evacuating the endoscope 14 to generate suction, for example, for withdrawing fluid from an anatomical region in which the endoscope 14 is inserted. The output unit 18 and the input unit 20 may be used by the operator of the endoscope system 10 to control the functions of the endoscope system 10 and to view the output of the endoscope 14. The control unit 16 may also generate signals or other outputs from treatment of the anatomical region in which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical output, acoustic output, fluid output, etc., for use in treating anatomical areas using methods such as cauterization, cutting, freezing, etc.

[0048] Fluid source 24 can communicate with control unit 16 and may include one or more air sources, saline sources, or other fluid sources, as well as associated fluid passages (e.g., air passages, flushing passages, suction passages, etc.) and connectors (barbed fittings, fluid seals, valves, etc.). Fluid source 24 can be used as activation energy for biasing or pressure application devices used in this disclosure. Imaging and control system 12 may also include drive unit 46, which may include a motorized actuator for advancing the distal end of endoscope 14.

[0049] Endoscope 14 may include an insertion portion 28, a functional portion 30, and a handle portion 32, which may be coupled to a cable portion 34 and a coupler portion 36. The insertion portion 28 may extend distally from the handle portion 32, and the cable portion 34 may extend proximally from the handle portion 32. The insertion portion 28 may be elongated and may include a bend and a distal end to which the functional portion 30 may be attached. The bend may be controllable (e.g., via a control knob 38 on the handle portion 32) to manipulate the distal end through tortuous anatomical pathways (e.g., the stomach, duodenum, kidney, ureter, etc.). The insertion portion 28 may also include one or more working channels (e.g., lumens), which may be elongated and may support the insertion of one or more therapeutic instruments of the functional portion 30. The working channels may extend between the handle portion 32 and the functional portion 30. The insertion portion 28 may also provide additional functionality (e.g., via aspiration or flushing pathways, etc.), such as fluid access, guidewires, and traction lines.

[0050] The coupler section 36 can be connected to the control unit 16 to connect the endoscope 14 to various features of the control unit 16, such as the input unit 20, the light source unit 22, the fluid source 24, and the suction pump 26.

[0051] The handle portion 32 may include a knob 38 and a port 40A. The knob 38 may be connected to a traction cable or other actuating mechanism that can extend through the insertion portion 28. Port 40A and other ports such as port 40B (… Figure 2 It can be configured to couple various cables, guide wires, auxiliary observation instruments, tissue collection devices, fluid tubes, etc. to the handle portion 32, for example, to the insertion portion 28.

[0052] According to the example, the imaging and control system 12 can be mounted on a mobile platform (e.g., a trolley 41) having a design for housing the light source unit 22, the suction pump 26, and the image processing unit 42. Figure 2 Shelves, etc. Alternatively, several components of the imaging and control system 12 ( Figure 1 and Figure 2 (As shown) can be directly mounted on endoscope 14 so that the endoscope is "self-contained".

[0053] Functional unit 30 may include components for treating and diagnosing a patient's anatomy. Functional unit 30 may include an imaging device, an illumination device, and a lift. Functional unit 30 may also include optically enhanced biomaterial and tissue collection and retrieval devices. For example, functional unit 30 may include one or more electrodes electrically connected to handle portion 32 and functionally connected to imaging and control system 12 to analyze biomaterial in contact with the electrodes based on comparative biodata stored in imaging and control system 12. In other examples, functional unit 30 may be directly integrated with a tissue collector.

[0054] In some examples, endoscope 14 can be robotically controlled, for example, via a robotic arm attached thereto. The robotic arm can automatically or semi-automatically (e.g., using some user manual control or command) position and navigate endoscope 14 (e.g., functional section 30 and / or insertion section 28) within the target anatomical structure via actuators, or position the device in a desired pose to facilitate manipulation of the anatomical target. According to the various examples discussed in this document, a controller can generate control signals to the actuators of the robotic arm to facilitate the examination and diagnosis of abnormalities in the target or recommended imaging mode during robot-assisted endoscopy.

[0055] Figure 2 yes Figure 1 A schematic diagram of an endoscope system 10, which includes an imaging and control system 12 and an endoscope 14. Figure 2Components of an imaging and control system 12 coupled to an endoscope 14, which in the illustrated example includes a colonoscope, are schematically shown. The imaging and control system 12 may include a control unit 16, a light source unit 22, an input unit 20, and an output unit 18. The control unit 16 may include or be coupled to an image processing unit 42, a treatment generator 44, and a drive unit 46. The control unit 16 may include, or communicate with, an endoscope, surgical instruments 48, and an endoscopic system. The endoscopic system may include means configured to engage tissue and collect and store a portion of that tissue, and an imaging device (e.g., a camera) may be used to view the target tissue via means including optically enhanced materials and components. The control unit 16 may be configured to activate the camera to view the target tissue distal to the endoscopic system. Similarly, the control unit 16 can be configured to enable the light source unit 22 to illuminate the surgical instrument 48, which may include selection components configured to reflect light in a particular manner, such as an enhanced tissue cutter with reflective particles.

[0056] The coupler section 36 can be connected to the control unit 16 to connect the endoscope 14 to various features of the control unit 16, such as the image processing unit 42 and the treatment generator 44. In the example, port 40A can be used to insert another surgical instrument 48 or device (e.g., a sub-observation instrument or auxiliary observation instrument) into the endoscope 14. Such instruments and devices can be independently connected to the control unit 16 via cable 47. In the example, port 40B can be used to connect the coupler section 36 to various inputs and outputs, such as video, air, light, and electricity.

[0057] The image processing unit 42 and the light source unit 22 can each interface with the endoscope 14 (e.g., at the functional section 30) via a wired or wireless connection. Therefore, the imaging and control system 12 can illuminate the anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display an image representing the anatomical region on the display unit 18. The imaging and control system 12 may include the light source unit 22 to illuminate the anatomical region using light of a desired spectrum (e.g., broadband white light, narrowband imaging using a preferred electromagnetic wavelength, etc.). The imaging and control system 12 can be connected to the endoscope 14 (e.g., via an endoscope connector) for signal transmission (e.g., light output from the light source, video signals from the imaging system in the distal end, diagnostic and sensor signals from the diagnostic device, etc.).

[0058] The treatment generator 44 can generate a treatment plan that can be used by the control unit 16 to control the operation of the endoscope 14 during an endoscopic examination or to provide guidance to the operator for manipulating the endoscope 14. In this example, the treatment generator 44 can use patient information, including images of the target anatomy, to generate an endoscopic navigation plan that includes estimates of one or more cannulation or navigation parameters (e.g., angles, forces, etc.) for manipulating a steerable, elongated instrument. The endoscopic navigation plan can help guide the operator in cannulating and navigating the endoscope within the patient's anatomy. The endoscopic navigation plan can be additionally or alternatively used to adjust the position, angle, force, and / or navigation of the endoscope or other instruments robotically.

[0059] Figure 3 This is a block diagram illustrating an example of an endoscope system 300 that can determine a target or recommended imaging modality for examining tissue or foreign bodies of interest during an endoscopic examination. The target or recommended imaging modality can help improve the real-time examination, identification, and optical diagnosis of various pathologies. By way of example, and not limitation, the system 300 can be used in colonoscopy procedures to better detect and manage abnormalities such as polyps or colorectal cancer. The system 300 can be implemented as... Figure 1 It is part of the control unit 16.

[0060] System 300 may include one or more of an endoscope 310, an auxiliary input 315, a controller circuit 320, a user interface 330, and a storage device 340. In some examples, system 300 may also include a robot system 350 for robot-assisted endoscopic examination or be communicatively coupled to such robot system 350.

[0061] Endoscope 310 can be as described above and in Figures 1 to 2 An example of endoscope 14 is shown. Endoscope 310 may include an imaging system 312 and an illumination system 314, etc. Imaging system 312 may include at least one imaging sensor or device (e.g., a camera device) configured to acquire images or video streams of a patient's target anatomical structure during an endoscopic examination. The imaging sensor or device may be located in the distal portion or distal end of endoscope 310. Illumination system 314 may include one or more light sources to produce illumination on the target anatomical structure via one or more illumination lenses.

[0062] Imaging system 312 can be controllably adjusted to operate at different settings, including zoom settings, contrast settings, exposure levels, or viewing angles toward the target anatomical structure. Illumination system 314 can be controllably adjusted to provide different lighting or illumination conditions. Imaging system 312 and illumination system 314 can together define an imaging mode or modality for capturing endoscopic images or video streams of the target anatomical structure. In this document, imaging mode or modality can include one or more of illumination modalities, optical magnification, or the viewing angle of the imaging device. Examples of imaging or illumination modalities can include high-definition white light imaging (WLI), such as dye-based pigment endoscopy, or virtual CE such as narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichroism imaging (RDI), etc. Optical magnification defines zoom settings (e.g., reduction or magnification of suspicious abnormalities in the target anatomical structure). The viewing angle (also called the field of view) of the imaging device describes the angular range of a given scene imaged by the imaging device.

[0063] Controller circuitry 320 may include a set of circuits comprising one or more other circuits or sub-circuits that can individually or in combination perform the functions, methods, or techniques described herein. In an example, controller circuitry 320 and the set of circuits therein may be implemented as part of a microprocessor circuit, which may be a dedicated processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a microprocessor, or other type of processor for processing information including bodily activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that can receive and execute a set of instructions for performing the functions, methods, or techniques described herein. In an example, the hardware of the circuit set may be invariably designed to perform a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) to encode instructions for a specific operation, said variably connected physical components including computer-readable media that are physically modified (e.g., magnetically, electrically, with movable placement of invariant aggregated particles, etc.). When physical components are connected, the underlying electrical characteristics of the hardware composition are altered, for example, from an insulator to a conductor or vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of a circuit set within the hardware via variable connections to perform specific operations during operation. Thus, while the device is operating, a computer-readable medium is communicatively coupled to other components of the circuit set member. In the example, any component of the physical components can be used in more than one member of more than one circuit set. For example, during operation, an execution unit can be used at one point in time in a first circuit of a first circuit set and reused at different times by a second circuit of the first circuit set or by a third circuit of the second circuit set.

[0064] The controller circuit 320 can determine a personalized pathology-specific imaging modality substantially in real time during an endoscopic examination, using at least endoscopic images or video streams or features extracted from them. As mentioned above, imaging modality plays a crucial role in determining image or video quality during endoscopy and thus in the accuracy and efficiency of optical detection, diagnosis, and treatment of various types of pathologies. Conventionally, determining the desired or “best” imaging modality best suited for detecting or diagnosing certain pathologies is a highly manual process. It typically requires the user (e.g., an endoscopist) to manually switch between available imaging modalities, identify the current type of pathology being examined, and recall which imaging modality was used to examine such a pathology. Based on visual examination, the user manually adjusts the imaging or illumination system to achieve the desired illumination modality, optical magnification, or viewing angle. Such manual processing is time-consuming and can be laborious, and may introduce unpredictable inter-user variations. The system 300 described herein can automate the imaging modality selection and optimization process using artificial intelligence (AI) or machine learning (ML) based techniques, as described in further detail below.

[0065] The controller circuit 320 may include one or more of an image processor 321, an anomaly detector circuit 322, an endoscope positioning circuit 323, and an image mode selector circuit 324. The image processor 321 may analyze the image or video stream acquired from the imaging system 312 and generate endoscopic image or video features. Examples of image or video features include statistical features of pixel values, or morphological features such as corners, edges, spots, curvature, acceleration robust features (SURF), or scale-invariant feature transform (SIFT) features. In some examples, the image processor 321 may preprocess the image or video stream, such as filtering, resizing, or orienting, or color or grayscale correction, and may extract endoscopic features from the preprocessed image or video stream. In some examples, the image processor 321 may post-process the image features to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0066] The anomaly detector circuit 322 can detect abnormalities at a target anatomical structure based at least in part on features from endoscopic images or videos. Anomaly detection includes detecting and / or identifying one or more of the presence, type, size, shape, location, or number of pathological tissue or anatomical structures and other objects in the environment of the anatomical structure. In an example of colonoscopy, detected abnormalities may include segments of pathological tissue, such as mucosal abnormalities (polyps, inflammatory bowel disease, Merkel's diverticulum, lipomas, bleeding, vascularized mucosa, etc.) or obstructive mucosa (e.g., segments with poor bowel preparation, colonic dilatation, etc.).

[0067] In one example, the anomaly detector circuit 322 can use template matching techniques to detect anomalies, where anomalies can be identified based on comparisons of endoscopic features (e.g., features characterizing the shape or contour of a structure) with one or more pre-generated templates of known anomalous structures. In another example, the anomaly detector circuit 322 can use AI or ML-based techniques to detect anomalies, where features from or extracted from endoscopic images or video streams can be applied to a trained ML model to automatically identify the presence or absence, type, size, location, and / or other characteristics of anomalies. The ML model can be trained to establish correspondences between features from or extracted from endoscopic images or video streams and one or more anomalous characteristics. Examples of ML models used to identify anomalies from endoscopic images or video streams include convolutional neural networks, bidirectional LSTMs, recurrent neural networks, conditional random fields, dictionary learning or other machine learning techniques (support vector machines, Bayesian models, decision trees, k-means clustering), and other ML techniques. The trained ML model can be stored in storage device 340.

[0068] Endoscope positioning circuit 323 can determine the position of the endoscope within a specific segment of an anatomical structure in essentially real-time. Using this real-time endoscope position information, anomaly detector circuit 322 can associate detected anomalies with segments of the anatomical structure. Endoscope position can be determined using electromagnetic tracking, or by anatomical landmarks detected from image or video streams acquired during the insertion phase of an endoscopy, or by image or video stream features extracted by image processor 321. In an example of colonoscopy, endoscope positioning circuit 323 can identify colonic landmarks from the image or video stream, such as the anus, left ascending colon, splenic flexure, transverse colon, hepatic flexure, right descending colon, cecum, appendix, and terminal ileum. In examples, endoscope positioning circuit 323 can use a template matching algorithm to identify landmarks. In some examples, endoscope positioning circuit 323 can use AI or ML methods, such as a trained ML model, to identify landmarks, which has been trained to establish a correspondence between endoscopic images or video streams or features extracted from them and landmark recognition. Examples of ML models used for identifying anatomical landmarks include deep belief networks, ResNet, DenseNet, autoencoders, capsule networks, generative adversarial networks, Siamese networks, convolutional neural networks (CNNs), deep reinforcement learning, support vector machines (SVMs), Bayesian models, decision trees, k-means clustering, and other ML models. The trained ML model can be stored in storage device 340. Once the endoscope position is determined, endoscope localization circuitry 323 can register the endoscope position, essentially in real-time, to a pre-generated template of the anatomical structure. Endoscope position information within segments of the anatomical structure can be presented to the user on user interface 330.

[0069] The imaging mode selector circuit 324 can determine a personalized pathology-specific imaging modality using at least information about detected anomalies (e.g., generated by the anomaly detector circuit 322) and the substantially real-time endoscope position when the anomaly is detected (e.g., generated by the endoscope positioning circuit 323). The personalized pathology-specific imaging modality can include illumination modality, optical magnification (i.e., the zoom setting of the imaging system 312), or the field of view of the imaging system 312. Examples of illumination modalities can include high-definition white light imaging (WLI), such as dye-based chromoendoscopy, or virtual CE such as narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichroism imaging (RDI), etc. Optical magnification defines the zoom setting (e.g., reduction or magnification of suspicious anomalies in the target anatomical structure). The field of view (also called the field of view) of the imaging device describes the angular range of a given scene imaged by the imaging device.

[0070] In the example, AI- or ML-based techniques can be used to determine personalized pathology-specific imaging modalities. In the example, information about detected abnormalities, landmarks, and substantially real-time endoscopic positioning can be applied to a trained ML model 360. The ML model 360 can be trained to establish a correspondence between abnormal characteristics and target or recommended imaging modalities. Once trained, the trained ML model 360 can be stored in storage device 340. The following is about... Figure 6 Examples of training ML models and using trained ML models to determine personalized pathology-specific imaging modalities are discussed.

[0071] In some examples, the imaging mode selector circuit 324 can calculate anomaly scores based on one or more anomalous characteristics such as the type, size, shape, location, or number of anomalies. Anomaly scores can have numerical values, for example, in the range of 0 to 10. In some examples, composite anomaly scores can be generated, for example, using linear or nonlinear combinations of multiple anomaly scores, each quantifying anomaly characteristics. The imaging mode selector circuit 324 can map anomaly scores to imaging modes (e.g., illumination modality, optical magnification, or the field of view of the imaging system 312) based on comparisons with one or more thresholds or value ranges, such that higher anomaly scores, which typically indicate more severe anomalies, can be mapped to more intense illumination modalities or higher magnifications. Established correspondences between anomaly scores or ranges of anomalies and corresponding imaging modes can be determined by a lookup table in a specification database or via a rule-based system. The established mappings can be stored in storage device 340. In another example, the imaging mode selector circuit 324 can apply anomaly scores to a trained ML model 360 to determine personalized pathology-specific imaging modalities. In some examples, anomaly score calculation can be included in the ML model, for example, in a layer of a neural network model.

[0072] In some examples, the imaging mode selector circuit 324 can also use auxiliary input 315 to determine a personalized pathology-specific imaging modality. (See reference...) Figure 4 By way of example, and not limitation, auxiliary input 315 may include image or video streams and clinical data 410 from previous endoscopic procedures performed on the patient, patient information and medical history 420, or pre-procedure imaging study data 430 (e.g., X-ray or fluorescence images, potential or impedance maps, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and other imaging modalities). In the example of colonoscopy, auxiliary input may include previous colonoscopies in a monitoring case, which includes polyps left in situ by the endoscopist or the colonic region operated on. In a typical endoscopic scenario, patient information and medical history 420 may include clinical demographic information, past and current indications, and treatments received, etc.

[0073] In some examples, auxiliary input 315 may additionally or alternatively include a user (e.g., endoscopist) profile 440, which includes the user's experience, work environment (e.g., hospital setting or outpatient screening center), affinity for new technologies, and preferences for specific endoscopic procedures or imaging modalities. The user profile 440 may be provided by the user via user interface 330. Alternatively, the user profile 440 may be automatically generated or updated by learning from the user's past choices and training.

[0074] In the example, auxiliary input 315 may additionally or alternatively include endoscope and device information 450. This may include, for example: the specifications of the endoscope 310, including the type, size, dimensions, shape, and structure of the endoscope or other steerable maneuverable instruments such as cannulas, catheters, or guidewires that support imaging and illumination modes (e.g., NBI, RDI, WLI, TXI, etc.); the specifications of the size, dimensions, shape, and structure of tissue sections, sampling, or treatment tools; and the current status of the device, including which illumination mode is activated, which endoscopic buttons (e.g., water jet, air delivery) are used, which illumination modes are supported, whether magnification is enabled, and the current magnification selection.

[0075] If additional AI or ML algorithms are running in the background, the state of these AI or ML algorithms (collectively referred to below as "AI Discovery" 460) can be an additional element that can be passed to the auxiliary input 315 of the imaging mode selector circuit 324. Examples of AI Discovery 460 may include AI algorithms for detecting anomalies, landmarks, or other features or structural elements of interest in the target anatomical structure.

[0076] Information regarding the personalized, pathology-specific imaging modality determined by the imaging modality selector circuit 324 can be displayed on the user interface 330 to assist the endoscopist during endoscopic procedures, such as image-guided colonoscopy. If the determined imaging modality differs from the current imaging modality used to examine and detect abnormalities, this difference can be notified or alerted to the user (e.g., the endoscopist) via the user interface 330. Notification can be delivered optically on the diagnostic monitor or audibly via a warning alarm. In the example, highlighting, flashing alarms, audible, or tactile feedback can be provided to the user. See below. Figure 5 Examples of automatically switching between pathology-specific imaging modalities are discussed. Additional information can also be displayed on the user interface 330, including endoscopic images and image features, information about abnormalities and landmarks detected in each segment of the anatomical structure, or the substantially real-time position of the endoscope during the endoscopic procedure. Upon exit (e.g., rectum detected), information about detected and treated abnormalities, total net exit time, and other information can be generated in a post-procedure summary. Such post-procedure analysis can be used for quality assurance and to determine whether the colonoscopy procedure was successful.

[0077] In some examples, personalized pathology-specific imaging modalities may be stored in storage device 340. Information about detected anomalies may also be stored in storage device 340. Storage device 340 may be local to the endoscope system 300. Alternatively, storage device 340 may be a remote storage device, such as part of a cloud that includes one or more storage and computing devices (e.g., servers) providing secure access to cloud-based services, including, for example, data storage, computing services, and customer service provision. In some examples, at least some of the data processing and computations related to anomaly detection, sign recognition, endoscope positioning, and imaging modality selection may be performed in the cloud. For example, images or video streams, or features extracted from them, may be streamed to the cloud for processing, and the computation results (e.g., personalized pathology-specific imaging modalities) may be relayed back to the local endoscope system.

[0078] In some examples, system 300 can operate in a closed-loop manner, where the feedback loop is used to continuously learn pathology-specific imaging modalities, such as updating target or recommended imaging modalities. Continuous learning can be achieved via explicit endoscopist feedback (e.g., satisfaction with recommendations, "like" buttons, etc.). Alternatively, system 300 can run in shadow mode to experienced endoscopists and monitor their exit actions, correcting segmented exit plans through reinforcing feedback.

[0079] In some examples, the robotic system 350 may be used to perform image-guided endoscopy or thus perform a portion of an endoscope withdrawal procedure. The robotic system 350 may automatically adjust its imaging modality based on a recommended pathology-specific imaging modality. The robotic system 350 may include a robotic arm detachably attached to the endoscope 310. The robotic arm may automatically or semi-automatically (e.g., using some user manual control or command) position and manipulate the endoscope 310 within the anatomical target via actuators, or position the device in a desired pose to facilitate manipulation of the anatomical target.

[0080] Figure 5This diagram illustrates, by way of example, the different stages of an image-guided colonoscopy and the automatic imaging modality switching based on the characteristics of abnormalities detected during an image-guided endoscopic procedure 500 (e.g., a colonoscopy as shown in this example). The procedure includes an endoscopic insertion phase followed by an exit phase. The image-guided endoscopic procedure can be performed using system 300. During the insertion phase, endoscopic images or video streams can be acquired, for example, using imaging system 312 for each of a plurality of colonic segments, including the rectosigmoid colon, sigmoid colon, descending colon, transverse colon, ascending colon, and cecum. Abnormalities can be automatically detected during the insertion phase using an abnormality detector circuit 322, or alternatively, such as... Figure 5 As shown, abnormalities are detected or confirmed during the exit phase. In the example, at the end of the insertion phase when the distal tip of the endoscope reaches the cecum, the imaging mode selection can be triggered manually or automatically by automatic cecum detection. During exit, the default imaging mode can be used, including modes such as... Figure 5The default illumination mode for white light imaging (WLI) is shown. When the endoscope withdraws to the first colonic segment, endoscopic image 510 is acquired in WLI mode, and a first abnormality is detected from endoscopic image 510. Based at least on image features extracted from image 510 (optionally, and auxiliary input 315), imaging mode selector circuitry 324 recommends a pathology-specific imaging modality, including a recommended illumination mode for narrow-band imaging (NBI), substantially in real-time, using, for example, a trained ML model 360. The NBI mode can enhance the discernibility of the first abnormality from the background of image 510 compared to the default WLI mode. The imaging mode can then be automatically switched from WLI to NBI, and the first abnormality is re-examined and diagnosed in NBI mode. Alternatively, the user is prompted to switch to NBI mode to re-examine the first abnormality. Image 512 of the first abnormality in NBI mode can be displayed to the user substantially in real-time. The imaging mode can then be switched back to the default WLI mode to examine other colonic segments. Subsequent examinations show a second abnormality detected in endoscopic image 520 of another segment. Based at least on image features extracted from image 520 (optionally, and auxiliary input 315), imaging mode selector circuit 324 recommends pathology-specific imaging modalities, including a recommended illumination mode for red dichroism (RDI), substantially in real time, using, for example, a trained ML model 360. The RDI mode enhances the discernibility of the second abnormality from the background of image 520 compared to the default WLI mode. The imaging mode then automatically switches from WLI to RDI, and the first abnormality is re-examined and diagnosed in RDI mode. Alternatively, the user is prompted to switch to RDI mode to re-examine the second abnormality. Image 522 of the second abnormality in RDI mode can be displayed to the user substantially in real time. After diagnosis, the imaging mode can be switched back to the default WLI mode to examine other colonic segments.

[0081] Figure 6 Example images of a portion of the colon are shown in different imaging modes. Image 610 was acquired in the "default" WLI mode. Lesion 615 was detected from endoscopic image 610. Based at least on image features extracted from image 610 (optionally, and auxiliary input 315), a recommendation to switch to narrow-band imaging (NBI) is generated. The imaging mode can be switched to the NBI model automatically or manually. Lesion 615 is then re-examined from image 620 acquired in NBI mode. Figure 6 As shown, compared to the default WLI mode, the NBI mode can enhance the discernibility of lesion 615 from the background of image 620, thereby enabling more accurate and effective real-time examination and optical diagnosis of lesion 615.

[0082] Figures 7A to 7BThis is a diagram illustrating an example of training an ML model and using the trained ML model to determine recommended pathology-specific imaging modalities for anomaly examination and diagnosis. Figure 7A The diagram illustrates the training (or learning) phase of the ML model, during which the ML model 720 can be trained to determine a target or recommended pathology-specific imaging modality based at least in part on endoscopic images of abnormalities and the surrounding environment. The training dataset may include multiple endoscopic images 710 of the same type of abnormality (e.g., polyps or lesions in the lining of a colonic segment) obtained from colonoscopy procedures performed on multiple patients. In some examples, the training data may also include images as described above regarding... Figure 3 The auxiliary input 315 is described. Examples of auxiliary inputs may include supporting imaging and illumination patterns 732 (examples or portions of endoscope and equipment information 450), endoscopist preferences 734 (examples of user profiles 440), clinical information 736 (which may include one or more of image or video streams and clinical data 410, patient information and medical history 420, or pre-procedure imaging study data 430 from previous endoscopic examinations), or AI findings 738 (examples of AI findings 460). Endoscope position information for each of the multiple endoscopic images 710 may also be included in the training dataset. The ML model 720 may have a neural network structure including an input layer, one or more hidden layers, and an output layer. The multiple endoscopic images 710 or features generated from them, along with one or more of the auxiliary inputs 732, 734, 736, or 738, may be fed into the input layer of the ML model 720, which propagates the input data or data features through one or more hidden layers to the output layer, which outputs a recommended pathology-specific imaging modality. ML Model 720 is capable of performing tasks by inferring based on patterns discovered in data analysis, without explicit programming. ML Model 720 explores the research and construction of algorithms (e.g., ML algorithms) that can learn from existing data and make predictions on new data. Such algorithms operate by building ML Model 720 based on training data to make data-driven predictions or decisions that are represented as outputs or evaluations.

[0083] ML models can be trained using either supervised or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that correlate inputs with outputs or results) to learn the relationship between inputs and outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between training inputs and outputs, so that the ML model can achieve the same relationship given an input to generate the corresponding output. Unsupervised learning trains the ML algorithm using information that is neither classified nor labeled, allowing the algorithm to operate on that information without guidance. Unsupervised learning is useful in exploratory analytics because it can automatically identify structures in the data.

[0084] Common tasks for supervised learning are classification and regression problems. Classification problems (also known as categorization problems) aim to classify items into one of several class values. Regression algorithms aim to quantify some items (e.g., by assigning scores to some input values). Some examples of commonly used supervised ML algorithms are logistic regression (LR), Naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM). Examples of DNNs include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN), or hybrid neural networks that include two or more neural network models of different types or configurations. Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0085] Another type of machine learning is federated learning (also known as collaborative learning), which trains algorithms on multiple distributed devices that store local data without exchanging data. This approach differs significantly from traditional centralized machine learning techniques, where all local datasets are uploaded to a single server, and also from more classic distributed methods, which typically assume that local data samples are uniformly distributed. Federated learning enables multiple participants to build shared, robust machine learning models without sharing data, thus enabling the resolution of critical issues such as data privacy, data security, data access permissions, and access to heterogeneous data.

[0086] Training of the ML model 720 can be performed continuously or periodically, or near real-time when additional procedural data becomes available. The training process involves algorithmically tuning one or more ML model parameters (e.g., weights or biases of any particular layer in a neural network model) until the trained ML model meets a specified training convergence criterion. The ML model 720 can be trained using weighted squared loss (for explicit feedback) or binary cross-entropy loss (for implicit feedback) by way of example rather than restriction. Other training techniques can be used, such as deep factorization machines, wide and deep learning, deep structured semantic models, or autoencoder-based recommendation systems. The trained ML model 720 can establish a correspondence between endoscopic images 710 (or features extracted from them) and recommended imaging modalities and the associated probabilities indicating the likelihood of the recommended imaging modality being selected.

[0087] Figure 7BThis illustrates, for example, an inference phase during colonoscopy withdrawal used to determine a target or recommended imaging modality for examining and diagnosing suspected abnormalities in the existing imaging modality. Real-time endoscopic images 750 of the suspected abnormality (e.g., polyps or lesions) and auxiliary inputs 760 (including, for example, information about the endoscopist's profile, patient clinical information, AI findings, device setup characteristics, and endoscopic position characteristics) can be applied to a trained ML modality 720 to determine an output 770, which includes the recommended imaging modality and a related probability indicating the likelihood that the target or recommended imaging modality was selected. Output 770 can be provided to the endoscopist or robotic system for manual or robot-assisted adjustment of the imaging modality to enhance abnormality examination and diagnosis.

[0088] In some examples, such as Figure 7A Training the illustrated ML model 720 may include predicting the probability associated with each of a plurality of supporting imaging modes, such as various illumination patterns (e.g., NBI, RDI, WLI, TXI, etc.). Such probabilities represent the likelihood that the corresponding imaging mode will be selected as the target or recommended imaging mode. The probabilities can be predicted based on the current system state (including the current imaging mode used to generate the analyzed endoscopic images) conditioned on various inputs including image features and various auxiliary input features. For example, for an endoscopist... Clinical information AI discovery Device setting function and endoscope position characteristics It can be derived from composite feature vectors. Common representation. This allows the features of the endoscopic image to be represented as... And the imaging or lighting mode is represented as Training the ML model 720 involves training based on the current video stream. Furthermore, it predicts the "affinity" for a given feature vector x, which is the endoscopist's choice of light source. What is the probability of this? Such "affinity" can be expressed using a conditional probability that takes values ​​between 0 and 1. This is indicated by a higher affinity value, which indicates the imaging or lighting mode. The higher probability of being selected as the recommended imaging mode. Conditional probability. It can learn from previous implicit or explicit user feedback, and be limited by learning. Recommendation system To predict ,in These are the learning parameters for the recommender system. In some examples, the recommender system... Can be used as Figure 5The multi-input collaborative learning process shown is used for learning. In, for example... Figure 7B In the inference phase shown, the trained ML model 720 can predict the conditional probability for each of multiple candidate imaging or illumination patterns. The target or recommended imaging mode is determined, at least in part, based on the probability corresponding to the candidate imaging modes. In the example, the target or recommended imaging mode can be selected based on the probability corresponding to the highest conditional probability. Associated candidate imaging modes.

[0089] Figure 8 This is a flowchart illustrating an example method 800 for determining or adjusting an endoscopic imaging modality during a patient's endoscopic procedure, such as a colonoscopy. A target or recommended pathology-specific imaging modality can be determined using at least endoscopic images or video streams of anatomical structures. Imaging modalities can help improve the real-time examination, identification, and diagnosis of various pathologies. Method 800 can be implemented in an endoscopic system 300. While the processes of method 800 are depicted in a flowchart, they do not need to be performed in a specific order. In various examples, some of the processes may be performed in a different order than that shown herein.

[0090] At 810, an imaging system associated with an endoscope can be used to acquire images or video streams of different segments of the target anatomical structure. Images or video streams can be acquired when the imaging system is set to one of several available imaging modes. An imaging mode or modality refers to one or more of the illumination modality, optical magnification, or the field of view of the imaging device. Examples of illumination modalities can include high-definition white light imaging (WLI), such as dye-based chromoendoscopy, or virtual CE such as narrow-band imaging (NBI), texture and color enhancement (TXI) imaging, red dichroism imaging (RDI), etc. Optical magnification defines the zoom setting (e.g., reduction or magnification of suspicious abnormalities in the target anatomical structure). The field of view (also called the field of view) of the imaging device describes the angular range of a given scene imaged by the imaging device.

[0091] At point 820, the acquired image or video stream can be analyzed to generate endoscopic image or video features for each of the different segments of the anatomical structure. Examples of image or video features include statistical features of pixel values, or morphological features such as corners, edges, spots, curvature, acceleration robust features (SURF), or scale-invariant feature transform (SIFT) features. In some examples, segment-specific image or video streams can be preprocessed, such as filtered, resized, oriented, or color or grayscale corrected, and endoscopic image features can be extracted from the preprocessed image or video stream. In some examples, endoscopic image or video features can be post-processed to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0092] At 830, for example, one can use, such as Figure 3 The imaging mode selector circuit 324 shown automatically determines a personalized target or recommended pathology-specific imaging modality, at least in part, based on endoscopic image or video features. In the example, the target or recommended pathology-specific imaging modality can be estimated using information about abnormalities detected from any specific segment of an anatomical structure. Abnormalities can be detected from endoscopic image or video features using template matching techniques or techniques based on artificial intelligence (AI) or machine learning (ML). Abnormality detection includes detecting and / or identifying one or more of the presence, type, size, shape, location, or number of pathological tissue or anatomical structures and other objects in the environment of the anatomical structure. In the example of colonoscopy, detected abnormalities may include segments of pathological tissue, such as mucosal abnormalities (polyps, inflammatory bowel disease, Merkel's diverticulum, lipomas, bleeding, vascularized mucosa, etc.) or obstructive mucosa (e.g., segments with poor bowel preparation, colonic dilatation, etc.). In some examples, when an abnormality is detected, the basic real-time position of the endoscope can also be used to determine the target or recommended pathology-specific imaging modality. Endoscopic positioning can be determined using electromagnetic tracking, or by detecting anatomical landmarks, such as those obtained from image or video streams acquired during the insertion phase of an endoscopic examination, or by extracting features from those image or video streams. In examples of landmark-based endoscopic positioning, AI or ML methods, such as trained ML models, can be used to identify landmarks. These trained ML models have been trained to establish a correspondence between endoscopic images or video streams, or features extracted from them, and landmark recognition, as described above regarding… Figure 3 Described.

[0093] In the example, auxiliary inputs may also be used to estimate the target or recommended pathology-specific imaging modality. These auxiliary inputs, by way of example and not limitation, include: image or video streams and clinical data from previous endoscopic procedures performed on the patient; patient information and medical history; or pre-procedure imaging study data; user (e.g., endoscopist) profiles (including user experience, work environment, affinity for new technologies, preferences for specific endoscopic procedures, etc.); endoscope and equipment information; or “AI discovery” including the state of an AI or ML algorithm used to detect abnormalities, landmarks, or other features or structural elements of interest in the target anatomy, as described above regarding… Figure 4 Described.

[0094] AI or ML methods can be used to estimate the target or recommended pathology-specific imaging modality. In the example, the results of anomaly detection, landmark detection, and real-time endoscopic position information, along with auxiliary inputs, can be applied to a trained ML model. The ML model can be trained to establish a correspondence between the composite inputs and the target or recommended values ​​for the target or recommended pathology-specific imaging modality, as described above. Figures 7A to 7B Described. In some examples, training the ML model may include predicting the probability associated with each of a plurality of supporting imaging modes, such as various illumination modes (e.g., NBI, RDI, WLI, TXI, etc.). Such probabilities represent the likelihood that the corresponding imaging mode is selected as the target or recommended imaging mode. The probabilities can be predicted based on the current system state conditioned on various inputs, including image features and various auxiliary input features, including the current imaging mode used to generate the endoscopic images being analyzed. Training the ML model involves based on the current video stream. Furthermore, given a composite input feature vector (which may include, for example, endoscopist information, patient clinical information, AI discovery, equipment setting features, and endoscope position features), the system predicts "affinity," that is, the probability that the endoscopist will choose a specific light source. This "affinity" can be represented by conditional probability, which can be obtained through methods such as... Figure 5 The multi-input collaborative learning process illustrated learns from previous implicit or explicit user feedback. During the inference phase, the trained ML model can predict conditional probabilities for each of multiple candidate imaging or illumination patterns. The target or recommended imaging pattern can be selected as the candidate imaging pattern associated with the highest conditional probability.

[0095] At point 830, the target or recommended pathology-specific imaging modality can be provided to the user or processing device to facilitate manual or robotic exit of the endoscope. At point 842, the target or recommended pathology-specific imaging modality can be displayed to the user to assist the endoscopist during endoscopic procedures, such as image-guided colonoscopy. If the target or recommended imaging modality differs from the current imaging modality used to examine and detect abnormalities, at point 844, an alarm can be generated to notify or alert the endoscopist of this effect. The notification or alarm can be delivered optically on the diagnostic monitor or audibly, such as via a warning alarm. A recommendation to switch to the target or recommended imaging modality can be provided to the user. Additionally or alternatively, in some examples, at point 846, a control signal can be generated to control the imaging system to automatically switch to the target or recommended imaging modality and recapture images or video streams of the target anatomy for further diagnosis.

[0096] Figure 9A block diagram of an example machine 900 is shown in general, to which any or more of the techniques (e.g., methods) discussed herein can be performed. Parts of this specification can be applied to the computational framework of various parts of the endoscope system 300.

[0097] In alternative implementations, machine 900 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 900 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 900 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 900 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by that machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0098] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership can be flexible with time and the variability of the underlying hardware. A circuit set includes members that can perform a specified operation individually or in combination during operation. In the example, the hardware of the circuit set may be invariably designed to perform a specific operation (e.g., hardwired). In the example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) to encode instructions for a specific operation, said variably connected physical components including computer-readable media that are physically modified (e.g., magnetically, electrically, with movable placement of invariant aggregate particles, etc.). When connecting physical components, the underlying electrical characteristics of the hardware composition are changed, for example, from an insulator to a conductor or from a conductor to an insulator. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of the circuit set in the hardware via variably connected portions to perform a specific operation during operation. Therefore, when the device is operating, the computer-readable medium is communicatively coupled to other components of a circuit set member. In the example, any component of the physical components can be used in more than one member of more than one circuit set. For example, during operation, an execution unit can be used at one point in time in a first circuit of a first circuit set, and can be reused at different times by a second circuit of the first circuit set or by a third circuit of the second circuit set.

[0099] Machine (e.g., computer system) 900 may include a hardware processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 904, and static memory 906, some or all of which may communicate with each other via interconnect (e.g., bus) 908. Machine 900 may also include a display unit 910 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) navigation device 914 (e.g., a mouse). In the example, display unit 910, input device 912, and UI navigation device 914 may be a touchscreen display. Machine 900 may additionally include a storage device (e.g., a drive unit) 916, a signal generation device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921 such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 900 may include output controller 928, such as serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connections, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0100] Storage device 916 may include machine-readable medium 922 on which one or more sets of data structures or instructions 924 (e.g., software) implementing or utilizing any one or more of the techniques or functions described herein are stored. During execution of instructions 924 by machine 900, instructions 924 may also reside wholly or at least partially within main memory 904, static memory 906, or hardware processor 902. In this example, one or any combination of hardware processor 902, main memory 904, static memory 906, or storage device 916 may constitute the machine-readable medium.

[0101] Although machine-readable medium 922 is shown as a single medium, the term “machine-readable medium” can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 924.

[0102] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for execution by machine 900 and causing machine 900 to perform any one or more of the technologies of this disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory, as well as optical and magnetic media. In examples, mass-capacity machine-readable media includes machine-readable media having a plurality of particles having invariant (e.g., rest) mass. Therefore, mass-capacity machine-readable media are not transient propagating signals. Specific examples of mass-capacity machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0103] Commands 924 can also be sent or received via a communication network 926 using a transmission medium via network interface device 920, utilizing any of several transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 family of standards known as WiFi®, the IEEE 802.16 family of standards known as WiMax®), the IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, etc. In the example, network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to communication network 926. In the example, network interface device 920 may include multiple antennas to perform wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 900, and includes digital or analog communication signals or other intangible media to facilitate communication of such software. Additional notes

[0104] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The drawings illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples (or one or more aspects thereof) using any combination or arrangement of those elements shown or described relative to a particular example (or one or more aspects thereof) or relative to other examples (or one or more aspects thereof).

[0105] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more, regardless of any other instance or usage of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive "or", such that "A or B" includes "A but not B", "B but not A", and "A and B". In this document, the terms "comprising" and "in" are used as concise English equivalents to the corresponding terms "including" and "wherein". Furthermore, in the appended claims, the terms "including" and "comprising" are open-ended, meaning that a system, apparatus, article, composition, formulation, or treatment that includes elements other than those listed after such terms in the claim is still considered to fall within the scope of that claim. Additionally, in the following claims, the terms "first", "second", and "third", etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0106] The above description is intended to be illustrative and not restrictive. For example, the examples above (or one or more aspects thereof) may be used in combination with each other. For example, other embodiments may be used by those skilled in the art after consulting the above description. An abstract is provided to enable the reader to quickly determine the nature of the technical disclosure. The abstract is submitted based on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the detailed embodiments above, various features may be combined together to simplify the disclosure. This should not be construed as implying that any unclaimed disclosed feature is necessary for any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the following claims are incorporated herein by way of example or embodiment, wherein each claim exists independently as a separate embodiment, and it is contemplated that such embodiments may be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of the equivalents conferred by such claims.

Claims

1. An endoscope system, comprising: An endoscope, the endoscope including an imaging system configured to acquire images or video streams of target anatomical structures in a patient during an endoscopic examination; as well as The controller circuit is configured to: The acquired images or video streams are analyzed to generate endoscopic image or video features characterizing abnormalities in the target anatomical structure; Based at least in part on the features of the endoscopic images or videos and one or more auxiliary features different from the features of the endoscopic images or videos, the target or recommended imaging mode of the imaging system is automatically determined to enhance the discernibility of the abnormality from the image or video stream of the target anatomical structure in subsequent imaging of the target anatomical structure. as well as The target or recommended imaging mode is provided to the user or robotic system to facilitate manual or automatic adjustment of the endoscopic imaging mode during the endoscopic examination procedure.

2. The endoscope system according to claim 1, wherein, The target or recommended imaging mode includes one or more of the illumination mode, zoom setting, or perspective relative to the anomalous feature.

3. The endoscope system according to any one of claims 1 to 2, wherein, The target or recommended imaging mode includes one of the following: narrowband imaging (NBI) mode, red dichromatic imaging (RDI) mode, white light imaging (WLI) mode, or texture and image enhancement (TXI) mode.

4. The endoscope system according to any one of claims 1 to 3, wherein, The endoscope in question is a colonoscope. The imaging system is configured to acquire images or video streams of each of the different segments of the colon during a colonoscopy. The controller circuitry is configured to determine a corresponding target or recommended imaging mode to enhance the discernibility of anomalies from the image or video stream during subsequent imaging of the different colonic segments.

5. The endoscopic system according to any one of claims 1 to 4, wherein, The controller circuit is configured to: The abnormalities in the target anatomical structure are detected using the features of the endoscopic images or videos. as well as The target or recommended imaging mode is determined at least in part based on the results of the anomaly detection.

6. The endoscopic system according to claim 5, wherein, Detecting the abnormality includes detecting the presence or absence of the abnormality and one or more of the type, size, shape, location, or number of pathological tissues or obstructing mucosa in the target anatomical structure.

7. The endoscope system according to any one of claims 5 to 6, wherein, The controller circuitry is configured to use a first trained machine learning (ML) model to detect the anomaly.

8. The endoscope system according to any one of claims 1 to 7, wherein, The controller circuit is configured to: The position of the endoscope in the target anatomical structure is determined substantially in real time, at least in part, based on the features of the endoscopic images or videos. The position of the endoscope is aligned with a pre-generated template of the target anatomical structure; as well as The location of the endoscope in the target anatomical structure is displayed on the user interface.

9. The endoscope system according to claim 8, wherein, The controller circuit is configured to use the endoscopic image or video features to identify anatomical landmarks and determine the position of the endoscope based on the identified anatomical landmarks.

10. The endoscopic system according to any one of claims 8 to 9, wherein, The controller circuitry is configured to also use the determined position of the endoscope within the target anatomical structure to determine the target or recommended imaging mode.

11. The endoscopic system according to any one of claims 1 to 10, wherein, To determine the target or recommended imaging mode, the controller circuitry is configured to apply the endoscopic image or video features to a second trained machine learning (ML) model, which is trained to establish a correspondence between the endoscopic image or video features and one of a set of candidate imaging modes.

12. The endoscopic system according to claim 11, wherein, The second trained ML model is also trained using one or more of the auxiliary features, which include one or more of the following: A brief profile of the endoscopist who performed the endoscopic procedure; Patient information and the patient's medical history data; Endoscopic system settings information; or Pre-process imaging study data, In order to determine the target or recommended imaging mode, the controller circuit is configured to apply auxiliary inputs and enhanced inputs, including features of the endoscopic image or video, to the second trained ML model.

13. The endoscopic system according to any one of claims 11 to 12, wherein, The second trained ML model is trained to predict each of a plurality of candidate imaging modes: representing the probability that the corresponding imaging mode is selected as the target or recommended imaging mode. The controller circuit is configured to determine the target or recommended imaging mode based at least in part on the probability corresponding to the candidate imaging mode.

14. The endoscopic system according to any one of claims 1 to 13, wherein, In response to the target or recommended imaging mode being different from the existing imaging mode used to obtain the image or video stream of the target anatomical structure, the controller circuitry is configured to provide the user with a recommendation to switch to the target or recommended imaging mode to recapture the image or video stream of the target anatomical structure.

15. The endoscopic system according to any one of claims 1 to 14, wherein, In response to the target or recommended imaging mode being different from the existing imaging mode used to obtain the image or video stream of the target anatomical structure, the controller circuitry is configured to generate a control signal to cause the imaging system to automatically switch to the target or recommended imaging mode and recapture the image or video stream of the target anatomical structure.

16. A method for determining or adjusting an endoscopic imaging mode during an endoscopic examination, the method comprising: Images or video streams of the target anatomical structure are acquired during the endoscopic examination using an imaging system associated with the endoscope. The acquired images or video streams are analyzed to generate endoscopic image or video features characterizing abnormalities in the target anatomical structure; Based at least in part on the features of the endoscopic images or videos and one or more auxiliary features different from the features of the endoscopic images or videos, the target or recommended imaging mode of the imaging system is automatically determined to enhance the discernibility of the abnormality from the image or video stream of the target anatomical structure in subsequent imaging of the target anatomical structure. as well as The target or recommended imaging mode is provided to the user or robotic system to facilitate manual or automatic adjustment of the endoscopic imaging mode during the endoscopic examination procedure.

17. The method according to claim 16, wherein, The target or recommended imaging mode includes one or more of the illumination mode, zoom setting, or perspective relative to the anomalous feature.

18. The method according to any one of claims 16 to 17, wherein, The target or recommended imaging mode includes one of the following: narrowband imaging (NBI) mode, red dichromatic imaging (RDI) mode, white light imaging (WLI) mode, or texture and image enhancement (TXI) mode.

19. The method according to any one of claims 16 to 18, further comprising: The abnormalities in the target anatomical structure are detected using the features of the endoscopic images or videos. as well as The target or recommended imaging mode is determined at least in part based on the results of the anomaly detection.

20. The method according to any one of claims 16 to 19, further comprising: Use the features of the endoscopic images or videos to identify anatomical landmarks; The position of the endoscope in the target anatomical structure is determined substantially in real time, at least in part, based on the identified anatomical landmarks; The position of the endoscope is aligned with a pre-generated template of the target anatomical structure; as well as The user interface displays the location of the endoscope within the target anatomical structure. The determination of the target or recommended imaging mode is also based on the determined position of the endoscope within the target anatomical structure.

21. The method according to any one of claims 16 to 20, wherein, Determining the target or recommended imaging mode includes applying the features of the endoscopic image or video to a second trained machine learning (ML) model, which is trained to establish a correspondence between the features of the endoscopic image or video and a candidate imaging mode from a set of candidate imaging modes.

22. The method according to claim 21, wherein, The second trained ML model is also trained using one or more of the auxiliary features, which include one or more of the following: A brief profile of the endoscopist who performed the endoscopic procedure; Patient information and the patient's medical history data; Endoscopic system settings information; or Pre-process imaging study data, Determining the target or recommended imaging mode includes applying auxiliary input and enhanced input, including features of the endoscopic image or video, to the second trained ML model.

23. The method according to claim 22, wherein, The second trained ML model is trained to predict each of a plurality of candidate imaging modes: representing the probability that the corresponding imaging mode is selected as the target or recommended imaging mode. Determining the target or recommended imaging mode includes identifying a candidate imaging mode among the plurality of candidate imaging modes whose corresponding probability is higher than any other candidate imaging mode among the candidate imaging modes.

24. The method according to any one of claims 16 to 23, wherein, In response to the target or recommended imaging mode being different from the existing imaging mode used to obtain the image or video stream of the target anatomical structure, the user is provided with a recommendation to switch to the target or recommended imaging mode to recapture the image or video stream of the target anatomical structure.

25. The method according to any one of claims 16 to 24, wherein, In response to the target or recommended imaging mode being different from the existing imaging mode used to acquire the image or video stream of the target anatomical structure, a control signal is generated to cause the imaging system to automatically switch to the target or recommended imaging mode and recapture the image or video stream of the target anatomical structure.

Images