Image-guided endoscope withdrawal control

Abstract

Systems, devices, and methods are disclosed for creating and using personalized segment-specific endoscope withdrawal plans to guide an endoscopic procedure. An endoscope system includes an endoscope and a controller circuit. The endoscope includes an imaging system to obtain an image or video stream of different segments of an anatomical structure within a patient during an endoscopic procedure. The controller circuit may analyze the image or video stream to generate image or video features for each of the different segments of the anatomical structure. Based on the image or video features, the controller circuit may generate an endoscope withdrawal plan that includes a target or suggested segment-specific endoscope withdrawal parameter value for each of the segments. The endoscope withdrawal plan may be provided to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

Description

[0001] Priority Statement

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 582,026, filed on September 12, 2023, the contents of which are incorporated herein by reference. Technical Field

[0003] This document relates generally to endoscopic medical systems, and more specifically to systems and methods for image-guided withdrawal of an endoscope during an endoscopic examination. Background Technology

[0004] Endoscopy has been used in a variety of clinical procedures, including, for example, illumination, imaging, detection, and diagnosis of one or more disease states; providing fluid delivery toward anatomical regions (e.g., saline or other preparations via a fluid channel); providing access to one or more therapeutic devices or biological material collection devices for sampling or treating anatomical regions (e.g., via a working channel); and providing aspiration channels for collecting fluids (e.g., saline or other preparations). Examples of such anatomical regions may include the gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary ducts, intestines, colon, etc.), the renal region (e.g., kidneys, ureters, bladder, urethra), and other internal organs (e.g., the reproductive system, sinuses, submucosal regions, respiratory tract), etc.

[0005] Some endoscopes include a working channel through which the operator can perform aspiration, place diagnostic or therapeutic devices (e.g., brushes, biopsy needles or forceps, stents, baskets, or balloons), or perform minimally invasive procedures such as tissue sampling or removal of unwanted tissue (e.g., benign or malignant strictures) or foreign bodies (e.g., stones). Some endoscopes can be used with laser or plasma systems to deliver energy to anatomical targets (e.g., soft or hard tissue or stones) to achieve the desired treatment. For example, lasers have been used in applications of tissue ablation, coagulation, vaporization, fragmentation, and lithotripsy to break down stones in the kidneys, gallbladder, ureters, and other stone-forming areas, or to ablate large stones into smaller fragments.

[0006] An endoscopic procedure includes an insertion phase, in which the endoscope passes through a natural opening and into the body cavity until the target site is reached; and a subsequent withdrawal phase, in which the endoscope is carefully withdrawn from the body. Diagnostic or therapeutic procedures (e.g., biopsy or removal of certain abnormal or pathological tissues or foreign bodies) can occur at the target anatomical structure or during withdrawal. For example, a colonoscopy involves inserting a colonoscope through the anus all the way to the beginning of the colon (called the cecum). During withdrawal, segments of the colon can be examined, and polyps of certain sizes detected during insertion can be removed (polypectomy). Colonoscopy, along with the diagnosis and treatment of polyps, has been used to reduce the incidence and mortality of colorectal cancer. Summary of the Invention

[0007] Colonoscopy withdrawal time during a colonoscopy is an important quality metric, especially for negative procedures. To ensure adequate examination and treatment time, minimum standards of withdrawal times exceeding 6 minutes and ideal standards of withdrawal times exceeding 10 minutes have been reported as quality assurance criteria for colonoscopy. Longer withdrawal times may increase the adenoma detection rate (ADR).

[0008] To ensure sufficient examination time and prevent the colonoscope from being withdrawn too quickly, speedometers and other methods have been proposed to display withdrawal speed. However, withdrawal speed limits or withdrawal time thresholds used to assess withdrawal during endoscopy are often predetermined, for example, based on generally accepted criteria (e.g., at least six minutes or at least ten minutes). These “generic” withdrawal criteria are highly generalized and are determined without taking into account the unique circumstances presented by each procedure. For example, a target withdrawal time of more than 6 minutes is a quality indicator for outcomeless colonoscopies primarily used in patients of average risk with an intact colon. This includes patients without prior surgical resection and without biopsy or polyp removal. Such generic withdrawal criteria may be less effective or suboptimal in cases involving the colon of patients with colon cancer or those under surveillance (e.g., due to polyps previously identified and / or removed during past colonoscopies for that specific patient). Additionally, if the bowel is poorly prepared in one or more colonic segments, the universal withdrawal time may be ineffective. In such cases, the endoscopist may need to withdraw diligently and carefully, cleaning the bowel slowly along the way and ensuring a complete mucosal examination to avoid any hidden polyps.

[0009] Aside from patient-to-patient variability in withdrawal time requirements due to factors such as patient medical conditions and / or bowel preparation quality as described above, universal withdrawal time standards (e.g., a minimum of 6 or 10 minutes) focus on a global withdrawal time applicable to the entire colonoscopy withdrawal phase. It lacks specificity for withdrawal time at any particular colonic segment. Because different colonic segments often have varying bowel preparation quality and / or different chances of presenting abnormal structures (e.g., polyps or cancer), the expected withdrawal time can vary segment by segment. Universal global withdrawal times rarely provide guidance on how quickly or slowly the colonoscope should be withdrawn at different segments of the colon.

[0010] The inventors of this disclosure have identified an unmet need to improve quality control of endoscopic withdrawal during colonoscopy. This document describes systems and methods for image-based segment-specific endoscopic withdrawal during endoscopic procedures. Exemplary systems can receive endoscopic images or video streams acquired during insertion or at any time before the endoscope is withdrawn beyond a target segment, analyze the images or video streams, and generate a personalized segment-specific endoscopic withdrawal plan. The segment-specific endoscopic withdrawal plan may include target endoscopic withdrawal rate limits or withdrawal time limits for each of several different segments of an anatomical structure. The personalized segment-specific endoscopic withdrawal plan may be presented to the user in graphical form, such as a withdrawal rate graph or a withdrawal time graph. During withdrawal, the personalized segment-specific endoscopic withdrawal plan can be used as guidance for mucosal examination and necessary treatments (e.g., polyp removal). The actual withdrawal rate can be tracked and measured. Endoscopy physicians can be provided with substantially real-time feedback on the withdrawal rate or time spent on segments of anatomy (e.g., colon segments), along with recommendations for adjusting the endoscopic withdrawal rate. Real-time monitoring of endoscopic withdrawal with reference to personalized segment-specific withdrawal plans can improve the quality of mucosal examinations and the accuracy and efficiency of screenings. In some examples, personalized segmental endoscopic withdrawal plans can be provided to robotic colonoscopy systems to facilitate robot-assisted endoscopy and ensure that withdrawal rates are within recommended limits.

[0011] Example 1 is an endoscope system comprising: an endoscope including an imaging system configured to acquire images or video streams of different segments of a patient's anatomical structure during an endoscopic examination procedure including insertion and subsequent withdrawal of the endoscope into the anatomical structure; and controller circuitry configured to: analyze the acquired image or video streams to generate endoscopic image or video features for each segment of the different segments of the anatomical structure; generate an endoscopic withdrawal plan based at least in part on the endoscopic image or video features, the endoscopic withdrawal plan including target or suggested segment-specific endoscopic withdrawal parameter values ​​for each segment of the different segments of the anatomical structure; and provide the endoscopic withdrawal plan to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0012] In Example 2, the subject matter of Example 1 may optionally include: target or suggested segment-specific endoscopic withdrawal parameter values, which may include target segment-specific endoscopic withdrawal speed limits (WSLs) for individual segments within different segments of an anatomical structure.

[0013] In Example 3, the subject matter of any one or more of Examples 1 to 2 may optionally include: target or suggested segment-specific endoscopic withdrawal parameter values, which may include target segment-specific endoscopic withdrawal time limits (WTLs) for individual segments in different segments of an anatomical structure.

[0014] In Example 4, the subject matter of any one or more of Examples 1 to 3 may optionally include: a colonoscope for use during a colonoscopy, wherein the imaging system is configured to acquire images or video streams from each of different colonic segments during the colonoscopy procedure.

[0015] In Example 5, the subject matter of any one or more of Examples 1 to 4 may optionally include: wherein determining the target or recommended segment-specific endoscopic withdrawal parameter value includes: for the first segment of the anatomical structure, determining the first target segment-specific endoscopic withdrawal parameter using at least endoscopic image or video features generated based on an image or video stream obtained during manual or robotic withdrawal of the endoscope before the endoscope is withdrawn beyond the first segment.

[0016] In Example 6, the subject matter of Example 5 may optionally include: an image or video stream obtained before the endoscope reaches the first segment, which may include an image or video stream obtained during the insertion of the endoscope into the anatomical structure.

[0017] In Example 7, the subject matter of any one or more of Examples 1 to 6 may optionally include: controller circuitry that can be configured to: perform anomaly detection, the anomaly detection including detecting one or more anomalies in various segments based at least in part on endoscopic image or video features; and determine, at least in part on the results of the anomaly detection, target or recommended segment-specific endoscopic withdrawal parameter values ​​for each segment in the various segments.

[0018] In Example 8, the subject of Example 7 may optionally include: anomaly detection, which may include identifying one or more of the following: the presence or absence, type, size, shape, location, or quantity of pathological tissue or obstructive mucosa.

[0019] In Example 9, the subject of any one or more of Examples 7 to 8 may optionally include: a controller circuit that can be configured to detect one or more anomalies using a first trained machine learning (ML) model trained to establish a correspondence between (i) endoscopic images or video streams or features extracted therefrom, and (ii) one or more anomalous features.

[0020] In Example 10, the subject matter of any one or more of Examples 7 to 9 may optionally include: a controller circuit that can be configured to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for each segment in different segments by applying the results of anomaly detection to a second trained machine learning (ML) model trained to establish a correspondence between (i) one or more anomalous features and (ii) target or recommended segment-specific endoscopic withdrawal parameter values.

[0021] In Example 11, the subject of Example 10 may optionally include: a controller circuit that can be configured to: determine anomaly scores based on the type, size, shape, location, or number of anomalies; and determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for each segment in different segments by applying the determined anomaly scores to a second trained ML model.

[0022] In Example 12, the subject matter of any one or more of Examples 1 to 11 may optionally include: controller circuitry that can be configured to also use one or more of the following to generate an endoscopic withdrawal plan: image or video streams and clinical data from previous endoscopic examination procedures; patient information and medical history; or preoperative imaging study data.

[0023] In Example 13, the subject matter of any one or more of Examples 1 to 12 may optionally include: controller circuitry that can be configured to display a graphical representation of an endoscope withdrawal plan on a user interface, the graphical representation including depictions of different segments of anatomy, each segment being color-coded or grayscale-coded to indicate the corresponding target or suggested segment-specific endoscope withdrawal parameter value.

[0024] In Example 14, the subject matter of Example 13 may optionally include: controller circuitry that can be configured to: determine, at least in part, the position of an endoscope in a different segment of a different anatomical structure based substantially in real time on endoscopic image or video features; register the position of the endoscope to a pre-generated template of the anatomical structure; and display the position of the endoscope in a different segment of a different anatomical structure on a user interface as an overlay on a graphical representation of the endoscope withdrawal plan.

[0025] In Example 15, the subject matter of any one or more of Examples 2 to 14 may optionally include: controller circuitry configured to: measure the endoscopic withdrawal speed in one of the different segments of the anatomical structure; and generate an alert to the user and provide suggestions to adjust the withdrawal speed to substantially conform to the target segment-specific endoscopic WSL when the measured endoscopic withdrawal speed deviates from the target segment-specific endoscopic WSL for said one of the different segments by a certain margin.

[0026] In Example 16, the subject matter of any one or more of Examples 2 to 15 may optionally include: a robotic system that can be configured to robotically withdraw an endoscope, wherein the controller circuitry is configured to: measure the endoscope withdrawal speed in one of the different segments of the anatomical structure; and when the measured endoscope withdrawal speed deviates from the target segment-specific endoscope WSL for said one of the different segments by a certain margin, generate a control signal to the robotic system to automatically adjust the withdrawal speed to substantially conform to the target segment-specific endoscope WSL.

[0027] In Example 17, the subject matter of any one or more of Examples 2 to 16 may optionally include: controller circuitry configured to: identify aberrant segments from different segments based at least in part on endoscopic image or video features; and display a visual indicator of the aberrant segment on a user interface and generate an alert to the user during manual or robotic withdrawal of the endoscope and before the endoscope reaches the identified aberrant segment, thereby withdrawing the endoscope at a rate lower than that of the target segment-specific endoscope WSL for the identified aberrant segment.

[0028] Example 18 is a method for planning the withdrawal of an endoscope from a patient's anatomical structure during an endoscopic examination, the method comprising the steps of: acquiring images or video streams of different segments of the anatomical structure during the insertion phase of the endoscopic examination using an imaging system associated with the endoscope; analyzing the acquired images or video streams to generate endoscopic image or video features for each segment of the different segments of the anatomical structure; generating an endoscopic withdrawal plan based at least in part on the endoscopic image or video features, the endoscopic withdrawal plan including target or suggested segment-specific endoscopic withdrawal parameter values ​​for each segment of the different segments of the anatomical structure; and providing the endoscopic withdrawal plan to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0029] In Example 19, the subject matter of Example 18 may optionally include: target or suggested segment-specific endoscopic withdrawal parameter values, which may include target segment-specific endoscopic withdrawal speed limits (WSL) or target segment-specific endoscopic withdrawal time limits (WTL) for individual segments in different segments of an anatomical structure.

[0030] In Example 20, the subject matter of any one or more of Examples 18 to 19 may optionally include: determining first target segment-specific endoscopic withdrawal parameters for the first segment of an anatomical structure using at least endoscopic image or video features generated from an image or video stream obtained during manual or robotic withdrawal of the endoscope before the endoscope is withdrawn beyond the first segment.

[0031] In Example 21, the subject matter of Example 20 may optionally include: an image or video stream obtained before the endoscope reaches the first segment, which may include an image or video stream obtained during the insertion of the endoscope into the anatomical structure.

[0032] In Example 22, the subject matter of any one or more of Examples 18 to 21 may optionally include: performing anomaly detection, which includes detecting one or more anomalies in various segments of different segments based at least in part on features of endoscopic images or videos; and determining, at least in part on the results of the anomaly detection, target or recommended segment-specific endoscopic withdrawal parameter values ​​for various segments of different segments.

[0033] In Example 23, the subject of Example 22 may optionally include: anomaly detection using a first trained machine learning (ML) model trained to establish a correspondence between (i) features of an endoscopic image or video stream or extracted therefrom, and (ii) one or more anomalous features.

[0034] In Example 24, the subject matter of any one or more of Examples 22 to 23 may optionally include: determining target or recommended segment-specific endoscopic withdrawal parameter values ​​for each segment in different segments, wherein determining target or recommended segment-specific endoscopic withdrawal parameter values ​​for each segment in different segments may include applying the results of anomaly detection to a second trained machine learning (ML) model, which is trained to establish a correspondence between (i) one or more anomalous features and (ii) target or recommended segment-specific endoscopic withdrawal parameter values.

[0035] In Example 25, the subject matter of any one or more of Examples 18 to 24 may optionally include: generating an endoscopic withdrawal plan, which may further include using one or more of the following: image or video streams and clinical data from previous endoscopic procedures; patient information and medical history; or preoperative imaging study data.

[0036] In Example 26, the subject matter of any one or more of Examples 18 to 25 may optionally include: displaying a graphical representation of an endoscopic withdrawal plan on a user interface, the graphical representation including depictions of different segments of anatomy, each segment being color-coded or grayscale-coded to indicate the corresponding target or suggested segment-specific endoscopic withdrawal parameter values.

[0037] In Example 27, the subject matter of Example 26 may optionally include: determining the position of the endoscope substantially in real time; registering the position of the endoscope to a pre-generated template of the anatomical structure; and displaying the position of the endoscope in a different segment of a different segment on a user interface as an overlay on a graphical representation of the endoscope withdrawal plan.

[0038] In Example 28, the subject matter of any one or more of Examples 19 to 27 may optionally include: measuring the endoscopic withdrawal speed in one of the different segments of an anatomical structure; and generating an alert to the user and providing suggestions to adjust the endoscopic withdrawal speed to substantially conform to the target segment-specific endoscopic WSL when the measured endoscopic withdrawal speed deviates from the target segment-specific endoscopic WSL for said one of the different segments by a certain margin.

[0039] In Example 29, the subject matter of any one or more of Examples 19 to 28 may optionally include: measuring the endoscopic withdrawal speed in one of the different segments of an anatomical structure; and generating a control signal to a robotic system to automatically adjust the withdrawal speed to substantially conform to the target segment-specific endoscopic WSL when the measured endoscopic withdrawal speed deviates from the target segment-specific endoscopic WSL for said one of the different segments by a certain margin.

[0040] In Example 30, the subject matter of any one or more of Examples 19 to 29 may optionally include: identifying aberrant segments from different segments based at least in part on endoscopic image or video features; and displaying a visual indicator of the aberrant segment on a user interface and generating a reminder to the user during manual or robotic withdrawal of the endoscope and before the endoscope reaches the identified aberrant segment, thereby withdrawing the endoscope at a rate lower than that of the target segment-specific endoscope WSL for the identified aberrant segment.

[0041] The proposed techniques are described in relation to the controlled withdrawal of the endoscope during colonoscopy, but are not limited thereto. Systems, apparatus, and techniques described according to various embodiments in this document may be used additionally or alternatively in other procedures involving different types of endoscopes, including, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, colposcopy, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy, etc.

[0042] This disclosure is an overview of some of the teachings of this application and is not intended to be exclusive or exhaustive. Further details regarding the subject matter can be found in the detailed description and the appended claims. Other aspects of this disclosure will become apparent to those skilled in the art upon reading and understanding the following detailed description and viewing the accompanying drawings, which form a part thereof, and each of the drawings should not be considered limiting. The scope of this disclosure is defined by the appended claims and their legal equivalents. Attached Figure Description

[0043] Figures 1 to 2 This is a schematic diagram illustrating an example of an endoscopic system used in endoscopic procedures such as colonoscopy.

[0044] Figure 3 An example of an endoscope system configured to create personalized segment-specific endoscope withdrawal plans and use said plans to guide endoscope withdrawal during an endoscopic examination is shown.

[0045] Figure 4 Examples of auxiliary inputs that can be used in the estimation of withdrawal parameters and the creation of personalized segment-specific endoscopic withdrawal plans are shown.

[0046] Figure 5 Different stages of image-guided colonoscopy according to the implementation methods discussed herein are illustrated by way of example.

[0047] Figures 6A to 6BExamples are shown of training a machine learning (ML) model and using the trained ML model to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for any specific segment of an anatomical structure.

[0048] Figures 7A to 7B This is a graph illustrating an example of guided withdrawal and speed tracking during a colonoscopy using a personalized segment-specific endoscopic withdrawal plan.

[0049] Figure 8 This is a flowchart illustrating an example method for creating a personalized segment-specific endoscope withdrawal plan and using the plan to guide endoscope withdrawal during an endoscopic examination.

[0050] Figure 9 This is a block diagram illustrating an example machine on which any or more of the techniques (e.g., methods) discussed herein can be performed. Detailed Implementation

[0051] This document describes systems, apparatus, and methods for creating and using personalized segment-specific endoscopic withdrawal plans to guide withdrawal during endoscopic procedures. According to one embodiment, the endoscopic system includes an endoscope and controller circuitry. The endoscope includes an imaging system to acquire images or video streams of different segments of an anatomical structure within a patient during the endoscopic procedure. The controller circuitry can analyze the acquired image or video streams and generate image or video features for individual segments within the different segments of the anatomical structure. Based on the image or video features, the controller circuitry can generate a personalized segment-specific endoscopic withdrawal plan that includes target or suggested endoscopic withdrawal parameter values ​​(e.g., withdrawal speed or withdrawal time) for each segment within the different segments. Personalized endoscopic withdrawal plans can be provided to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0052] Figure 1 This is a schematic diagram of an endoscope system 10 for use in endoscopic procedures such as colonoscopy. System 10 may include an imaging and control system 12 and an endoscope 14. System 10 is an illustrative example of an endoscope system suitable for use with the systems, apparatus, and methods described herein, such as a colonoscopy system for use in image-guided colonoscopy according to a personalized segmental endoscopic withdrawal plan as described in this document.

[0053] Endoscope 14 may be inserted into an anatomical region for imaging, or to provide access to one or more sampling devices for biopsy or therapeutic devices for treating disease conditions associated with the anatomical region, or to be attached (e.g., via tethering) to said one or more sampling devices or said therapeutic devices. Endoscope 14 may interface with and be connected to imaging and control system 12. Endoscope 14 may also include a colonoscope, but other types of endoscopes may be used in conjunction with the features and teachings of this disclosure. Imaging and control system 12 may include control unit 16, output unit 18, input unit 20, light source unit 22, fluid source 24, and suction pump 26.

[0054] The imaging and control system 12 may include various ports for coupling with the endoscope system 10. For example, the control unit 16 may include data input / output ports for receiving data from and transmitting data to the endoscope 14. The light source unit 22 may include output ports for transmitting light to the endoscope 14, such as via an optical fiber link. The fluid source 24 may include ports for transmitting fluid to the endoscope 14. The fluid source 24 may include, for example, a pump and a fluid tank, or may be connected to an external tank, container, or storage unit. The suction pump 26 may include ports for creating a vacuum from the endoscope 14 to generate suction, for example, for extracting fluid from an anatomical region in which the endoscope 14 is inserted. The operator of the endoscope system 10 can use the output unit 18 and the input unit 20 to control the functions of the endoscope system 10 and view the output of the endoscope 14. The control unit 16 may also generate signals or other outputs depending on the anatomical region in which the endoscope 14 is inserted. In some examples, the control unit 16 may generate electrical output, acoustic output, fluid output, etc., for use in treating anatomical areas by means of methods such as cauterization, cutting, freezing, etc.

[0055] Fluid source 24 may be in communication with control unit 16 and may include one or more air sources, saline sources, or other fluid sources, as well as associated fluid channels (e.g., air channels, flushing channels, suction channels, etc.) and connectors (barbed fittings, fluid seals, valves, etc.). Fluid source 24 may be used as activation energy for biasing or pressure application devices used in this disclosure. Imaging and control system 12 may also include drive unit 46, which may include a motorized actuator for advancing the distal segment of endoscope 14.

[0056] Endoscope 14 may include an insertion section 28, a functional section 30, and a handle section 32, which may be coupled to a cable section 34 and a coupler section 36. The insertion section 28 may extend distally from the handle section 32, and the cable section 34 may extend proximally from the handle section 32. The insertion section 28 may be elongated and may include a curved section and a distal end to which the functional section 30 may be attached. The curved section may be controllable (e.g., via a control knob 38 on the handle section 32) to manipulate the distal end through tortuous anatomical passages (e.g., the stomach, duodenum, kidney, ureter, etc.). The insertion section 28 may also include one or more working channels (e.g., lumens), which may be elongated and may support the insertion of one or more therapeutic instruments of the functional section 30. The working channels may extend between the handle section 32 and the functional section 30. Additional functions such as fluid pathways, guide wires, and pull wires can also be provided by the insertion section 28 (e.g., via suction or flushing channels).

[0057] The coupler section 36 can be connected to the control unit 16 to connect the endoscope 14 to various features of the control unit 16, such as the input unit 20, the light source unit 22, the fluid source 24, and the suction pump 26.

[0058] The handle section 32 may include a knob 38 and a port 40A. The knob 38 may be connected to a pull cable or other actuation mechanism that may extend through the insertion section 28. Port 40A and ports such as 40B (…) Figure 2 Other ports can be configured to couple various cables, guide wires, auxiliary mirrors, tissue collection devices, fluid tubes, etc. to the handle section 32, for example, for coupling with the insertion section 28.

[0059] According to the example, the imaging and control system 12 can be mounted on a mobile platform (e.g., a trolley 41) having a design for housing the light source unit 22, the suction pump 26, and the image processing unit 42. Figure 2 Shelves such as those for imaging and control systems 12. Alternatively, several components of the imaging and control system 12 (such as...) Figure 1 and Figure 2 (As shown) can be directly mounted on endoscope 14 to make the endoscope "self-contained".

[0060] Functional segment 30 may include components for treating and diagnosing the anatomy of a patient. Functional segment 30 may include an imaging device, an illumination device, and a lifter. Functional segment 30 may also include optically enhanced biomaterial and tissue collection and retrieval devices. For example, functional segment 30 may include one or more electrodes electrically connected to handle segment 32 and functionally connected to imaging and control system 12 to analyze biomaterial in contact with the electrodes based on contrast biodata stored in imaging and control system 12. In other examples, functional segment 30 may be directly integrated with a tissue collector.

[0061] In some examples, endoscope 14 can be robotically controlled, for example, via a robotic arm attached thereto. The robotic arm can automatically or semi-automatically (e.g., using some user manual control or command) locate and navigate endoscope 14 (e.g., functional segment 30 and / or insertion segment 28) within the target anatomical structure via actuators, or position the device in a desired orientation to facilitate manipulation of the anatomical target. According to the various examples discussed in this document, a controller can generate control signals to the actuators of the robotic arm to facilitate the manipulation of such an instrument or tool according to a personalized segment-specific endoscope withdrawal plan during robot-assisted endoscopy.

[0062] Figure 2 yes Figure 1 A schematic diagram of an endoscope system 10, which includes an imaging and control system 12 and an endoscope 14. Figure 2 Components of an imaging and control system 12 coupled to an endoscope 14, which in the illustrated example includes a colonoscope, are schematically shown. The imaging and control system 12 may include a control unit 16, which may include or be coupled to an image processing unit 42, a treatment generator 44, a drive unit 46, a light source unit 22, an input unit 20, and an output unit 18. The control unit 16 may include, or communicate with, an endoscope, surgical instruments 48, and an endoscopic system, which may include means configured to engage tissue and collect and store a portion of that tissue, and an imaging device (e.g., a camera) may be used to view the target tissue via means including optically enhanced materials and components. The control unit 16 may be configured to activate the camera to view the target tissue distal to the endoscopic system. Similarly, the control unit 16 can be configured to activate the light source unit 22 to illuminate a surgical instrument 48, which may include selection components configured to reflect light in a particular manner, such as an enhanced tissue cutter with reflective particles.

[0063] Coupler section 36 can be connected to control unit 16 to connect endoscope 14 to various features of control unit 16, such as image processing unit 42 and treatment generator 44. In an example, port 40A can be used to insert another surgical instrument 48 or device, such as a daughter scope or auxiliary scope, into endoscope 14. Such instruments and devices can be independently connected to control unit 16 via cable 47. In an example, port 40B can be used to connect coupler section 36 to various inputs and outputs, such as video, air, light, and electricity.

[0064] Image processing unit 42 and light source unit 22 can each interface with endoscope 14 via wired or wireless connection (e.g., at functional section 30). Therefore, imaging and control system 12 can illuminate the anatomical region, collect signals representing the anatomical region, process signals representing the anatomical region, and display an image representing the anatomical region on display unit 18. Imaging and control system 12 may include light source unit 22 to illuminate the anatomical region using light of a desired spectrum (e.g., broadband white light, narrowband imaging using preferred electromagnetic wavelengths, etc.). Imaging and control system 12 can be connected to endoscope 14 (e.g., via endoscope connector) for signal transmission (e.g., light output from the light source, video signals from the imaging system at the distal end, diagnostic and sensor signals from diagnostic devices, etc.).

[0065] During an endoscopic procedure, the treatment generator 44 can generate a treatment plan, which can be used by the control unit 16 to control the operation of the endoscope 14 or to provide guidance to the operator for manipulating the endoscope 14. In this example, the treatment generator 44 can use patient information, including images of the target anatomy, to generate an endoscopic navigation plan that includes estimates of one or more cannulation or navigation parameters (e.g., angles, forces, etc.) for manipulating a steerable, elongated instrument. The endoscopic navigation plan can help guide the operator in cannulation and navigation of the endoscope within the patient's anatomy. The endoscopic navigation plan can be additionally or alternatively used to robotically adjust the position, angle, force, and / or navigation of the endoscope or other instruments.

[0066] Figure 3 This is a block diagram illustrating an example of an endoscope system 300 that can create personalized, segment-specific endoscopic withdrawal plans and use these plans as guidance during endoscopic withdrawal. In the example, the endoscope system 300 can be used during colonoscopy to provide guided mucosal examination and necessary treatments (e.g., polyp removal) during withdrawal. The system 300 can be implemented as... Figure 1 It is part of the control unit 16.

[0067] System 300 may include one or more of an endoscope 310, an auxiliary input 315, a controller circuit 320, a user interface 330, and a storage device 340. In some examples, system 300 may also include or be communicatively coupled to a robotic system 350 in the process of robot-assisted endoscopy.

[0068] Endoscope 310 can be as described above and as follows Figures 1 to 2 An example of endoscope 14 is shown. Furthermore, endoscope 310 may include an imaging system 312 and an illumination system 314. Imaging system 312 may include at least one imaging sensor or device (e.g., a camera device) configured to acquire endoscopic images or video streams of a patient's target anatomical structure during an endoscopic examination. The imaging sensor or device may be located at a distal portion or distal end of endoscope 310. Illumination system 314 may include one or more light sources to provide illumination on the target anatomical structure through one or more illumination lenses. In some examples, imaging system 312 may be controllably adjusted to operate in one of several different imaging modes. These imaging modes may differ in one or more aspects of zoom settings, contrast settings, exposure levels, or the angle of view toward the target anatomical structure. In some examples, illumination system 314 may be controllably adjusted to provide different lighting or illumination conditions.

[0069] An endoscopic examination procedure typically includes an insertion phase, in which the endoscope is inserted into a body cavity through a natural orifice; and a subsequent withdrawal phase, in which the endoscope is withdrawn from the body through the body cavity and the natural orifice. Imaging system 312 can acquire image or video streams during the insertion phase (hereinafter referred to as "insertion image or video stream") and during the withdrawal phase (hereinafter referred to as "withdrawal image or video stream"). Insertion and withdrawal image or video streams can be used in various applications. When the target anatomical structure comprises multiple anatomically distinct or artificially defined segments or portions, imaging system 312 can acquire image or video streams from each of those distinct segments of the anatomical structure during the insertion phase (hereinafter referred to as "segment-specific insertion image or video stream") or during the withdrawal phase (hereinafter referred to as "segment-specific withdrawal image or video stream"). As will be discussed further below, during colonoscopy, segment-specific inserted images or video streams can be used to detect abnormalities at any particular segment, and based at least in part on the detected abnormalities, segment-specific endoscopic withdrawal plans can be created that define withdrawal speed or time limits at each segment. Segment-specific withdrawal images or video streams can be analyzed to locate the endoscope position and track real-time withdrawal speeds substantially in real time. Using segmental withdrawal plans as a reference, endoscopists can receive timely feedback on endoscopic withdrawal and recommendations for adjusting the withdrawal.

[0070] Controller circuitry 320 may include a circuit set comprising one or more other circuits or sub-circuits that can individually or in combination perform the functions, methods, or techniques described herein. In an example, controller circuitry 320 and the circuit set therein may be implemented as part of a microprocessor circuit, which may be a dedicated processor such as a digital signal processor, application-specific integrated circuit (ASIC), microprocessor, or other type of processor for processing information including bodily activity information. Alternatively, the microprocessor circuit may be a general-purpose processor that can receive and execute a set of instructions for performing the functions, methods, or techniques described herein. In an example, the hardware of the circuit set may be immutably designed to perform a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variable-connected physical components (e.g., execution units, transistors, simple circuits, etc.) including computer-readable media that are physically modified (e.g., magnetically grounded, electrically grounded, movable placement of massless particles, etc.) to encode instructions for a specific operation. When connecting physical components, the fundamental electrical properties of the hardware composition change, for example, from an insulator to a conductor, and vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of a circuit set within the hardware via variable connections to perform portions of a specific operation during operation. Thus, when the device is operational, a computer-readable medium is communicatively coupled to other components of the circuit set member. In the example, any of the physical components can be used in more than one member of more than one circuit set. For example, under operation, an execution unit can be used at one point in time in a first circuit of a first circuit set and reused by a second circuit of the first circuit set or at a different time by a third circuit of a second circuit set.

[0071] The controller circuit 320 may include one or more of the following: an image processor 321, an anomaly detector circuit 322, an endoscope positioning circuit 323, an withdrawal planning generator 324, a real-time endoscope withdrawal tracker circuit 327, and an endoscope withdrawal controller 328. The image processor 321 may analyze segment-specific image or video streams obtained from the imaging system 312 and generate endoscopic image or video features for individual segments within different segments of an anatomical structure. Examples of image or video features include statistical or morphological features of pixel values, such as corners, edges, spots, curvature, accelerated robust feature transform (SURF) or scale-invariant feature transform (SIFT) features, etc. In some examples, the image processor 321 may preprocess the segment-specific image or video stream, such as filtering, resizing, or orienting, or color or grayscale correction, and may extract endoscopic image or video features from the preprocessed image or video stream. In some examples, the image processor 321 can post-process image or video features to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0072] Anomaly detector circuit 322 can detect abnormalities at any specific segment of an anatomical structure, at least in part, based on segment-specific endoscopic image or video features. Anomaly detection includes detecting and / or identifying one or more of the following: the presence or absence, type, size, shape, location, or quantity of pathological tissue or anatomical structures and other objects in the environment of the anatomical structure. In an example of colonoscopy, detected abnormalities may include segments of pathological tissue, such as mucosal abnormalities (polyps, inflammatory bowel disease, Merkel's diverticulum, lipomas, hemorrhage, vascularized mucosa, etc.) or obstructive mucosa (segments used for, for example, poor bowel preparation, colonic dilatation, etc.). Anomaly detector circuit 322 can use segment-specific images or video streams (or features extracted from them) obtained before the endoscope is withdrawn to a specific segment to detect abnormalities in that segment of the anatomical structure. During the insertion phase, such segment-specific images or video streams typically include segment-specific insertion images or video streams, or features generated therefrom. In some examples, certain segment-specific withdrawal images or video streams, or features generated therefrom, may also be used to detect abnormalities.

[0073] In the example, the anomaly detector circuit 322 can use template matching techniques to detect anomalies, where anomalies can be identified based on a comparison of segment-specific endoscopic features (e.g., features characterizing the shape or contour of a structure) with one or more pre-generated templates of known anomalous structures. In another example, the anomaly detector circuit 322 can use techniques based on artificial intelligence (AI) or machine learning (ML) to detect anomalies. Features from segment-specific endoscopic images or video streams, or extracted from them, can be applied to a trained ML model to automatically identify the presence or absence, type, size, location, and / or other features of anomalies. In the example, the ML model can be trained to establish a correspondence between features from endoscopic images or video streams, or extracted from them, and one or more anomalous features. Examples of ML models used to identify anomalies from endoscopic images or video streams include convolutional neural networks, bidirectional LSTMs, recurrent neural networks, conditional random fields, dictionary learning, or other machine learning techniques (support vector machines, Bayesian models, decision trees, k-means clustering), and other ML techniques. The trained ML model can be stored in storage device 340. The following is about... Figures 6A to 6B This paper discusses examples of training ML models and using trained ML models to detect anomalies, determine segment-specific withdrawal parameter values, and generate personalized segmented withdrawal plans.

[0074] Endoscope positioning circuit 323 can determine the position of the endoscope within a specific segment of an anatomical structure in essentially real-time. Using this real-time endoscope position information, anomaly detector circuit 322 can associate detected abnormalities with specific segments of the anatomical structure. Endoscope position can be determined using electromagnetic tracking or by detecting anatomical landmarks based on features detected from an image or video stream, such as those acquired during the insertion phase of an endoscopic examination or extracted by image processor 321. In an example of colonoscopy, endoscope positioning circuit 323 can analyze the image or video stream to identify colonic landmarks, such as the anus, left ascending colon, splenic flexure, transverse colon, hepatic flexure, right descending colon, cecum, appendix, and terminal ileum. In this example, endoscope positioning circuit 323 can use template matching techniques to identify landmarks. In another example, endoscope positioning circuit 323 can use AI or ML techniques, such as a trained ML model trained to establish correspondences between endoscopic images or video streams or features extracted from them and landmark recognition, to identify landmarks. Examples of ML models used for identifying anatomical landmarks include deep belief networks, ResNet, DenseNet, autoencoders, capsule networks, generative adversarial networks, Siamese networks, convolutional neural networks (CNNs), deep reinforcement learning, support vector machines (SVMs), Bayesian models, decision trees, k-means clustering, and other ML models. The trained ML model can be stored in storage device 340. Once the endoscope position is determined, endoscope localization circuitry 323 can register the substantially real-time endoscope position to a pre-generated template of the anatomical structure. Information about the endoscope position within segments of the anatomical structure can be presented to a user (e.g., an endoscopist) on user interface 330, such as information about... Figure 5 Further discussion is needed.

[0075] The withdrawal planning generator 324 can generate a personalized segmental withdrawal plan using information about detected anomalies, such as those generated by the anomaly detector circuit 322, and the substantially real-time position of the endoscope when the anomaly is detected, such as that generated by the endoscope positioning circuit 323. The personalized segmental withdrawal plan can include target or suggested endoscopic withdrawal parameter (P) values ​​for individual segments within different segments of the anatomical structure. The target or suggested segment-specific endoscopic withdrawal parameter values ​​can be estimated using the withdrawal parameter estimator 325. In one example, the endoscopic withdrawal parameter P includes a target or suggested segment-specific endoscopic withdrawal speed limit (WSL). WSL represents the upper limit of speed or an acceptable range of withdrawal speeds for any particular segment of the anatomical structure. In another example, the endoscopic withdrawal parameter P includes a target or suggested segment-specific endoscopic withdrawal time limit (WTL). WTL represents the maximum permissible time or an acceptable range of time for the endoscope to remain in any particular segment of the anatomical structure during withdrawal. In the example of colonoscopy, segment-specific endoscopic withdrawal parameters may have corresponding target or recommended withdrawal parameter values, including, for example, P for the cecal segment. C P in the ascending colon segment A P in the transverse colon segment T P in the descending colon segment D P in the sigmoid colon segment S P and rectosigmoid colon segment R Other segments or sub-segments can be specified by the user, and the corresponding target or suggested withdrawal parameter values ​​can be determined by the withdrawal parameter estimator 325.

[0076] The withdrawal parameter estimator 325 can use AI or ML methods to determine target or proposed segment-specific endoscopic withdrawal parameter (e.g., WSL or WTL) values ​​for each segment within different segments. In the example, the results of anomaly detection from anomaly detector circuit 322, along with sign detection and real-time endoscopic position information from endoscope positioning circuit 323, can be applied to a trained ML model. The ML model can have been trained to establish a correspondence between anomaly features and target or proposed segment-specific endoscopic withdrawal parameter (e.g., WSL or WTL) values. The trained ML model can be stored in storage device 340. The following is about... Figures 6A to 6B This paper discusses examples of training ML models and using trained ML models to detect anomalies, determine segment-specific withdrawal parameter values, and generate personalized segmented withdrawal plans.

[0077] In some examples, the withdrawal parameter estimator 325 may use one or more anomalous features, such as the type, size, shape, location, or number of anomalous features, to calculate an anomalous score. The anomalous score may have a numerical value, for example, in the range of 0 to 10. In some examples, a composite anomalous score may be generated, for example, using a linear or non-linear combination of multiple anomalous scores, each quantifying annomalous feature. In examples, a composite anomalous score may be calculated by averaging or taking the worst-case anomalous score within the segment. The withdrawal parameter estimator 325 may map the anomalous score (or composite anomalous score) to withdrawal parameter values ​​based on comparisons with one or more thresholds or ranges of values. A higher anomalous score, typically indicating a more severe anomalous condition, may be mapped to a lower WSL to facilitate a slower withdrawal in that segment, or to a longer WTL to facilitate a longer withdrawal time in that segment. The established correspondence between the anomalous score or range of anomalous scores and the corresponding target or recommended segment-specific endoscopic withdrawal parameter value may be determined by a lookup table in a specification database or via a rule-based system. The established mapping may be stored in storage device 340. In another example, the withdrawal parameter estimator 325 can apply anomaly scores or composite anomaly scores to a trained ML model to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for individual segments within different segments. In some examples, anomaly score calculation can be included within the ML model, such as within a layer of a neural network model.

[0078] In some examples, the withdrawal parameter estimator 325 can further use auxiliary input 315 to determine target or suggested endoscopic withdrawal parameter values ​​for individual segments within different segments of the anatomical structure. (See reference...) Figure 4 As an example and not a limitation, auxiliary input 315 may include image or video streams and clinical data from previous endoscopic procedures 410 performed on the patient, patient information and medical history 420, or preoperative imaging study data 430 (e.g., X-ray or fluorescence images, potentiograms or impedance maps, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and other imaging modalities). In the example of colonoscopy, auxiliary input may include monitoring previous colonoscopies in the case, including polyps left in situ by the endoscopist or the colonic region where surgery was performed. In a typical endoscopic scenario, patient information and medical history 420 may include clinical demographic information, past and current indications, and treatments received, etc.

[0079] In some examples, auxiliary input 315 may additionally or alternatively include a user (e.g., endoscopist) profile 440, which includes the user's experience, work environment (e.g., hospital setup or mobile screening center), adaptability to new technologies, preferences for specific endoscopic procedures, etc. The user profile 440 may be provided by the user via user interface 330. Alternatively, the user profile 440 may be automatically generated or updated by learning from the user's past choices and training. For example, an experienced endoscopist may prefer a higher WSL (corresponding to a higher withdrawal speed) or a lower WTL (corresponding to a shorter examination time) at a particular segment, while a trained endoscopist may be advised to a lower WSL (corresponding to a slower withdrawal speed) or a higher WTL (corresponding to a longer examination time).

[0080] In some examples, auxiliary input 315 may additionally or alternatively include endoscope and device information 450. This may include, for example, specifications of endoscope 310, including the type, size, shape, and structure of the endoscope or other steerable instruments (e.g., cannulas, catheters, or guide wires that support imaging and illumination modes); specifications of the size, shape, and structure of tissue sections, sampling, or treatment tools; and the current status of the device, including which illumination mode is on, which endoscope buttons such as water jet or inflation are engaged, which illumination modes are supported, or whether magnification is on and the current magnification selection is available.

[0081] If there are additional AI or ML algorithms running in the background, the state of these AI or ML algorithms (hereinafter collectively referred to as "AI Discovery" 460) can be an additional element of the auxiliary input 315, which can be passed to the withdrawal planning generator 324. Examples of AI Discovery 460 may include AI algorithms for detecting anomalies, landmarks, or other features or structural elements of interest in a target anatomical structure.

[0082] Estimated segment-specific target withdrawal parameter values ​​(e.g., segment-specific WSL or WTL) can be used to generate an endoscope withdrawal map 326, which is a graphical representation of the endoscope withdrawal plan. The endoscope withdrawal map 326 can be displayed on a user interface 330 and used as a visual guide to assist the endoscopist during the endoscopic procedure. The endoscope withdrawal map 326 may include depictions of different segments of the anatomical structure, each segment being color-coded or grayscale-coded to indicate the corresponding target or suggested segment-specific endoscope withdrawal parameter value. The endoscope withdrawal map 326 can be displayed on the user interface 330 as a reference to guide the endoscopist in withdrawing the endoscope during the endoscopic procedure. In some examples, estimated withdrawal parameter values ​​generated by the withdrawal parameter estimator 325 can be displayed on the user interface 330. In some examples, information about abnormalities detected previously (e.g., during the insertion phase of the endoscopy or from a previous endoscopic procedure) may also be displayed on the endoscope withdrawal diagram 326 as an additional preventative layer to prevent excessive withdrawal speed in abnormal segments. Other information may be displayed on the user interface 330, including endoscopic images and image features, information about abnormalities and landmarks detected in each segment of the anatomical structure, or the substantially real-time position of the endoscope during the endoscopic procedure.

[0083] In some examples, the segmented withdrawal plan, including estimated withdrawal parameter values ​​and endoscopic withdrawal diagram 326, can be stored in storage device 340. Information about previously detected anomalies (e.g., during the insertion phase) can also be stored in storage device 340. Storage device 340 can be located locally within the endoscopy system 300. Alternatively, storage device 340 can be a separate remote storage device, such as part of a cloud comprising one or more storage and computing devices (e.g., servers) that provides secure access to cloud-based services, including, for example, data storage, computing services, and the provision of customer service. In some examples, at least some of the data processing and computations related to anomaly detection, sign recognition, endoscopy localization, withdrawal parameter estimation, and endoscopic withdrawal diagram generation can be performed in the cloud. For example, an image or video stream, or features extracted from it, can be streamed to the cloud, processed therein, and the computational results, such as the segmented withdrawal plan (e.g., endoscopic withdrawal diagram 326), can be relayed back to the local endoscopy system for image-guided endoscopic withdrawal.

[0084] The real-time endoscopic withdrawal tracker circuit 327 can track and measure withdrawal parameters (e.g., withdrawal velocity or total withdrawal time) of specific segments of an anatomical structure, such as a colonic segment, substantially in real time. Various techniques can be used to track and measure endoscopic withdrawal velocity. In one example, an optical flow-based method can be used to estimate endoscopic withdrawal velocity. Optical flow is the pattern of apparent motion of an image object between two consecutive frames caused by the movement of the object or camera device. Endoscopic withdrawal velocity is positively correlated with the rate of change of endoscopic image frames. A faster rate of change of frames corresponds to a faster flow, or endoscopic withdrawal velocity. In another example, electromagnetic (EM) tracking techniques can be used to estimate endoscopic withdrawal velocity. An EM coil, incorporated along the length of the endoscope's insertion cannula, generates a pulsed low-intensity magnetic field that can be picked up by a receiver device. The EM pulses are used to calculate the precise position and orientation of the insertion cannula. The endoscopic withdrawal velocity can then be estimated based on the rate of change of the endoscope's position. In yet another example, the endoscope withdrawal speed can be estimated relative to a marker, such as one identified from the endoscope image by the endoscope positioning circuit 323 as described above. Using the marker as a reference or benchmark, the endoscope withdrawal speed can be estimated based on the change in its relative position to the identified marker.

[0085] In some examples, the real-time endoscopic withdrawal tracker circuit 327 can determine the start of the withdrawal phase and activate withdrawal velocity tracking and measurement only during the withdrawal phase. The real-time endoscopic withdrawal tracker circuit 327 can additionally determine whether the endoscopist is performing any procedures. Examples of such procedures include, but are not limited to, water jetting and debris aspiration, insertion of instruments (forceps, snares, cell brushes, instruments required for sclerotherapy or mucosal injection, and aspiration catheters) via the working channel. Once the withdrawal phase is identified, the withdrawal clock starts and activates withdrawal velocity tracking as the mucosa is examined. If the colonoscopy is in another phase, withdrawal velocity tracking can be put into hibernation so that it does not interfere with the endoscopist's actions.

[0086] During withdrawal, the endoscope positioning circuit 323 can determine, at least in part, the position of the endoscope in a different segment of the anatomical structure based substantially in real time on endoscopic images or video streams or features extracted therefrom. As discussed above similarly regarding positioning the endoscope during the insertion phase, the endoscope position can be determined based on anatomical landmarks identified from endoscopic images or video streams. The substantially real-time position of the endoscope can be registered to a pre-generated template of the anatomical structure. The substantially real-time position of the endoscope indicates the current segment of the anatomical structure from which the endoscopic images or video streams are obtained. Measured real-time endoscope withdrawal parameter values ​​(e.g., withdrawal velocity) generated by the real-time endoscope withdrawal tracker circuit 327 can be correlated with the currently identified segment to produce measured segment-specific withdrawal parameter values. The endoscope withdrawal controller 328 can compare the measured segment-specific withdrawal parameter values ​​with target or suggested segment-specific withdrawal parameter values ​​and determine whether the measured withdrawal parameter values ​​are within a specific margin of the target values. For example, the endoscope withdrawal controller 328 can measure the endoscope withdrawal speed S in segment "j". j Withdrawal speed limit (WSL) for the same segment "j" j Compare them. If the measured withdrawal speed S j Landed in WSL j Within the specified margin δ, i.e., WSL j - δ≤ S j ≤ WSL j + δ, then S j It is considered appropriate. When S j Deviation from WSL j When the specified margin δ is exceeded, i.e., S j WSL j - δ or S j WSL j + δ can generate an alert to remind the user of an inappropriately fast withdrawal speed at segment "j" (if S j WSL j + δ) or an inappropriately slow withdrawal speed (if S j WSL j - δ). Suggestions can be provided to users to adjust the withdrawal speed to substantially match the target speed WSL. j (For example, WSL) j - δ ≤ S j ≤ WSL j + δ).

[0087] In some examples, the segment-specific endoscope withdrawal rate S measured during the endoscope withdrawal phase jThe substantially real-time endoscopic position within the anatomical segment (as determined by the endoscopic positioning circuit 323) can be displayed on the user interface 330 along with the endoscopic withdrawal diagram 326 (e.g., side-by-side or superimposed on the endoscopic withdrawal diagram 326). This provides the user with direct visual feedback regarding endoscopic withdrawal, as described below. Figure 5 Further examples and discussions are provided. In some examples, the user can be notified of the impending abnormal segment during withdrawal and before withdrawing the endoscope beyond a previously identified abnormal segment (e.g., an abnormal segment identified during the insertion phase), and preemptively reminded to withdraw at a rate not exceeding the segment-specific WSL as the abnormal segment is passed. The reminder to the user can be delivered via optical devices on the diagnostic monitor or via auditory devices such as alert alarms. In examples, the user can be provided with highlighting, flashing warnings, auditory or tactile feedback to emphasize the impending abnormal segment. Upon completion of withdrawal (e.g., rectum detected), the endoscopist can be shown the total net withdrawal time, postoperative analysis, and withdrawal summary, including, for example, segment-by-segment withdrawal time and / or average speed, total net withdrawal time and average speed, detected abnormalities, etc. Postoperative analysis can be used for quality assurance and to determine the success of the colonoscopy procedure.

[0088] In some examples, system 300 can operate in a closed-loop manner, where the feedback loop is used to continuously learn segmental withdrawal planning, such as updating target or suggested segment-specific endoscopic withdrawal parameter values. Continuous learning can be via explicit endoscopist feedback (e.g., satisfaction with the suggestion, a "like" button, etc.). Alternatively, system 300 can be run in shadow mode for experienced endoscopists and their withdrawal actions can be monitored, with segmental withdrawal planning corrected through reinforcing feedback.

[0089] In some examples, the robotic system 350 may be used to perform image-guided endoscopy or a portion thereof, such as endoscope withdrawal. The robotic system 350 may include a robotic arm detachably attached to the endoscope 310. The robotic arm may automatically or semi-automatically (e.g., using some user manual control or command) locate and manipulate the endoscope 310 in an anatomical target, or position the device in a desired orientation to facilitate manipulation of the anatomical target, via actuators. The robotic system 350 may include a robot controller to control the movement of the robotic arm according to a segmental withdrawal plan, which includes estimated withdrawal parameter values ​​generated by the withdrawal parameter estimator 325 and / or the endoscope withdrawal diagram 326.

[0090] Figure 5The different stages of an image-guided endoscopic procedure 500 (e.g., a colonoscopy as illustrated in this example) are shown by way of example. The procedure includes an insertion phase followed by a withdrawal phase. This procedure can be performed using system 300. During the insertion phase, endoscopic images or video streams can be acquired from each of a plurality of colonic segments, including the rectosigmoid colon, sigmoid colon, descending colon, transverse colon, ascending colon, and cecum, for example using imaging system 312. Endoscope positioning circuitry 323 can identify the various colonic segments. Furthermore, abnormalities can be automatically detected during insertion using an anomaly detector circuitry 322. At the end of the insertion phase, when the distal tip of the endoscope has reached the cecum, the segment-specific images or video streams acquired during the insertion phase can be analyzed, and a personalized segmental withdrawal plan can be generated, for example using a withdrawal plan generator 324. The segmental withdrawal plan can be graphically represented by a colonoscope withdrawal speed diagram 510 (an example of an endoscope withdrawal diagram 326). The withdrawal speed limit (WSL) for each of the multiple colonic segments can be color-coded (or grayscale-coded) and displayed on the colonoscopy withdrawal speed map 510. In, as shown... Figure 5 In the example shown, the color-coded or grayscale-coded WSL can include a “normal” or relatively fast WSL range, a “cautious” or moderate WSL range, and a “slow” withdrawal WSL range. As an example and not a limitation, the “normal” WSL range is approximately 3 mm / sec to 4 mm / sec, the “cautious” WSL range is approximately greater than or equal to 40 mm / sec, and the “slow” WSL range is approximately 1 mm / sec to 3 mm / sec. Other information obtained or calculated, such as abnormalities detected during the insertion phase or based on previous endoscopic procedures, or identified abnormal segments, can also be displayed on the colonoscopy withdrawal speed diagram 510.

[0091] In some examples, the image or video stream used to determine target or recommended endoscopic withdrawal parameter values ​​(and to generate personalized segmental withdrawal plans, such as colonoscopy withdrawal velocity diagram 510) may not be limited to endoscopic image or video streams acquired during endoscopic insertion, but may include endoscopic image or video streams acquired during the withdrawal phase before the endoscope is withdrawn beyond the segment to be examined and treated. For example, a target or recommended withdrawal velocity limit (WSL) for any particular colonic segment “j” may be determined at any time before the colonoscope is withdrawn beyond that segment “j”.

[0092] The colonoscopy withdrawal speed diagram 510 can be displayed to the user throughout the withdrawal phase to assist the endoscopist in manipulating the endoscope to examine the mucosa in the colonic segment and perform necessary treatments (e.g., polyp removal). Additionally or alternatively, the colonoscopy withdrawal speed diagram 510 can be used to control the robotic system 350 in robot-assisted colonoscopy. Figure 5 As shown, during withdrawal, the substantially real-time endoscope position 522, such as that determined by the endoscope positioning circuit 323, can be displayed as a superposition on the colonoscope withdrawal speed map 510 to produce a superimposed map 520. When the endoscope tip is at or about to reach a specific colonic segment, a notification such as a marker 524 can be displayed on the user interface to inform the user of the WSL at the current or upcoming segment. At each colonic segment, the actual endoscopic withdrawal speed can be measured, for example, using the real-time endoscopic withdrawal tracker circuit 327. The measured segment-specific withdrawal speed can be compared with the WSL stored in a personalized segmental withdrawal plan, and it can be determined whether the measured speed is within a specific margin of the WSL for that segment. As described above, when the measured speed exceeds the specific margin of the WSL, an alert can be generated to remind the user (e.g., an endoscopist) of the inappropriate withdrawal speed. Suggestions can be provided to the user to adjust the withdrawal speed to substantially conform to the WSL.

[0093] Figures 6A to 6B Examples of training an ML model and using the trained ML model to determine target or suggested segment-specific endoscopic withdrawal parameter values ​​(e.g., WSL or WTL) for each of several different segments of an anatomical structure are shown. Target or suggested segment-specific endoscopic withdrawal parameter values ​​can be used to construct endoscopic withdrawal diagram 326 or colonoscopy withdrawal velocity diagram 510. Figure 6A The diagram illustrates the ML model training (or learning) phase, during which the ML model 620 can be trained to determine, at least in part, a target or suggested colonoscopy withdrawal speed limit (WSL) at a specific segment “j” (e.g., the cecum) based on endoscopic images of that specific segment “j”. The training dataset may include a set of endoscopic images 610 of the same colonic segment “j” obtained from colonoscopy procedures performed on multiple patients. In some examples, the training dataset may also include images as described above regarding… Figure 3The auxiliary input 315 is described. The ML model 620 may have a neural network structure including an input layer, one or more hidden layers, and an output layer. Multiple endoscopic images 610 or features generated therefrom can be fed into the input layer of the ML model 620, which propagates the input data or data features through one or more hidden layers to the output layer, which outputs the WSL of a colonic segment (e.g., the cecum). The ML model 620 is capable of performing tasks by making inferences based on patterns discovered in data analysis without explicit programming. The ML model 620 explores the research and construction of algorithms (e.g., ML algorithms) that can learn from existing data and make predictions on new data. Such algorithms operate by constructing the ML model 620 based on training data to make data-driven predictions or decisions represented as outputs or evaluations.

[0094] ML models can be trained using either supervised or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples that correlate inputs with outputs or outcomes) to learn the relationship between inputs and outputs. The goal of supervised learning is to learn a function that, given some training data, best approximates the relationship between training inputs and outputs, so that the ML model can achieve the same relationship to generate the corresponding output given the input. Unsupervised learning trains the ML algorithm using information that is neither classified nor labeled, allowing the algorithm to process that information without guidance. Unsupervised learning is useful in exploratory analytics because it can automatically identify structures in the data.

[0095] Common tasks for supervised learning are classification and regression problems. Classification problems (also known as categorization problems) aim to classify items into one of several class values. Regression algorithms aim to quantify some items (e.g., by providing scores to some input values). Some examples of commonly used supervised ML algorithms are logistic regression (LR), Naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM). Examples of DNNs include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN), or hybrid neural networks that include two or more neural network models of different types or different model configurations. Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0096] Another type of machine learning is federated learning (also known as collaborative learning), which trains an algorithm across multiple distributed devices that store local data without exchanging data. This approach contrasts with traditional centralized machine learning techniques that upload all local datasets to a single server, as well as more classic distributed methods that typically assume local data samples are distributed similarly. Federated learning enables multiple participants to build a shared, robust machine learning model without sharing data, thus allowing for the resolution of critical issues such as data privacy, data security, data access permissions, and access to heterogeneous data.

[0097] Training of the ML model 620 can be performed continuously or periodically, or near real-time when additional process data is available. The training process involves algorithmically adjusting one or more ML model parameters (e.g., weights or biases at any specific layer of a neural network model) until the trained ML model meets a specified training convergence criterion. As an example, and not a limitation, the ML model 720 can be trained using weighted squared loss (for explicit feedback) or binary cross-entropy loss (for implicit feedback). Other training techniques can be used, such as deep factorization machines, breadth and deep learning, deep structured semantic models, or autoencoder-based suggestion systems. The trained ML model 620 can establish a correspondence between endoscopic images 610 (or features extracted from them) and the target or proposed WSL for segment "j".

[0098] Similar training procedures and techniques as described above can be used to train other ML models using corresponding training datasets, each consisting of a set of endoscopic images of the same colonic segment (e.g., segment "k" different from segment "j" above) obtained from colonoscopy procedures performed on multiple patients. The resulting multiple trained ML models can each predict the target or proposed WSL for their respective colonic segments. For example, a first ML model is used to predict the WSL in the cecum of the colon; a second ML model is used to predict the WSL in the ascending colon segment; a third ML model is used to predict the WSL in the transverse colon segment; a fourth ML model is used to predict the WSL in the descending colon segment; a fifth ML model is used to predict the WSL in the sigmoid colon segment; and a sixth ML model is used to predict the WSL in the rectosigmoid colon segment.

[0099] Figure 6B The inference phase is illustrated, during which real-time endoscopic images 630 of a specific segment are applied to a trained ML model 620 to automatically determine the segment-specific WSL in segment "j", i.e., the WSL. j 640. Real-time endoscopic images 630 can be obtained during the insertion phase of the endoscopic examination, allowing the WSL to be determined at the end of the insertion phase.j 640. Alternatively, real-time endoscopic images 630 can be obtained during the withdrawal phase before the endoscope is withdrawn beyond segment "j", making it possible to determine the WSL at any time before the colonoscope is withdrawn beyond segment "j". j 640. WSL j 640 and WSL estimated for other segments by applying images of the corresponding colonic segments to the corresponding trained ML model can be used to construct personalized colonoscopy withdrawal velocity maps 510.

[0100] During withdrawal, the endoscope position can be determined substantially in real time, for example, using endoscope positioning circuit 323. The endoscope position can be displayed on top of colonoscope withdrawal speed graph 510 to produce an overlay graph 520 that can be displayed to the endoscopist during the procedure. The endoscope withdrawal speed can be tracked and measured substantially in real time, for example, using real-time endoscope withdrawal tracker circuit 327. The actual withdrawal speed S is measured when the endoscope is withdrawn to segment "j". j 650. S can be... j 650 and WSL j 640 were compared to determine S j Is 650 in WSL? j Within a specific margin of 640. As mentioned above, if the measured rate falls within the recommended margin of the WSL, the withdrawal rate is considered appropriate. If the measured rate exceeds the recommended margin of the WSL, an alert can be generated to remind the user (e.g., an endoscopist) of an inappropriate withdrawal rate. Suggestions can be provided to the user to adjust the withdrawal rate to substantially conform to the WSL.

[0101] The aforementioned ML models are trained to estimate target or suggested segment-specific endoscopic withdrawal parameter values ​​(e.g., WSL or WTL) for any specific segment of an anatomical structure. In some examples, multiple ML models can be trained, validated, and used (during the inference phase) in other applications such as anomaly detection or anatomical landmark recognition. In one example, an ML model can be trained and used by anomaly detector circuit 322 to detect anomalies based on an input endoscopic image of a specific colonic segment, and another ML model can be trained and used by endoscopy localization circuit 323 to detect anatomical landmarks based on an input endoscopic image of a specific colonic segment.

[0102] Figures 7A to 7BExample graphs of guided withdrawal and speed tracking during a colonoscopy using a personalized segment-specific endoscopic withdrawal plan are shown. The actual colonoscopy withdrawal speed (e.g., the speed tracked and measured by the real-time endoscopic withdrawal tracker circuit 327) and the target or recommended WSL for each of multiple (e.g., N) segments can be plotted on the same graph and displayed to the user during withdrawal. Figure 7A The diagram illustrates a “good” endoscopic procedure in which the actual withdrawal rate 710 measured at each of the N segments remains below the corresponding segment-specific WSL (e.g., WSL1 720A for segment 1, WSL2 720B for segment 2, WSL3 720C for segment 3, WSL4 720D for segment 4, WSL5 720E for segment 5, ..., WSL for segment N). N 720N). Figure 7B The diagram illustrates withdrawal speed tracking in another process. The actual withdrawal speed 730 in segments 1 through 4 is lower than the corresponding segment-specific WSL (i.e., WSL1 through WSL4). However, in segment 5, the actual measured withdrawal speed 732 exceeds the speed limit WSL5 720E. In response, an alert can be triggered, prompting the endoscopist to slow the withdrawal in that segment. Since the endoscope has already examined a portion of segment 5 at speed 732, in order to re-examine that portion at a slower speed, the endoscope is reinserted to the beginning of segment 5 (or the end of the previous segment 4) and withdrawn from there through segment 5 at a slower speed 734, below WSL5 720E.

[0103] Figure 8 This is a flowchart illustrating an example method 800 for creating a personalized segment-specific endoscopic withdrawal plan and using said plan to guide endoscopic withdrawal during an endoscopic procedure (e.g., a colonoscopy). Personalized segment-specific endoscopic withdrawal plans can be created using image or video streams of different segments of an anatomical structure. Method 800 can be implemented in an endoscope system 300. Although the procedures of method 800 are depicted in a flowchart, they do not need to be performed in a specific order. In various examples, some of the procedures may be performed in a different order than that shown herein.

[0104] At point 810, an imaging system associated with an endoscope can be used to acquire images or video streams of different segments of the target anatomical structure. In the example of colonoscopy, different segments (e.g., one or more of the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, or rectosigmoid colon segments) can be imaged individually to produce segmental image or video streams or segment-specific image or video streams. Image or video streams can be acquired when the imaging system is set to one of a plurality of available imaging modes. An imaging mode refers to one or more of the following: zoom setting, contrast setting, exposure level, viewing angle toward the target anatomical structure, or lighting or illumination conditions. In the example, image or video streams can be acquired during the insertion phase of the endoscopic procedure, wherein segmental insertion image or video streams or segment-specific insertion image or video streams can be acquired from each of a plurality of segments (e.g., colonic segments).

[0105] At point 820, the acquired segmental image or video stream can be analyzed to generate endoscopic image or video features for individual segments within different segments of an anatomical structure. Examples of image or video features include statistical or morphological features of pixel values, such as corners, edges, spots, curvature, accelerated robust feature transfer (SURF) features, or scale-invariant feature transform (SIFT) features. In some examples, segment-specific image or video streams can be preprocessed, such as filtered, resized, or oriented, or color or grayscale corrected, and endoscopic image features can be extracted from the preprocessed image or video stream. In some examples, endoscopic image or video features can be post-processed to enhance feature quality, such as edge interpolation or extrapolation to produce continuous and smooth edges.

[0106] At point 830, at least in part based on endoscopic image or video features, personalized segmental endoscopic withdrawal plans can be generated, for example using... Figure 3 The withdrawal plan generator 324 shown is used for implementation. Segmental endoscopic withdrawal planning can include target or suggested segment-specific endoscopic withdrawal parameter values ​​for individual segments within different segments of the anatomical structure. This is for example and not a limitation, and as mentioned above regarding... Figure 3The target or recommended segment-specific endoscopic withdrawal parameters discussed may include target or recommended segment-specific withdrawal speed limits (WSL) or target or recommended segment-specific withdrawal time limits (WTL) for individual segments within different segments. WSL represents the upper limit or acceptable range of withdrawal speeds for individual segments within different anatomical structures. WTL represents the maximum permissible time or acceptable range of time the endoscope is held in individual segments within different segments during the endoscopic withdrawal procedure. In the example of colonoscopy, segment-specific endoscopic withdrawal parameters may have corresponding target or recommended values, such as WSL or WTL, for one or more cecal segments, ascending colon segments, transverse colon segments, descending colon segments, sigmoid colon segments, or rectosigmoid colon segments. Other segments or subsegments may be user-defined, and corresponding target or recommended withdrawal parameter values ​​may be determined similarly.

[0107] In the examples, information about abnormalities detected based on any specific segment of the anatomical structure can be used to estimate target or recommended segment-specific endoscopic withdrawal parameter values. Template matching techniques or artificial intelligence (AI) or machine learning (ML) based techniques can be used to detect abnormalities at any specific segment based on at least segment-specific endoscopic image or video features. Abnormality detection includes detecting and / or identifying one or more of the following: the presence or absence, type, size, shape, location, or quantity of pathological tissue or anatomical structures and other objects in the anatomical environment. In the colonoscopy example, detected abnormalities may include pathological tissue segments, such as mucosal abnormalities (polyps, inflammatory bowel disease, Merkel's diverticulum, lipomas, bleeding, vascularized mucosa, etc.) or obstructive mucosa (segments used for, for example, poor bowel preparation, colonic dilatation, etc.). In some examples, when an abnormality is detected, the substantially real-time position of the endoscope can also be used to estimate target or recommended segment-specific endoscopic withdrawal parameter values. Endoscopic positioning can be determined using electromagnetic tracking or by identifying anatomical landmarks based on features from images or video streams acquired or extracted during the insertion phase of an endoscopic examination. In examples of landmark-based endoscopic positioning, landmarks can be identified using AI or ML methods, such as a trained ML model that has been trained to establish a correspondence between endoscopic images or video streams (or features extracted from them) and landmark recognition, as described above. Figure 3 As described.

[0108] In the example, auxiliary inputs can also be used to estimate target or suggested segment-specific endoscopic withdrawal parameter values. As an example, and not a limitation, such auxiliary inputs include image or video streams and clinical data from previous endoscopic procedures performed on the patient, patient information and medical history, or preoperative imaging study data, user (e.g., endoscopist) profiles (including user experience, work environment, adaptability to new technologies, preferences for specific endoscopic procedures, etc.), endoscope and equipment information, or “AI discoveries” that include the state of an AI or ML algorithm used to detect abnormalities, landmarks, or other features or structural elements of interest in the target anatomy, as described above regarding… Figure 4 As described.

[0109] AI or ML methods can be used to estimate target or proposed segment-specific endoscopic withdrawal parameter values ​​(e.g., target WSL or WTL) for individual segments within different segments. In the example, the results of anomaly detection, landmark detection, and real-time endoscopic position information, along with auxiliary inputs, can be applied to a trained ML model. The ML model may have been trained to establish a correspondence between the composite inputs and the target or proposed segment-specific endoscopic withdrawal parameter (e.g., WSL or WTL) values, as described above. Figures 6A to 6B As described.

[0110] Target or recommended segment-specific endoscopic withdrawal parameter values ​​(e.g., segment-specific WSL or WTL) can be used to generate an endoscopic withdrawal diagram that graphically represents the endoscopic withdrawal plan, such as... Figure 5 The colonoscopy withdrawal speed diagram 510 is shown. The colonoscopy withdrawal speed diagram may include depictions of different segments of the anatomical structure, each segment being color-coded or grayscale-coded to indicate the corresponding target or suggested segment-specific endoscopic withdrawal parameter values. The endoscopic withdrawal diagram may be displayed on the user interface as a reference to guide the endoscopist during the endoscopic procedure. In some examples, information about abnormalities detected previously, such as during the insertion phase of the endoscopic procedure or from previous endoscopic procedures, may also be displayed on the endoscopic withdrawal diagram as an additional preventative layer to prevent excessive withdrawal speed in abnormal segments during endoscopic withdrawal.

[0111] At point 840, during the withdrawal phase, endoscopic withdrawal parameters (e.g., withdrawal speed or total withdrawal time) in specific segments of the anatomical structure can be tracked and measured. Various techniques can be used to track and measure endoscopic withdrawal speed. In one example, an optical flow-based method can be used to estimate the endoscopic withdrawal speed, where the rate of change of endoscopic image frames can be used. In another example, electromagnetic (EM) tracking techniques can be used to estimate the endoscopic withdrawal speed. In yet another example, the endoscopic withdrawal speed can be estimated relative to markers such as those identified from endoscopic images as described above. Using markers as reference or benchmarks, the endoscopic withdrawal speed can be estimated based on changes in the relative position of the identified markers. In some examples, during the endoscopic withdrawal phase, the position of the endoscope in a different segment of the anatomical structure can be detected substantially in real time, at least in part, based on endoscopic images or video streams or features extracted from them. The measured real-time endoscopic withdrawal parameter values ​​(e.g., withdrawal speed) can be correlated with the currently identified segment to produce segment-specific withdrawal parameter values.

[0112] The endoscope withdrawal plan generated at step 830 and the actual endoscope withdrawal parameter values ​​measured at step 840 can be provided to the user or process to facilitate manual or robotic endoscope withdrawal. At 852, the endoscope withdrawal plan, such as an endoscope withdrawal diagram, can be displayed to the user. Figure 5 As shown, a measured segment-specific withdrawal parameter value (e.g., withdrawal speed) can be compared to a target or recommended segment-specific withdrawal parameter value to determine if the measurement is within a specific margin of the target value. If the measured withdrawal speed falls within a specified margin of the speed limit for that segment, the withdrawal speed is considered appropriate. When the measured withdrawal speed deviates from the speed limit by more than a specified margin, an alert can be generated at 854 to remind the user (e.g., an endoscopist) of an inappropriately fast or slow withdrawal speed. Suggestions can be provided to the user to adjust the withdrawal speed to substantially conform to the speed limit for that segment. In some examples, substantially real-time endoscopic positioning within a segment of the anatomical structure can be used in conjunction with endoscopic withdrawal. Figure 1 Displayed (e.g., side-by-side or superimposed on the endoscope withdrawal image), such as Figure 5 As shown, this provides the user with direct visual feedback regarding endoscope withdrawal. Additionally or alternatively, in some examples, at 856, control signals can be generated to a robotic system that can robotically adjust the endoscope withdrawal speed according to a segmented withdrawal plan.

[0113] Figure 9A block diagram of an example machine 900 is shown in general, on which any or more of the techniques (e.g., methods) discussed herein can be performed. Parts of this description can be applied to the computational framework of various parts of the endoscope system 300.

[0114] In alternative implementations, machine 900 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 900 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 900 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 900 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, network device, network router, switch, or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by that machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0115] As described herein, examples may include logic or multiple components or mechanisms, or may be operated by logic or multiple components or mechanisms. A circuit set is a collection of circuits implemented in a tangible entity including hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership can be flexible with time and the variability of the underlying hardware. A circuit set includes members that can perform a specified operation individually or in combination during operation. In the examples, the hardware of the circuit set may be immutably designed to perform a specific operation (e.g., hardwired). In the examples, the hardware of the circuit set may include variable-connected physical components (e.g., execution units, transistors, simple circuits, etc.), which include computer-readable media that are physically modified (e.g., magnetically grounded, electrically grounded, movable placement of massless particles, etc.) to encode instructions for a specific operation. When connecting physical components, the basic electrical properties of the hardware composition change, for example, from an insulator to a conductor, and vice versa. Instructions enable embedded hardware (e.g., an execution unit or loading mechanism) to create members of the circuit set in the hardware via variable connections to perform a specific operation during operation. Therefore, when the device is operational, the computer-readable medium is communicatively connected to other components of the circuit set member. In the example, any one of the physical components can be used in more than one member of more than one circuit set. For example, under operation, an execution unit can be used at one point in time in a first circuit of a first circuit set and reused by a second circuit of the first circuit set or by a third circuit of a second circuit set at a different time.

[0116] Machine (e.g., computer system) 900 may include a hardware processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 904, and static memory 906, some or all of which may communicate with each other via interconnect (e.g., bus) 908. Machine 900 may also include a display unit 910 (e.g., a raster display, vector display, holographic display, etc.), an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) navigation device 914 (e.g., a mouse). In the example, display unit 910, input device 912, and UI navigation device 914 may be a touchscreen display. Machine 900 may additionally include a storage device (e.g., a drive unit) 916, a signal generation device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 900 may include output controller 928, which may be serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connected to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0117] Storage device 916 may include machine-readable medium 922 on which one or more sets of data structures or instructions 924 (e.g., software) are stored, said set of data structures or instructions 924 implementing or being utilized by any or more of the techniques or functions described herein. Instructions 924 may also reside wholly or at least partially within main memory 904, static memory 906, or hardware processor 902 during execution by machine 900. In this example, one or any combination of hardware processor 902, main memory 904, static memory 906, or storage device 916 may constitute the machine-readable medium.

[0118] Although machine-readable medium 922 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store one or more instructions 924.

[0119] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for use by machine 900 and causing machine 900 to perform any or more of the techniques of this disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory as well as optical and magnetic media. In examples, mass-capacity machine-readable media includes machine-readable media with a plurality of particles having invariant (e.g., rest) mass. Therefore, mass-capacity machine-readable media are not transiently propagating signals. Specific examples of mass-capacity machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices; magnetic disks, such as internal hard disks and removable hard disks; magneto-optical disks; and CD-ROM and DVD-ROM discs.

[0120] Commands 924 can also be sent or received via a communication network 926 using a transmission medium via network interface device 920, utilizing any of several transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., IEEE 802.11 series standards known as WiFi®, IEEE 802.16 series standards known as WiMax®), IEEE 802.15.4 series standards, peer-to-peer (P2P) networks, etc. In the example, network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to communication network 926. In the example, network interface device 920 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for execution by machine 900, and includes digital or analog communication signals or other intangible media to facilitate communication of such software.

[0121] Additional notes

[0122] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The drawings illustrate specific embodiments in which the invention may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. Furthermore, the inventors contemplate examples using any combination or arrangement of those elements (or one or more aspects thereof) shown or described relative to specific examples (or one or more aspects thereof) shown or described herein, or relative to other examples (or one or more aspects thereof).

[0123] In this document, the terms “a” or “an” are used as commonly found in patent literature to include one or more, regardless of any other instances or uses of “at least one” or “one or more.” In this document, unless otherwise indicated, the term “or” is used to mean a non-exclusive “or,” such that “A or B” includes “A but not B,” “B but not A,” and “A and B.” In this document, the terms “comprising” and “in…” are used as simple English equivalents to the corresponding terms “including” and “wherein.” Furthermore, in the appended claims, the terms “comprising” and “including” are open-ended, meaning that a system, apparatus, article, composition, formulation, or process that includes elements other than those listed after such terms in a claim is still considered to fall within the scope of that claim. Additionally, in the appended claims, the terms “first,” “second,” and “third,” etc., are used only as designations and are not intended to impose numerical requirements on their objects.

[0124] The above description is intended to be illustrative and not restrictive. For example, the examples above (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by those skilled in the art when consulting the above description. An abstract is provided to allow the reader to quickly determine the nature of the technical disclosure. The abstract is submitted on the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the detailed embodiments above, various features may be grouped together to simplify the disclosure. This should not be construed as meaning that any disclosed feature not claimed is essential to any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Therefore, the appended claims are incorporated herein by way of example or embodiment, wherein each claim exists independently as a separate embodiment, and it is contemplated that such embodiments may be combined with each other in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. An endoscope system, comprising: An endoscope, the endoscope including an imaging system configured to acquire images or video streams of different segments of a patient’s anatomical structures during an endoscopic examination procedure, the endoscopic examination procedure including insertion of the endoscope into the anatomical structures and subsequent withdrawal; as well as The controller circuit is configured to: Analyze the obtained image or video stream to generate endoscopic image or video features for each segment of the different segments of the anatomical structure; An endoscope withdrawal plan is generated, at least in part based on the features of the endoscopic images or videos, the endoscope withdrawal plan including target or suggested segment-specific endoscope withdrawal parameter values ​​for each segment of the different segments of the anatomical structure; as well as Provide the endoscope withdrawal plan to the user or robotic system to facilitate the manual or robotic withdrawal of the endoscope.

2. The endoscope system according to claim 1, wherein, The target or recommended segment-specific endoscopic withdrawal parameter values ​​include target segment-specific endoscopic withdrawal speed limits (WSLs) for each segment of the different segments of the anatomical structure.

3. The endoscope system according to any one of claims 1 to 2, wherein, The target or recommended segment-specific endoscopic withdrawal parameter values ​​include target segment-specific endoscopic withdrawal time limits (WTLs) for each segment of the different segments of the anatomical structure.

4. The endoscope system according to any one of claims 1 to 3, wherein, The endoscope mentioned is a colonoscope used during a colonoscopy. The imaging system is configured to acquire images or video streams from each of the different colonic segments during the colonoscopy procedure.

5. The endoscopic system according to any one of claims 1 to 4, wherein, Determining the target or recommended segment-specific endoscope withdrawal parameter value includes: for the first segment of the anatomical structure, determining the first target segment-specific endoscope withdrawal parameter using at least the endoscopic image or video features generated based on the image or video stream obtained during the manual or robotic withdrawal of the endoscope before the endoscope is withdrawn beyond the first segment.

6. The endoscopic system according to claim 5, wherein, Images or video streams acquired before the endoscope reaches the first segment include images or video streams acquired during the insertion of the endoscope into the anatomical structure.

7. The endoscope system according to any one of claims 1 to 6, wherein, The controller circuit is configured to: Perform anomaly detection, which includes detecting one or more anomalies in each of the different segments based at least in part on features of the endoscopic images or videos; as well as The target or recommended segment-specific endoscopic withdrawal parameter values ​​for each of the different segments are determined at least in part based on the results of the anomaly detection.

8. The endoscope system according to claim 7, wherein, The anomaly detection includes identifying one or more of the following: the presence or absence, type, size, shape, location, or quantity of pathological tissue or obstructive mucosa.

9. The endoscope system according to any one of claims 7 to 8, wherein, The controller circuitry is configured to use a first trained machine learning (ML) model to detect the one or more anomalies, the first ML model being trained to establish a correspondence between (i) features of an endoscopic image or video stream or features extracted therefrom, and (ii) one or more anomalous features.

10. The endoscope system according to any one of claims 7 to 9, wherein, The controller circuit is configured to determine the target or recommended segment-specific endoscopic withdrawal parameter values ​​for each of the different segments by applying the results of the anomaly detection to a second trained machine learning (ML) model, the second ML model being trained to establish a correspondence between (i) one or more anomaly features and (ii) the target or recommended segment-specific endoscopic withdrawal parameter values.

11. The endoscopic system according to claim 10, wherein, The controller circuit is configured to: Anomaly scores are determined based on the type, size, shape, location, or number of anomalies. as well as The determined outlier scores are applied to the second trained ML model to determine the target or recommended segment-specific endoscopic withdrawal parameter values ​​for each of the different segments.

12. The endoscopic system according to any one of claims 1 to 11, wherein, The controller circuitry is configured to also use one or more of the following to generate the endoscope withdrawal plan: Image or video streams and clinical data from previous endoscopic procedures; Patient information and medical history; or Preoperative imaging study data.

13. The endoscopic system according to any one of claims 1 to 12, wherein, The controller circuitry is configured to display a graphical representation of the endoscope withdrawal plan on a user interface. The graphical representation includes depictions of different segments of the anatomical structure, each segment being color-coded or grayscale-coded to indicate the corresponding target or suggested segment-specific endoscope withdrawal parameter value.

14. The endoscopic system according to claim 13, wherein, The controller circuit is configured to: The position of the endoscope in one of the different segments of the anatomical structure is determined substantially in real time, at least in part, based on the features of the endoscopic images or videos. The position of the endoscope is registered to a pre-generated template of the anatomical structure; as well as The position of the endoscope in one of the different segments is displayed on the user interface as an overlay on the graphical representation of the endoscope withdrawal plan.

15. The endoscope system according to claim 2, wherein, The controller circuit is configured to: Measuring the endoscopic withdrawal rate in one of the different segments of the anatomical structure; and When the measured endoscope withdrawal speed deviates from the target segment-specific endoscope WSL for one of the different segments by a certain margin, an alert is generated to the user, and suggestions are provided to adjust the withdrawal speed to substantially conform to the target segment-specific endoscope WSL.

16. The endoscope system according to any one of claims 2 or 15, comprising the robotic system configured to robotically withdraw the endoscope, and wherein, The controller circuit is configured to: Measure the endoscopic withdrawal speed in one of the different segments of the anatomical structure; as well as When the measured endoscope withdrawal speed deviates by a certain margin from the target segment-specific endoscope WSL for one of the different segments, a control signal is generated to the robotic system to automatically adjust the withdrawal speed to substantially conform to the target segment-specific endoscope WSL.

17. The endoscopic system according to any one of claims 2 and 15 to 16, wherein, The controller circuit is configured to: Identifying abnormal segments from the different segments, at least in part, based on the features of the endoscopic images or videos; and During the manual or robotic withdrawal of the endoscope and before the endoscope reaches the identified aberrant segment, a visual indicator of the aberrant segment is displayed on the user interface, and an alert is generated to the user, thereby withdrawing the endoscope at a rate lower than that of the target segment-specific endoscope WSL for the identified aberrant segment.

18. A method for planning the withdrawal of an endoscope from a patient's anatomical structures during an endoscopic examination, the method comprising: Using an imaging system associated with the endoscope, images or video streams of different segments of the anatomical structure are obtained during the insertion phase of the endoscopic examination procedure; Analyze the obtained image or video stream to generate endoscopic image or video features for each segment of the different segments of the anatomical structure; An endoscope withdrawal plan is generated, at least in part based on the features of the endoscopic images or videos, the endoscope withdrawal plan including target or suggested segment-specific endoscope withdrawal parameter values ​​for each segment of the different segments of the anatomical structure; as well as Provide the endoscope withdrawal plan to the user or robotic system to facilitate the manual or robotic withdrawal of the endoscope.

19. The method according to claim 18, wherein, The target or recommended segment-specific endoscopic withdrawal parameter values ​​include target segment-specific endoscopic withdrawal speed limits (WSL) or target segment-specific endoscopic withdrawal time limits (WTL) for each segment of the different segments of the anatomical structure.

20. The method according to any one of claims 18 to 19, comprising: At least using the endoscopic image or video features generated from the image or video stream obtained during the manual or robotic withdrawal of the endoscope before the endoscope is withdrawn beyond the first segment, a first target segment-specific endoscopic withdrawal parameter for the first segment of the anatomical structure is determined.

21. The method according to claim 20, wherein, Images or video streams acquired before the endoscope reaches the first segment include images or video streams acquired during the insertion of the endoscope into the anatomical structure.

22. The method according to any one of claims 18 to 21, comprising: Perform anomaly detection, which includes detecting one or more anomalies in each of the different segments based at least in part on features of the endoscopic images or videos; as well as The target or recommended segment-specific endoscopic withdrawal parameter values ​​for each of the different segments are determined at least in part based on the results of the anomaly detection.

23. The method according to claim 22, wherein, The anomaly detection includes using a first trained machine learning (ML) model, which is trained to establish a correspondence between (i) features extracted from or derived from an endoscopic image or video stream, and (ii) one or more anomalous features.

24. The method according to any one of claims 22 to 23, wherein, Determining the target or recommended segment-specific endoscopic withdrawal parameter value for each of the different segments includes applying the results of the anomaly detection to a second trained machine learning (ML) model, which is trained to establish a correspondence between (i) one or more anomalous features and (ii) the target or recommended segment-specific endoscopic withdrawal parameter value.

25. The method according to any one of claims 18 to 24, wherein, Generating the endoscope withdrawal plan also includes using one or more of the following: Image or video streams and clinical data from previous endoscopic procedures; Patient information and medical history; or Preoperative imaging study data.

26. The method of any one of claims 18 to 25, further comprising displaying a graphical representation of the endoscopic withdrawal plan on a user interface, the graphical representation including depictions of different segments of the anatomical structure, each segment being color-coded or grayscale-coded to indicate a corresponding target or suggested segment-specific endoscopic withdrawal parameter value.

27. The method of claim 26, further comprising: The position of the endoscope is determined essentially in real time; The position of the endoscope is registered to a pre-generated template of the anatomical structure; as well as The position of the endoscope in one of the different segments is displayed on the user interface as an overlay on the graphical representation of the endoscope withdrawal plan.

28. The method of claim 19, further comprising: Measure the endoscopic withdrawal speed in one of the different segments of the anatomical structure; as well as When the measured endoscope withdrawal speed deviates from the target segment-specific endoscope WSL for one of the different segments by a certain margin, an alert is generated to the user, and suggestions are provided to adjust the endoscope withdrawal speed to substantially conform to the target segment-specific endoscope WSL.

29. The method according to any one of claims 19 or 28, further comprising: Measure the endoscopic withdrawal speed in one of the different segments of the anatomical structure; as well as When the measured endoscope withdrawal speed deviates by a certain margin from the target segment-specific endoscope WSL for one of the different segments, a control signal is generated to the robotic system to automatically adjust the withdrawal speed to substantially conform to the target segment-specific endoscope WSL.

30. The method according to any one of claims 19 and 28 to 29, further comprising: Abnormal segments can be identified from the different segments, at least in part, based on the features of the endoscopic images or videos. as well as During the manual or robotic withdrawal of the endoscope and before the endoscope reaches the identified aberrant segment, a visual indicator of the aberrant segment is displayed on the user interface, and an alert is generated to the user, thereby withdrawing the endoscope at a rate lower than that of the target segment-specific endoscope WSL for the identified aberrant segment.

Images