Image-guided endoscope withdrawal control

Abstract

Systems, devices, and methods are disclosed for creating and using segment-specific endoscopic withdrawal plans to guide endoscopic procedures. The endoscopic system includes an endoscope and a control circuit. The endoscope is equipped with an imaging system to obtain image or video streams of segment-specific biostructures in a patient during an endoscopic procedure. The control circuit can analyze the image or video streams and generate image or video features for each of the segment-specific biostructures. Based on the image or video features, the control circuit can generate an endoscopic withdrawal plan for each segment, including target or recommended segment-specific endoscopic withdrawal parameter values. The endoscopic withdrawal plan may be provided to the user or a robotic system to facilitate manual or robotic withdrawal of the endoscope.

Description

[Technical Field]

[0001] Claim of priority This application claims priority to U.S. Provisional Patent Application No. 63 / 582,026, filed on 12 September 2023, the contents of which are incorporated herein by reference.

[0002] This document generally relates to endoscopic medical systems, and more specifically to systems and methods for image-guided withdrawal of endoscopes during endoscopic procedures. [Background technology]

[0003] Endoscopes are used in a variety of clinical procedures, including, for example, illuminating, imaging, detecting, and diagnosing one or more pathological conditions, providing fluid delivery (e.g., saline or other preparations via fluid channels) to anatomical regions, providing passage (e.g., via working channels) for one or more therapeutic devices or biomaterial collection devices for sampling or treatment of anatomical regions, and providing suction channels for collecting fluids (e.g., saline or other preparations). Examples of such anatomical regions may include the gastrointestinal tract (e.g., esophagus, stomach, duodenum, pancreaticobiliary duct, intestine, colon, etc.), the renal region (e.g., kidneys, ureters, bladder, urethra), and other internal organs (e.g., reproductive system, sinuses, submucosa, respiratory system).

[0004] Some endoscopes feature a working passage that allows the operator to perform minimally invasive surgical procedures such as suction, placement of diagnostic or therapeutic devices (e.g., brushes, biopsy needles or forceps, stents, baskets, or balloons), or tissue sampling or removal of undesirable tissue (e.g., benign or malignant stenosis) or foreign bodies (e.g., gallstones). Some endoscopes may be used in conjunction with laser or plasma systems to deliver energy to an anatomical target (e.g., soft or hard tissue or gallstones) to achieve the desired procedure. For example, lasers are used in applications of tissue resection, coagulation, evaporation, fragmentation, and lithotripsy to break up gallstones in the kidneys, gallbladder, and ureters, among many other areas of gallstone formation, or to cut large gallstones into smaller fragments.

[0005] Endoscopic procedures involve an insertion phase, in which the endoscope is guided through the natural opening into the body cavity until it reaches the target site, and a subsequent withdrawal phase, in which the endoscope is carefully withdrawn from the body. Diagnostic or therapeutic actions (e.g., biopsy or removal of specific abnormal tissue, pathological tissue, or foreign body) are performed in the target biostructure or during the withdrawal. For example, a colonoscopy involves guiding a colonoscope through the anus all the way to the beginning of the colon, called the cecum. During the withdrawal, colonic segments can be examined, and polyps of a certain size detected during insertion can be removed (polypectomy). Colonoscopy for the diagnosis and treatment of polyps is used to reduce the incidence and mortality of colorectal cancer. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The time spent withdrawing the colonoscope during a colonoscopy is an important quality measure, especially for negative procedures. To ensure sufficient examination and procedure time, a minimum withdrawal time of over 6 minutes has been reported as a quality assurance standard for colonoscopies, with a withdrawal time of over 10 minutes being the aspiration standard. Longer withdrawal times can increase the adenoma detection rate (ADR).

[0007] To ensure sufficient examination time and to prevent the colonoscope from being withdrawn too quickly, speedometers or similar devices have been proposed to display the withdrawal speed. However, the withdrawal speed limits or withdrawal time thresholds used to evaluate withdrawal in endoscopic procedures are generally predetermined, such as based on generally accepted criteria (e.g., at least 6 minutes or at least 10 minutes). These "general" withdrawal criteria are highly generalized standards determined without considering the specific circumstances raised by individual procedures. For example, a target withdrawal time greater than 6 minutes is primarily a quality indicator for unsatisfactory outcomes in average-risk colonoscopies with a normal colon. This includes patients without prior surgical resection and without biopsy or polyp removal. In the case of colons in patients with colon cancer or patients under surveillance (e.g., due to polyps previously identified and / or restricted from past colonoscopy procedures for a particular patient), such general withdrawal criteria may be less effective or suboptimal. Furthermore, the typical withdrawal time may be ineffective if bowel preparation in one or more colonic segments is insufficient. In such cases, the endoscopist may need to diligently perform a slow withdrawal to ensure a complete mucosal examination, while also ensuring that polyps are hidden and the bowel is slowly flushed along the way.

[0008] In addition to patient-to-patient variations in withdrawal time requirements, such as the patient's medical condition and / or the quality of bowel preparation as previously mentioned, general withdrawal time standards (e.g., a minimum of 6 or 10 minutes) focus on a comprehensive withdrawal time applicable to the entire colonoscopy withdrawal phase. This lacks specificity for any particular colonic segment. Desired withdrawal times may vary from segment to segment, as different colonic segments generally have different bowel preparation qualities and / or different opportunities to exhibit abnormal structures (e.g., polyps or cancer). General, comprehensive withdrawal times offer little guidance on how quickly or how slowly the colonoscope should be withdrawn in different colonic segments. [Means for solving the problem]

[0009] The inventors of this disclosure have identified an unmet need to improve the quality control of endoscope withdrawal during colonoscopy procedures. This document describes a system and method for image-based segment-specific endoscope withdrawal in endoscopic procedures. An exemplary system can receive an image or video stream of the endoscope obtained during insertion or at any time before the endoscope is withdrawn beyond a target segment, analyze the image or video stream, and generate an individual segment-specific endoscope withdrawal plan. The segment-specific endoscope withdrawal plan may include target endoscope withdrawal rate limits or withdrawal time limits for each of several individual segments of the biological structure. The individual segment-specific endoscope withdrawal plan may be presented to the user in graphical form, such as a withdrawal rate map or withdrawal time map. During withdrawal, the individual segment-specific endoscope withdrawal plan can be used as a guide for mucosal examination and necessary procedures (e.g., polypectomy). The actual withdrawal rate may be tracked and measured. Substantially real-time feedback on the speed or time spent withdrawing the endoscope in each segment of the biological structure (e.g., the colon segment), along with recommendations for adjusting the endoscope withdrawal speed, may be provided to the endoscopist. Real-time monitoring of endoscope withdrawal with reference to segment-specific endoscope withdrawal plans can improve the quality of mucosal examination, screening accuracy, and efficiency. In some cases, segment-specific endoscope withdrawal plans may be provided to robotic colonoscopy systems to facilitate robot-assisted endoscopy and to ensure that the withdrawal speed remains within recommended limits.

[0010] Example 1 is an endoscope system comprising an endoscope provided with an imaging system, the imaging system being configured to obtain images or video streams of individual sections of a patient's anatomy during an endoscopic procedure including insertion and subsequent withdrawal of the endoscope in the patient's anatomy; and a control circuit, the control circuit being configured to analyze the obtained images or video streams to generate endoscopic image features or endoscopic video features for each individual one of the individual sections of the anatomy, generate an endoscopic withdrawal plan including targeted or recommended section-specific endoscopic withdrawal parameter values for each individual one of the individual sections of the anatomy based at least in part on the endoscopic image features or the endoscopic video features, and provide the endoscopic withdrawal plan to a user or a robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0011] In Example 2, the subject matter of Example 1 optionally includes the targeted or recommended section-specific endoscopic withdrawal parameter values, which may include a targeted section-specific endoscopic withdrawal speed limit (WSL) for each individual one of the individual sections of the anatomy.

[0012] In Example 3, the subject matter of any one or more of Examples 1 to 2 optionally includes the targeted or recommended section-specific endoscopic withdrawal parameter values, which may include a targeted section-specific endoscopic withdrawal time limit (WTL) for each individual one of the individual sections of the anatomy.

[0013] In Example 4, the subject matter of any one or more of Examples 1 to 3 optionally includes a colonoscope for use in a colonoscopy procedure, and the imaging system is configured to obtain an image or video stream from each of the individual colon sections during the colonoscopy procedure.

[0014] In Example 5, the subject matter of any one or more of Examples 1 to 4 optionally includes that determining the target or recommended section-specific endoscope withdrawal parameter value comprises determining a first target section-specific endoscope withdrawal parameter for a first section of a anatomical structure, at least using endoscope image features or endoscope video features generated from an image or video stream obtained before the endoscope is withdrawn beyond the first section during manual or robotic withdrawal of the endoscope.

[0015] In Example 6, the subject matter of Example 5 optionally includes the image or video stream obtained before the endoscope reaches the first section, which may comprise an image or video stream obtained during insertion of the endoscope in the anatomical structure.

[0016] In Example 7, the subject matter of any one or more of Examples 1 to 6 optionally includes a control circuit that may be configured to perform anomaly detection comprising detecting one or more anomalies in each individual section based at least in part on endoscope image features or endoscope video features, and determine the target or recommended section-specific endoscope withdrawal parameter value for each individual section based at least in part on a result of the anomaly detection.

[0017] In Example 8, the subject matter of Example 7 optionally includes anomaly detection that may comprise recognizing one or more of the presence, type, size, shape, location, or quantity of pathological tissue or obstructed mucosa.

[0018] In Example 9, the subject matter of any one or more of Examples 7 to 8 optionally includes a control circuit that may be configured to detect one or more anomalies using a first trained machine learning (ML) model, and the first ML model is trained to establish a correspondence between (i) an image or video stream of the endoscope, or features extracted therefrom, and (ii) one or more anomaly characteristics.

[0019] In Example 10, one or more subjects from Examples 7-9 optionally include a control circuit which may be configured to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for each individual segment by applying the results of anomaly detection to a second trained machine learning (ML) model, the second ML model being trained to establish a correspondence between (i) one or more anomaly characteristics and (ii) target or recommended segment-specific endoscopic withdrawal parameter values.

[0020] Example 11 optionally includes a control circuit that can be configured to determine an anomaly score based on the type, size, shape, location, or amount of the anomaly in Example 10, and to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for each individual segment by applying the determined anomaly score to a second trained ML model.

[0021] Example 12 optionally includes a control circuit that can be configured to generate an endoscopic withdrawal plan using one or more subjects from any one of Examples 1 to 11, further using one or more of the following: images or video streams and clinical data from a previous endoscopic procedure, patient information and medical history, or pre-procedure imaging study data.

[0022] In Example 13, one or more themes from Examples 1 to 12 optionally include a control circuit that can be configured to display a graphical representation of an endoscopic withdrawal plan in the user interface, the graphical representation including a depiction of individual segments of biological structures, each coded in color or grayscale to indicate endoscopic withdrawal parameter values ​​specific to each target or recommended segment.

[0023] Example 14 optionally includes a control circuit that can be configured to: determine the location of the endoscope in one of the individual segments of a biological structure in substantially real time, at least in part, based on endoscopic image features or endoscopic video features; register the location of the endoscope in a pre-generated template of the biological structure; and display the location of the endoscope in one of the individual segments in a user interface so as to be superimposed on a graphical display of the endoscope withdrawal plan.

[0024] Example 15 optionally includes a control circuit that can be configured to measure the endoscope withdrawal speed in one of the individual segments of a biological structure, and to generate an alarm to the user and provide a recommendation to adjust the withdrawal speed to substantially match the target segment-specific endoscope WSL when one or more subjects from Examples 2 to 14 deviates by a specific difference from the target segment-specific endoscope WSL for one of the individual segments.

[0025] In Example 16, one or more subjects from Examples 2 to 15 optionally include a robotic system which may be configured to robotically withdraw an endoscope, and the control circuit is configured to measure the endoscope withdrawal speed in one of the individual segments of the biological structure, and to generate a control signal to the robotic system to automatically adjust the withdrawal speed to substantially match the target segment-specific endoscope WSL when the measured endoscope withdrawal speed deviates by a specific difference from the target segment-specific endoscope WSL for one of the individual segments.

[0026] Example 17 optionally includes a control circuit that can be configured to: identify anomaly segments from individual segments based at least in part on endoscopic image features or endoscopic video features; and, during manual or robotic withdrawal of the endoscope, display a visual indication of the anomaly segment on the user interface before the endoscope reaches the identified anomaly segment, and generate a warning to the user to withdraw the endoscope at a rate lower than the target segment-specific endoscopic WSL for the identified anomaly segment.

[0027] Example 18 is a method for planning the withdrawal of an endoscope from a patient's biostructure during an endoscopic procedure, the method comprising: obtaining an image or video stream of individual segments of the biostructure during the insertion phase of the endoscopic procedure using an imaging system associated with the endoscope; analyzing the obtained images or video stream to generate endoscopic image features or endoscopic video features for each individual segment of the biostructure; generating an endoscopic withdrawal plan for each individual segment of the biostructure, including target or recommended segment-specific endoscopic withdrawal parameter values, at least in part based on the endoscopic image features or endoscopic video features; and providing the endoscopic withdrawal plan to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0028] In Example 19, the subject of Example 18 optionally includes target or recommended segment-specific endoscopic withdrawal parameter values, which may include target segment-specific endoscopic withdrawal speed limits (WSL) or target segment-specific endoscopic withdrawal time limits (WTL) for each individual segment of the biological structure.

[0029] Example 20 optionally includes a step of determining endoscopic withdrawal parameters specific to a first target segment for a first segment of a biological structure, using at least endoscopic image features or endoscopic video features generated from images or video streams obtained before the endoscope is withdrawn beyond a first segment during manual or robotic withdrawal of the endoscope.

[0030] In Example 21, the subject of Example 20 optionally includes images or video streams obtained before the endoscope reaches the first section, which may include images or video streams obtained during the insertion of the endoscope into a biological structure.

[0031] Example 22 optionally includes the steps of: performing anomaly detection on one or more subjects from any of Examples 18 to 21, which at least partially depend on endoscopic image features or endoscopic video features, including detecting one or more anomalies in each of the individual segments; and determining target or recommended segment-specific endoscopic withdrawal parameter values ​​for each of the individual segments, at least partially depending on the results of the anomaly detection.

[0032] In Example 23, the subject of Example 22 optionally includes anomaly detection using a first pre-trained machine learning (ML) model, which is trained to establish correspondences between (i) endoscopic images or video streams, or features extracted therefrom, and (ii) one or more anomaly characteristics.

[0033] Example 24 optionally includes a step of determining target or recommended segment-specific endoscopic withdrawal parameter values ​​for each individual segment, which may include applying the results of anomaly detection to a second trained machine learning (ML) model, the second ML model being trained to establish a correspondence between (i) one or more anomaly characteristics and (ii) target or recommended segment-specific endoscopic withdrawal parameter values.

[0034] Example 25 optionally includes a step of generating an endoscopic withdrawal plan, which may further include using one or more of the subjects from any one of Examples 18-24, including images or video streams and clinical data from previous endoscopic procedures, patient information and medical history, or pre-procedure imaging study data.

[0035] In Example 26, one or more themes from Examples 18–25 optionally include a step of displaying a graphical representation of an endoscopic withdrawal plan in the user interface, the graphical representation including a depiction of individual segments of biological structures, each coded in color or grayscale to indicate endoscopic withdrawal parameter values ​​specific to each target or recommended segment.

[0036] In Example 27, the subject of Example 26 optionally includes the steps of determining the location of the endoscope in virtually real time, registering the location of the endoscope in a pre-generated template of the biological structure, and displaying the location of the endoscope in one of the individual sections on the user interface so as to be superimposed on a graph of the endoscope withdrawal plan.

[0037] Example 28 optionally includes the steps of: measuring the endoscope withdrawal speed in one of the individual segments of a biological structure for one or more subjects from Examples 19 to 27; and generating an alarm to the user and providing a recommendation to adjust the endoscope withdrawal speed to substantially match the target segment-specific endoscopic WSL when the measured endoscope withdrawal speed deviates by a specific difference for one of the individual segments, and adjusting the endoscope withdrawal speed to substantially match the target segment-specific endoscopic WSL.

[0038] Example 29 optionally includes the steps of: measuring the endoscopic withdrawal speed in one of the individual segments of a biological structure for one or more subjects from Examples 19 to 28; and generating a control signal to a robotic system to automatically adjust the withdrawal speed to substantially match the target segment-specific endoscopic WSL when the measured endoscopic withdrawal speed deviates by a specific difference from the target segment-specific endoscopic WSL for one of the individual segments.

[0039] Example 30 optionally includes, for one or more subjects from Examples 19-29, the steps of identifying anomaly segments from individual segments based at least in part on endoscopic image features or endoscopic video features, and displaying a visual indication of the anomaly segment on the user interface during manual or robotic withdrawal of the endoscope and before the endoscope reaches the identified anomaly segment, and generating a warning to the user to withdraw the endoscope at a rate lower than the target segment-specific endoscopic WSL for the identified anomaly segment.

[0040] The techniques presented are described in terms of controlled withdrawal of an endoscope in colonoscopy, but are not limited thereto. Systems, devices, and techniques, such as those described according to the various embodiments described herein, may be used in addition to or as substitutes in other procedures involving different types of endoscopes, including, for example, anoscopy, arthroscopy, bronchoscopy, colonoscopy, vaginal and cervical examination, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, rectoscopy, sigmoidoscopy, and thoracoscopy.

[0041] This summary outlines some of the teachings of this application and is not intended to be an exclusive or comprehensive treatment of the subject matter. Further details of the subject matter can be found in the detailed description and the attached claims. Other aspects of this disclosure will be apparent to those skilled in the art by reading and understanding the following detailed description and by viewing the drawings that form part of the detailed description, and these descriptions and drawings are not to be understood in a limiting sense. The scope of this disclosure is defined by the attached claims and their legal equivalents. [Brief explanation of the drawing]

[0042] [Figure 1] This is a schematic diagram illustrating an example of an endoscopic system for use in endoscopic procedures, such as colonoscopy. [Figure 2] This is a schematic diagram illustrating an example of an endoscopic system for use in endoscopic procedures, such as colonoscopy. [Figure 3] This is a diagram illustrating an example of an endoscopic system configured to create individual, segment-specific endoscopic withdrawal plans and to guide endoscopic withdrawal during endoscopic procedures using those plans. [Figure 4] This figure shows examples of auxiliary inputs that can be used in estimating withdrawal parameters and creating endoscopic withdrawal plans specific to individual sections. [Figure 5] This figure illustrates exemplary different stages of an image-guided colonoscopy according to the embodiments described herein. [Figure 6A] This figure illustrates an example of training a machine learning (ML) model to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for any particular segment of a biological structure, and then using the trained ML model. [Figure 6B] This figure illustrates an example of training a machine learning (ML) model to determine target or recommended segment-specific endoscopic withdrawal parameter values ​​for any particular segment of a biological structure, and using the trained ML model. [Figure 7A]This graph shows an example of guided withdrawal and speed tracking in colonoscopy procedures using a segment-specific endoscopic withdrawal plan. [Figure 7B] This graph shows an example of guided withdrawal and speed tracking in colonoscopy procedures using a segment-specific endoscopic withdrawal plan. [Figure 8] This flowchart illustrates an exemplary method for creating a specific endoscope withdrawal plan for individual procedures and using the plan to guide the endoscope withdrawal during endoscopic treatment. [Figure 9] This is a block diagram illustrating an exemplary machine that can be implemented by any one or more of the techniques (e.g., methodologies) discussed herein. [Modes for carrying out the invention]

[0043] This document describes systems, devices, and methods for creating and using segment-specific endoscopic withdrawal plans to guide withdrawal during endoscopic procedures. According to one embodiment, the endoscopic system includes an endoscope and a control circuit. The endoscope is equipped with an imaging system to obtain image or video streams of segment-specific biological tissues in a patient during an endoscopic procedure. The control circuit can analyze the obtained image or video streams and generate image or video features for each individual segment of the biological tissue. Based on the image or video features, the control circuit can generate segment-specific endoscopic withdrawal plans for each individual segment, including target or recommended values ​​for endoscopic withdrawal parameters (such as withdrawal speed or withdrawal time). Individual endoscopic withdrawal plans may be provided to a user or robotic system to facilitate manual or robotic withdrawal of the endoscope.

[0044] Figure 1 is a schematic diagram of an endoscopic system 10 for use in endoscopic procedures such as colonoscopy. System 10 may comprise an imaging control system 12 and an endoscope 14. System 10 is an illustrated example of an endoscopic system suitable for use with the systems, devices, and methods described herein, such as a colonoscopy system for use in image-guided colonoscopy following an individualized segmental endoscopic withdrawal plan as described herein.

[0045] The endoscope 14 may be insertable into an anatomical region to provide access to, or attachment (e.g., via a tether) to, one or more sampling devices for imaging or biopsy, or a therapeutic device for treating a medical condition associated with an anatomical region. The endoscope 14 may interface with or connect to an imaging control system 12. The endoscope 14 may also include a colonoscope, but other types of endoscopes may be used in the features and teachings of this disclosure. The imaging control system 12 may comprise a control unit 16, an output unit 18, an input unit 20, a light source unit 22, a fluid supply source 24, and a suction pump 26.

[0046] The imaging control system 12 may have various ports for connection with the endoscope system 10. For example, the control unit 16 may have data input / output ports for receiving data from and communicating data to the endoscope 14. The light source unit 22 may have output ports for sending light to the endoscope 14, such as via an optical fiber link. The fluid supply source 24 may have ports for supplying fluid to the endoscope 14. The fluid supply source 24 may, for example, have a pump and a fluid tank, or it may be connected to an external tank, container, or storage unit. The suction pump 26 may have ports for generating suction by drawing a vacuum from the endoscope 14, such as to draw fluid from the anatomical region into which the endoscope 14 is inserted. The output unit 18 and input unit 20 may be used by the operator of the endoscope system 10 to control the functions of the endoscope system 10 and to view the output of the endoscope 14. The control unit 16 may also generate signals or other outputs from treating the anatomical region into which the endoscope 14 is inserted. In some examples, the control unit 16 can generate electrical, acoustic, or fluid outputs to treat anatomical areas, for example, by cauterizing, cutting, or freezing.

[0047] The fluid supply source 24 can communicate with the control unit 16 and may comprise one or more sources of air, saline solution, or other fluids, as well as associated fluid pathways (e.g., air passages, irrigation passages, suction passages, etc.) and connectors (barb fittings, fluid seals, valves, etc.). The fluid supply source 24 can be used as operating energy for the biasing device or pressure-applying device of this disclosure. The imaging control system 12 may also comprise a drive unit 46 which may include a motorized drive unit for advancing the distal section of the endoscope 14.

[0048] The endoscope 14 may comprise an insertion section 28, a functional section 30, and a handle section 32 that can be connected to a cable section 34 and a connector section 36. The insertion section 28 may extend distally from the handle section 32, and the cable section 34 may extend proximal to the handle section 32. The insertion section 28 may be elongated and comprise a bending section and a distal end to which the functional section 30 may be attached. The bending section may be controllable (e.g., by a control knob 38 in the handle section 32) to maneuver the distal end through a winding anatomical passage (e.g., stomach, duodenum, kidney, ureter, etc.). The insertion section 28 may also be elongated and include one or more working passages (e.g., internal lumens) that can assist in the insertion of one or more therapeutic instruments into the functional section 30. The working passages may extend between the handle section 32 and the functional section 30. Additional functionality, such as fluid passage, guide wires, and tension wires, may be provided by the insertion section 28 (e.g., via suction passages or irrigation passages).

[0049] The coupling area 36 may be connected to the control unit 16 to connect the endoscope 14 to several feature parts of the control unit 16, such as the input unit 20, the light source unit 22, the fluid supply source 24, and the suction pump 26.

[0050] The handle area 32 may include a knob 38 and a port 40A. The knob 38 may be connected to a tension wire or other operating mechanism that can extend through the insertion area 28. Port 40A and other ports, such as port 40B (Figure 2), may be configured to connect various electrical cables, guide wires, auxiliary scopes, tissue collection devices, and fluid tubes to the handle area 32, such as for connecting to the insertion area 28.

[0051] For example, the imaging control system 12 may be mounted on a movable platform (e.g., a cart 41) with shelves for housing the light source unit 22, suction pump 26, image processing unit 42 (Figure 2), etc. Alternatively, some components of the imaging control system 12 (shown in Figures 1 and 2) may be mounted directly on the endoscope 14 to make the endoscope "integrated".

[0052] The functional area 30 may comprise components for treating and diagnosing the patient's biological structures. The functional area 30 may comprise an imaging device, a lighting device, and an elevator. The functional area 30 may further comprise optically enhanced biomaterial and tissue collection and retrieval devices. For example, the functional area 30 may comprise one or more electrodes conductively connected to the handle area 32 and functionally connected to the imaging control system 12, allowing for the analysis of biomaterial in contact with the electrodes based on comparative biodata stored in the imaging control system 12. In other examples, the functional area 30 may directly incorporate a tissue collector.

[0053] In some examples, the endoscope 14 may be robotically controlled, such as by a robotic arm attached to it. The robotic arm can, automatically or semi-automatically (e.g., with minimal manual control or command from the user), position and guide the endoscope 14 (e.g., the functional area 30 and / or insertion area 28) in a target biological structure via actuators, or position the device in a desired posture and location to facilitate surgery on an anatomical target. According to the various examples discussed herein, the control device may generate control signals to the actuators of the robotic arm to facilitate the operation of such instrument or tool in robot-assisted endoscopic procedures, according to individual section-specific endoscope withdrawal plans.

[0054] Figure 2 is a schematic diagram of the endoscopic system 10 of Figure 1, including the imaging control system 12 and the endoscope 14. Figure 2 schematically shows the components of the imaging control system 12 connected to the endoscope 14, which in the illustrated example includes a colonoscope. The imaging control system 12 comprises a control unit 16, which may comprise, or be connected to, an image processing unit 42, a treatment generation device 44, and a drive unit 46, as well as a light source unit 22, an input unit 20, and an output unit 18. The control unit 16 may comprise, or be able to communicate with, an endoscope, surgical instruments 48, and an endoscopic system, which may comprise devices configured to engage with tissue and collect and store portions of that tissue, and through these endoscopes, surgical instruments, and endoscopic systems, an imaging device (e.g., a camera) may be able to view target tissue by comprising optically enhanced materials and components. The control unit 16 may be configured to activate the camera to view target tissue distal to the endoscopic system. Similarly, the control unit 16 can be configured to activate the light source unit 22 to illuminate the surgical instrument 48, and the surgical instrument 48 may include selected components configured to reflect light in a particular manner, such as a tissue cutter reinforced with reflective particles.

[0055] The coupling area 36 may be connected to the control unit 16 to connect the endoscope 14 to several feature units of the control unit 16, such as the image processing unit 42 and the treatment generation device 44. In the example, port 40A may be used to insert another surgical instrument 48 or device, such as a dotascope or auxiliary scope, into the endoscope 14. Such instruments and devices can be independently connected to the control unit 16 via cables 47. In the example, port 40B may be used to connect the coupling area 36 to various inputs and outputs, such as video, air, light, and electricity.

[0056] The image processing unit 42 and the light source unit 22 can interface with the endoscope 14 (e.g., in the functional area 30) by wired or wireless electrical connections, respectively. Thus, the imaging control system 12 can illuminate anatomical regions, collect signals representing anatomical regions, process signals representing anatomical regions, and display images representing anatomical regions on the display unit 18. The imaging control system 12 may include a light source unit 22 for illuminating anatomical regions using light of a desired spectrum (e.g., broadband white light, narrowband imaging using preferred electromagnetic wavelengths, etc.). The imaging control system 12 can be connected to the endoscope 14 (e.g., via an endoscope connector) for signal transmission (e.g., light output from the light source, video signals from the imaging system at the distal end, and diagnostic and sensor signals from diagnostic devices, etc.).

[0057] The procedure generation device 44 can generate a procedure plan, which can be used by the control unit 16 to control the movement of the endoscope 14 during the endoscopic procedure or to provide guidance to the surgeon for maneuvering the endoscope 14. For example, the procedure generation device 44 can use patient information, including images of the target biostructure, to generate an endoscopic guidance plan, which may include estimates for one or more intubations or guidance parameters (e.g., angle, force, etc.) for maneuvering a movable elongated instrument. The endoscopic guidance plan can help guide the surgeon for cannulation and guidance of the endoscope within the patient's biostructure. The endoscopic guidance plan may, in addition or alternatively, be used to robotically adjust the position, angle, force, and / or guidance of the endoscope or other instruments.

[0058] Figure 3 is a block diagram showing an example of an endoscope system 300 that can generate individual section-specific endoscope withdrawal plans, and that these plans can be used as guidance during endoscope withdrawal. In this example, the endoscope system 300 can be used in a colonoscopy procedure to provide guided mucosal examination and necessary procedures (e.g., polypectomy) during withdrawal. The system 300 may be implemented as part of the control unit 16 in Figure 1.

[0059] System 300 may include one or more of the following: an endoscope 310, an auxiliary input unit 315, a control circuit 320, a user interface 330, and a storage device 340. In some examples, System 300 may further include a robotic system 350 in robot-assisted endoscopic procedures, or may be communicatively connected to a robotic system 350.

[0060] The endoscope 310 may be an example of the endoscope 14 described above and shown in Figures 1 and 2. The endoscope 310 may comprise, among others, an imaging system 312 and an illumination system 314. The imaging system 312 may comprise at least one imaging sensor or imaging device (e.g., a camera) configured to obtain an endoscopic image or video stream of a target biological structure of the patient during an endoscopic procedure. The imaging sensor or imaging device may be located in the distal portion or distal end of the endoscope 310. The illumination system 314 may comprise one or more light sources to provide illumination to the target biological structure through one or more light lenses. In some examples, the imaging system 312 may be controllably tuned to operate in one of several individual imaging modes. These imaging modes may differ in one or more of the following: zoom setting, contrast setting, exposure level, or field of view toward the target biological structure. In some examples, the illumination system 314 may be controllably tuned to provide different light or illumination conditions.

[0061] Endoscopic procedures generally include an insertion phase in which the endoscope is guided into a body cavity through a natural orifice, and a subsequent withdrawal phase in which the endoscope is withdrawn from the body through the body cavity and the natural orifice. The imaging system 312 can obtain images or video streams during the insertion phase (hereinafter referred to as "insertion images or insertion video streams") and images or video streams during the withdrawal phase (hereinafter referred to as "withdrawal images or withdrawal video streams"). Insertion images or insertion video streams and withdrawal images or withdrawal video streams can be used for different purposes. When the target biological structure includes multiple anatomically distinct or artificially defined divisions or parts, the imaging system 312 can obtain images or video streams from each of those individual divisions of the biological structure during the insertion phase (hereinafter referred to as division-specific insertion images or insertion video streams) and images or video streams during the withdrawal phase (hereinafter referred to as division-specific withdrawal images or withdrawal video streams). As will be further discussed later, during colonoscopy, segment-specific insertion images or video streams can be used to detect abnormalities in any particular segment, and segment-specific endoscopic withdrawal plans can be created that set withdrawal rate limits or withdrawal time limits in each segment, at least in part, based on the detected abnormalities. Segment-specific withdrawal images or video streams can be analyzed to pinpoint the endoscope's position in substantially real time and to track the withdrawal rate in real time. When segment-specific withdrawal plans are used as a reference, timely feedback on endoscopic withdrawal and recommendations for adjusting the withdrawal can be provided to the endoscopist.

[0062] The control circuit 320 may include a set of circuits comprising one or more other circuits or subcircuits capable of performing the functions, methods, or techniques described herein, either individually or in combination. In an example, the control circuit 320 and the set of circuits in the control circuit 320 may be implemented as part of a microprocessor circuit, which may be a dedicated processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a microprocessor, or another type of processor for processing information, including physical activity information. Alternatively, the microprocessor circuit may be a general-purpose processor capable of receiving and executing a set of instructions for performing the functions, methods, or techniques described herein. In an example, the hardware of the set of circuits may be designed to be immutable for performing a particular operation (e.g., it may be connected by wiring). In an example, the hardware of the set of circuits may include variable-connected physical components (e.g., execution units, transistors, simple circuits, etc.) that include a computer-readable medium that is physically modified (e.g., a magnetically, electrically, or movably located location of an invariant mass particle) for coding instructions for a particular operation. When connected to a physical component, the fundamental electrical properties of the hardware component are changed, for example, from an insulator to a conductor, or vice versa. Instructions, while in operation, can cause embedded hardware (e.g., an execution unit or loading mechanism) to create a component unit of a circuit set in the hardware via a variable connection to execute a specific part of the operation. Thus, a computer-readable medium is communicatively linked to other components of the circuit set component unit while the device is in operation. In the example, any of the physical components can be used in two or more components of two or more circuit sets. For example, during operation, an execution unit may, at some point, be used in a first circuit of a first circuit set, and then again in a second circuit of the first circuit set, or at a different time, be used again in a third circuit of a second circuit set.

[0063] The control circuit 320 may comprise one or more of the following: an image processor 321, an anomaly detection circuit 322, an endoscope positioning circuit 323, an endoscope withdrawal plan generation device 324, a real-time endoscope withdrawal tracking circuit 327, and an endoscope withdrawal control device 328. The image processor 321 can analyze segment-specific images or video streams obtained from the imaging system 312 and generate endoscopic image features or endoscopic video features for individual segmental segments of the biological structure. Examples of image or video features include statistical features of pixel values, or morphological features such as angles, edges, clumps, curvature, Speeded Up Robust Feature (SURF), or Scale Invariant Feature Transform (SIFT). In some cases, the image processor 321 may pre-process segment-specific images or video streams, such as filtering, resizing, orientation, or color or grayscale correction, and the endoscopic image or video features may be extracted from the pre-processed images or video streams. In some examples, the image processor 321 may post-process image or video features to improve feature quality, such as by edge interpolation or edge extrapolation, in order to create continuous and smooth edges.

[0064] The anomaly detection circuit 322 can detect anomalies in any specific section of a biological structure, at least in part, based on section-specific endoscopic image or video features. Anomaly detection includes detecting and / or recognizing one or more of the presence, type, size, shape, location, or quantity of pathological or anatomical structures among many in the environment of the biological structure. In the example of colonoscopy, the detected anomalies may include pathological sections such as mucosal abnormalities (polyps, inflammatory bowel disease, Meckel's diverticulum, lipoma, bleeding, neovascular mucosa, etc.) or obstructed mucosa (e.g., sections with poor bowel preparation, dilated colon, etc.). The anomaly detection circuit 322 can detect anomalies in a section of the biological structure using section-specific images or video streams (or features extracted therefrom) obtained before the endoscope is withdrawn into the specific section of the biological structure. Such section-specific images or video streams generally include section-specific insertion images or insertion video streams, or features generated therefrom, during the insertion phase. In some cases, specific segment-specific extracted images or video streams, or features generated therefrom, may be used to detect anomalies.

[0065] In one example, the anomaly detection circuit 322 can detect anomalies using template matching techniques, which recognize anomalies based on a comparison of segmentation-specific endoscopic features (e.g., features characterizing the shape or contour of a structure) with one or more pre-generated templates of known anomalous structures. In another example, the anomaly detection circuit 322 can detect anomalies using techniques based on artificial intelligence (AI) or machine learning (ML). Segmentation-specific endoscopic images or video streams, or features extracted therefrom, can be applied to a trained ML model to automatically recognize the presence, type, size, location, and / or other characteristics of anomalies. In the example, the ML model may be trained to establish a correspondence between endoscopic images or video streams, or features extracted therefrom, and one or more anomalous characteristics. Examples of ML models used to recognize anomalies from endoscopic images or video streams include convolutional neural networks, bidirectional LSTMs, recurrent neural networks, conditional random fields, dictionary learning, or other machine learning techniques (support vector machines, Bayesian models, decision trees, k-means clustering), among many other ML techniques. The trained ML model may be stored in the storage device 340. Examples of training an ML model to detect anomalies, determine segment-specific extraction parameter values, and generate extraction plans for individual segmentation schemes, as well as examples of using a pre-trained ML model, are discussed later in relation to Figures 6A and 6B.

[0066] The endoscope positioning circuit 323 can determine the location of the endoscope in a specific segment of the biological structure in virtually real time. With such real-time information on the endoscope's location, the anomaly detection circuit 322 can associate the detected anomaly with a specific segment of the biological structure. The endoscope's location can be determined using electromagnetic tracking, or using anatomical landmarks detected from images or video streams obtained during the insertion phase of the endoscope, or from image or video stream features extracted therefrom by the image processor 321. In the example of a colonoscopy, the endoscope positioning circuit 323 can analyze images or video streams to recognize colonic landmarks such as the anus, left ascending colon, splenic flexure, transverse colon, right colic flexure, right descending colon, cecum, appendix, and terminal ileum. In this example, the endoscope positioning circuit 323 can recognize landmarks using template matching techniques. In another example, the endoscope localization circuit 323 can recognize markers using AI or ML techniques, such as a trained ML model trained to establish a correspondence between endoscopic images or video streams, or features extracted therefrom, and marker recognition. Examples of ML models used to recognize anatomical markers include, among many other ML models, deep belief networks, ResNet, DenseNet, autoencoders, capsule networks, generative adversarial networks, Siam networks, convolutional neural networks (CNNs), deep reinforcement learning, support vector machines (SVMs), Bayesian models, decision trees, and k-means clustering. The trained ML model may be stored in the storage device 340. Once the location of the endoscope is determined, the endoscope localization circuit 323 can register the endoscope's location in virtually real-time in a pre-generated template of biological structures. Information on the endoscope's location within the biological structure can be presented to the user (e.g., an endoscopist) in the user interface 330, as further examined in connection with Figure 5.

[0067] The withdrawal plan generation device 324 can generate individual segment withdrawal plans using information about detected anomalies, as generated by the anomaly detection circuit 322, and can generate the substantially real-time location of the endoscope when an anomaly is detected, as generated by the endoscope positioning circuit 323. The individual segment withdrawal plans may include target or recommended values ​​for the endoscope withdrawal parameter (P) for each individual segment of the biological structure. The target or recommended segment-specific endoscope withdrawal parameter values ​​can be estimated using the withdrawal parameter estimation device 325. In one example, the endoscope withdrawal parameter P includes a target or recommended segment-specific endoscope withdrawal speed limit (WSL). The WSL represents the speed limit or acceptable withdrawal speed range for any particular segment of the biological structure. In another example, the endoscope withdrawal parameter P includes a target or recommended segment-specific endoscope withdrawal time limit (WTL). The WTL represents the maximum allowable time or acceptable time range for the endoscope to remain in any particular segment of the biological structure during withdrawal. In the case of colonoscopy, segment-specific endoscopic withdrawal parameters include, for example, P for the cecal segment. C P for the ascending section A P for horizontal division T P for the lower row section D P for the sigmoid colon S , and P for the rectosigmoid segment R Each of these may have its own target or recommended extraction parameter value. Other categories or subcategories may be defined by the user, and the corresponding target or recommended extraction parameter value may be determined by the extraction parameter estimator 325.

[0068] The extraction parameter estimation device 325 can use AI or ML techniques to determine target or recommended values ​​for section-specific endoscopic extraction parameters (e.g., WSL or WTL) for each individual section. In the example, the results of anomaly detection from the anomaly detection circuit 322, along with marker detection and real-time endoscope location from the endoscope positioning circuit 323, may be applied to a trained ML model. The ML model may be trained to establish a correspondence between anomaly characteristics and target or recommended values ​​for section-specific endoscopic extraction parameters (e.g., WSL or WTL). The trained ML model may be stored in the storage device 340. Examples of training an ML model to detect anomalies, determine section-specific extraction parameter values, and generate extraction plans for individual section schemes, as well as examples of using a trained ML model, are discussed later in relation to Figures 6A and 6B.

[0069] In some examples, the extraction parameter estimator 325 can calculate an anomaly score using one or more anomaly characteristics, such as the type, magnitude, shape, location, or quantity of the anomaly. The anomaly score may have a numerical value, such as within a range of 0 to 10. In some examples, a composite anomaly score may be generated, such as by using a linear or nonlinear combination of multiple anomaly scores, each quantifying an anomaly characteristic. In the examples, the composite anomaly score may be calculated by averaging or by taking the anomaly score for the worst-case scenario within the section. The extraction parameter estimator 325 can map the anomaly score (or composite anomaly score) to an extraction parameter value based on comparison with one or more thresholds or value ranges. A larger anomaly score, which typically indicates a more severe anomaly, may be mapped to a smaller WSL to recommend a slower extraction in this section, or to a longer WTL to recommend a longer extraction time in this section. An established correspondence between an anomaly score or range of anomaly scores and the corresponding target or recommended value of a section-specific endoscopic withdrawal parameter can be determined by a lookup table to a normative database or through a rules-based system. The established mapping can be stored in a storage device 340. In another example, the withdrawal parameter estimation device 325 can apply the anomaly score or composite anomaly score to a trained ML model to determine the target or recommended value of a section-specific endoscopic withdrawal parameter for each individual section. In some examples, the anomaly score calculation may be included within the ML model, such as in a layer of a neural network model.

[0070] In some examples, the withdrawal parameter estimation device 325 may further use the auxiliary input unit 315 to determine target or recommended endoscopic withdrawal parameter values ​​for individual segments of the biological structure. Referring to Figure 4, the auxiliary input unit 315 may include, but are not limited to, images or video streams and clinical data from previous endoscopic procedures 410 performed on the patient, patient information and medical history 420, or pre-procedure imaging study data 430 (e.g., among several imaging diagnostic methods, X-ray or fluoroscopic images, electrophoretic or electrical impedance maps, computed tomography (CT) images, magnetic resonance imaging (MRI) images). In the example of a colonoscopy, the auxiliary input unit may include previous colonoscopies in the context of surveillance, including polyps left by the endoscopist in the site or region of the colon operated on. Patient information and medical history 420 may include clinical demographic information, past and present symptoms, and procedures received in a typical endoscopic setting.

[0071] In some examples, the auxiliary input unit 315 may additionally or alternatively include a user profile 440 (e.g., endoscopist) including the user's experience, work environment (e.g., hospital environment or outpatient screening center), compatibility with new technologies, and preferences for specific endoscopic protocols. The user profile 440 may be provided by the user via the user interface 330. Alternatively, the user profile 440 may be automatically generated or updated through learning from the user's past choices and training. For example, a larger WSL (corresponding to a higher withdrawal speed) or a smaller WTL (corresponding to a shorter examination time) in a particular category may be preferred by an experienced endoscopist, while a smaller WSL (corresponding to a slower withdrawal pace) or a larger WTL (corresponding to a longer examination time) may be recommended to an endoscopist in training.

[0072] In some examples, the auxiliary input section 315 may additionally or alternatively include information 450 about the endoscope and instrument. This may include, for example, the specifications of the endoscope 310, the size, dimensions, shape, and structure of the tissue area, specimen, or treatment instrument, including the type, size, dimensions, shape, and structure of the endoscope or other maneuverable instrument such as a cannula, catheter, or guide wire supporting imaging and illumination modes, as well as the current status of the instrument, including which illumination mode is on, which internal buttons are engaged with water jets, air insufflation, etc., which illumination mode is supported, or whether magnification is on, and the current magnification selection.

[0073] If there are additional AI or ML algorithms running in the background, the states of these AI or ML algorithms, hereafter collectively referred to as "AI results" 460, may be additional elements of the auxiliary input unit 315 that can be sent to the extraction plan generation device 324. Examples of AI results 460 may include AI algorithms for detecting anomalies, markers, or other features or structural elements of interest in the target biological structure.

[0074] The estimated segment-specific target withdrawal parameter values ​​(e.g., segment-specific SL or WTL) may be used to generate an endoscopy withdrawal map 326, which is a graphical representation of the endoscopy withdrawal plan. The endoscopy withdrawal map 326 may be displayed on the user interface 330 and may be used as a visual guide to assist the endoscopist during the endoscopic procedure. The endoscopy withdrawal map 326 may include depictions of individual segments of biological structures, each coded in color or grayscale to indicate the respective target or recommended segment-specific endoscopy withdrawal parameter values. The endoscopy withdrawal map 326 may be displayed on the user interface 330 as a reference to guide the endoscopist in withdrawing the endoscope during the endoscopic procedure. In some examples, the estimated withdrawal parameter values ​​created by the withdrawal parameter estimation device 325 may be displayed on the user interface 330. In some examples, information about previously detected anomalies, such as during the insertion phase of the endoscope or from a previous endoscopic procedure, may be displayed on the endoscopy withdrawal map 326 to serve as an additional layer of caution to prevent excessive withdrawal speed in anomaly segments. Other information may be displayed on the user interface 330, including endoscopic images and image features, information about abnormalities and landmarks detected in each section of the biological structure, or the substantially real-time location of the endoscope during the endoscopic procedure.

[0075] In some examples, segmented withdrawal plans, including estimated withdrawal parameter values ​​and an endoscope withdrawal map 326, may be stored in the storage device 340. Information about anomalies detected previously (e.g., during the insertion phase) may also be stored in the storage device 340. The storage device 340 may be local to the endoscope system 300. Alternatively, the storage device 340 may be a separate, remote storage device, such as part of a cloud comprising one or more storage devices and computer devices (e.g., servers) that provide secure access to cloud-based services, including, among many, data storage, computer services, and customer service provision. In some examples, at least some of the data processing and calculations related to anomaly detection, landmark recognition, endoscope positioning, withdrawal parameter estimation, and endoscope withdrawal map generation may be performed in the cloud. For example, images or video streams, or features extracted therefrom, may be streamed to the cloud and processed there, and the calculation results, such as segmented withdrawal plans (e.g., endoscope withdrawal maps 326), may be relayed back to a local endoscope system for use in image-guided endoscope withdrawal.

[0076] The real-time endoscopic withdrawal tracking circuit 327 can track and measure withdrawal parameters (e.g., withdrawal speed or total withdrawal time consumption) in a specific segment of a biological structure, such as a colon segment, in virtually real time. Various techniques can be used to track and measure the endoscopic withdrawal speed. In one example, the endoscopic withdrawal speed may be estimated using an optical flow-based method. Optical flow is the apparent pattern of motion of an image subject between two consecutive frames caused by the movement of the subject or camera. The endoscopic withdrawal speed is actively correlated with the rate of change of image frames in the endoscope. A faster rate of change of frames corresponds to a faster flow or endoscopic withdrawal speed. In another example, the endoscopic withdrawal speed may be estimated using electromagnetic (EM) tracking techniques. An EM coil incorporated along the length of the endoscope's insertion tube generates a pulsed, low-intensity magnetic field that can be received by a receiving device. The EM pulses are used to calculate the precise position and orientation of the insertion tube. The endoscopic withdrawal speed can then be estimated based on the rate of change in the endoscope position. In yet another example, the endoscope withdrawal speed may be estimated with respect to landmarks, such as those recognized from the endoscopic image by the endoscope positioning circuit 323, as described above. By using the landmarks as a reference or reference point, the endoscope withdrawal speed can be estimated based on the change in relative position with respect to the recognized landmarks.

[0077] In some cases, the real-time endoscopy withdrawal tracking circuit 327 can determine the start of the withdrawal phase and activate withdrawal rate tracking and measurement only during the withdrawal phase. The real-time endoscopy withdrawal tracking circuit 327 can additionally determine whether any action is performed by the endoscopist. Examples of such actions performed, but not limited to, include water jetting and suction of debris, insertion of instruments through the work passage (forceps, snare, cytology brush, need for sclerotherapy or mucosal injection, aspiration catheter). Once the withdrawal phase is recognized, the withdrawal clock starts and withdrawal rate tracking is activated while the mucosa is being examined. When the colonoscopy is in another phase, withdrawal rate tracking can be paused so as not to interfere with the endoscopist's actions.

[0078] During withdrawal, the endoscope positioning circuit 323 can determine the location of the endoscope in one of the individual segments of the biological structure in substantially real time, based at least partially on the endoscope image or video stream, or features extracted therefrom. As similarly discussed above with respect to locating the endoscope during the insertion phase, the location of the endoscope may be determined based on anatomical landmarks recognized from the endoscope image or video stream. The substantially real-time location of the endoscope may be registered in a pre-generated template of the biological structure. The substantially real-time location of the endoscope points to the current segment of the biological structure from which the endoscope image or video stream is obtained. Measured real-time endoscope withdrawal parameter values ​​(e.g., withdrawal speed) created by the real-time endoscope withdrawal tracking circuit 327 may be associated with the currently recognized segment to create measured segment-specific withdrawal parameter values. The endoscope withdrawal control device 328 can compare the measured segment-specific withdrawal parameter values ​​with target or recommended segment-specific withdrawal parameter values ​​and determine whether the measured withdrawal parameter values ​​are within a specific difference of the target values. For example, the endoscope withdrawal control device 328 measures the endoscope withdrawal speed S in category "j". jcan be compared with the withdrawal speed limit (WSL) WSL of the same section "j" j . The measured withdrawal speed S j is within a specific difference δ of WSL j , that is, when WSL j -δ≦S j ≦WSL j +δ, S j is considered appropriate. When S j deviates from WSL by exceeding the specific difference δ j , that is, S j <WSL j -δ or S j >WSL j +δ, an alarm may be generated to warn the user of an inappropriately fast withdrawal speed (when S j >WSL j +δ) or an inappropriately slow withdrawal speed (when S j <WSL j -δ) in section "j". By adjusting the withdrawal speed to substantially follow the target speed WSL j (for example, WSL j -δ≦S j ≦WSL j +δ), a recommendation may be provided to the user.

[0079] In some examples, during the endoscope withdrawal phase, the measured section-specific endoscope withdrawal speed S jThe substantially real-time endoscopic position in a biological structure segment (as determined by the endoscopic positioning circuit 323) may be displayed on the user interface 330 (e.g., adjacent to or superimposed on) along with the endoscopic withdrawal map 326. This provides direct visual feedback to the user during endoscopic withdrawal, as will be further illustrated and discussed later with respect to Figure 5. In some examples, during withdrawal, before the endoscope is withdrawn beyond a previously identified abnormal segment (such as one identified during the insertion phase), the user may be notified of the abnormal segment to be reached and, when passing through the abnormal segment, may be warned in advance to withdraw at a speed not exceeding the segment-specific WSL. Warnings to the user may be communicated either through optical means in the diagnostic monitor or through auditory means such as a warning alarm. In some examples, highlighting, flashing warnings, auditory or tactile feedback may be provided to the user to highlight the abnormal segment to be reached. Upon completion of the withdrawal (e.g., when the rectum is detected), the endoscopist may be presented with, for example, the withdrawal time and / or average speed for each section, the overall net withdrawal time and average speed, the overall net withdrawal time including any abnormalities detected, post-procedure analysis, and a summary of the withdrawal. The post-procedure analysis may be used for quality assurance and to determine whether the colonoscopy procedure was successful.

[0080] In some examples, the system 300 may operate in a closed-loop manner with a feedback loop for continuous learning of segmental withdrawal plans, such as updating target or recommended segment-specific endoscopic withdrawal parameter values. Continuous learning may be via explicit endoscopist feedback (e.g., satisfaction with recommendations, "preference" button, etc.). Alternatively, the system 300 may run in shadow mode to an experienced endoscopist, allowing monitoring of the endoscopist's withdrawal behavior and modification of the segmental withdrawal plan through reinforcement feedback.

[0081] In some cases, image-guided endoscopy, or a portion thereof, such as endoscopic withdrawal, may be performed using a robotic system 350. The robotic system 350 may comprise a robotic arm that is detachably attached to the endoscope 310. The robotic arm can position and manipulate the endoscope 310 at an anatomical target, or position the device at a desired location and in a desired posture, automatically or semi-automatically (e.g., with minimal manual control or command by the user) via actuators, to facilitate surgery at an anatomical target. The robotic system 350 may comprise a robotic control device to control the movement of the robotic arm according to a segmented withdrawal plan, including estimated withdrawal parameter values ​​created by an withdrawal parameter estimation device 325 and / or an endoscopic withdrawal map 326.

[0082] Figure 5 illustrates different stages of an image-guided endoscopic procedure 500, such as a colonoscopy as shown in this example. The procedure includes an insertion phase and a subsequent withdrawal phase. The procedure may be performed using system 300. During the insertion phase, images or video streams from the endoscope are obtained from each of several colonic segments, including the rectosigmoid segment, sigmoid colon segment, descending segment, transverse segment, ascending segment, and cecum, using the imaging system 312, for example. Various colonic segments can be recognized by the endoscope positioning circuit 323. During insertion, abnormalities can also be automatically detected using the abnormality detection circuit 322. At the end of the insertion phase, when the distal tip of the endoscope reaches the cecum, the segment-specific images or video streams obtained during the insertion phase can be analyzed, and individual segment-based withdrawal plans can be generated using the withdrawal plan generation device 324, for example. Segment-based withdrawal plans can be graphically represented by a colonoscopy withdrawal speed map 510 (example of endoscope withdrawal map 326). The withdrawal speed limit (WSL) for each of the multiple colonic segments can be coded in color (or grayscale) and displayed on the colonoscopy withdrawal speed map 510. In an example as shown in Figure 5, the color- or grayscale-coded WSLs may include a “normal” or relatively fast WSL range, a “cautious” or intermediate WSL range, and a “slow” withdrawal WSL range. As an example, and not an limitation, a “normal” WSL range is approximately 3–4 millimeters per second, a “cautious” WSL range is approximately 40 mm / sec or more, and a “slow” WSL range is approximately 1–3 mm / sec. Other information obtained or calculated, such as detected or recognized abnormal segments during the insertion phase or from previous endoscopic procedures, may also be displayed on the colonoscopy withdrawal speed map 510.

[0083] In some examples, the images or video streams used to determine target or recommended endoscopic withdrawal parameter values ​​(and to generate individual segmental withdrawal plans, such as a colonoscopy withdrawal speed map 510) may not be limited to endoscopic images or video streams obtained during endoscopic insertion, but may include endoscopic images or video streams obtained during the withdrawal phase before the endoscope is withdrawn beyond the segment being examined and treated. For example, a target or recommended withdrawal speed limit (WSL) for any particular colon segment "j" may be determined at any time before the colonoscope is withdrawn beyond that segment "j".

[0084] The colonoscopy withdrawal rate map 510 may be displayed to the user throughout the withdrawal phase to assist the endoscopist when maneuvering the endoscope to examine the mucosa in colonic segments and to perform necessary procedures (e.g., polypectomy). Additionally or alternatively, the colonoscopy withdrawal rate map 510 may be used to control a robotic system 350 in robot-assisted colonoscopy. As shown in Figure 5, the substantially real-time location of the endoscope 522, such as that determined by the endoscope positioning circuit 323 during withdrawal, may be displayed superimposed on the colonoscopy withdrawal rate map 510 to create an overlay plot 520. When the endoscope tip is in or about to reach a particular colonic segment, notifications such as a marker 524 may be displayed on the user interface to inform the user about the WSL in the current or about-to-reach segment. In each colonic segment, the actual endoscope withdrawal rate may be measured using, for example, a real-time endoscope withdrawal tracking circuit 327. The measured segment-specific withdrawal rate can be compared to the WSL stored in the withdrawal plan for the individual segment scheme to determine whether the measured rate falls within the WSL-specific variation for that segment. As described above, when the measured rate exceeds the WSL-specific variation, an alarm may be generated to warn the user (e.g., the endoscopist) about an inappropriate withdrawal rate. Recommendations may be provided to the user to adjust the withdrawal rate to substantially match the WSL.

[0085] Figures 6A and 6B illustrate an example of training an ML model to determine a target or recommended segment-specific endoscopic withdrawal parameter value (e.g., WSL or WTL) for each of several individual segments of a biological structure, and using the trained ML model. The target or recommended segment-specific endoscopic withdrawal parameter values ​​can be used to construct an endoscopic withdrawal map 326 or a colonoscopy withdrawal speed map 510. Figure 6A shows an ML model training (or learning) phase in which an ML model 620 can be trained to determine a target or recommended colonoscopy withdrawal speed limit (WSL) in a particular segment, based at least partially on endoscopic images of that segment "j" (e.g., cecum). The training database may include a group of endoscopic images 610 of the same colon segment "j" obtained from colonoscopy procedures performed in multiple patients. In some examples, the training dataset may also include an auxiliary input unit 315, as described above in relation to Figure 3. The ML model 620 may have a neural network structure comprising an input layer, one or more hidden layers, and an output layer. Multiple endoscopic images 610, or features generated therefrom, may be fed into the input layer of the ML model 620, which propagates the input data or data features through one or more hidden layers to an output layer that outputs WSLs for colonic segments (e.g., cecum). The ML model 620 can perform tasks without explicit programming by making inferences based on patterns observed in the analysis of the data. The ML model 620 explores the study and construction of algorithms (e.g., ML algorithms) that can learn from existing data and make predictions about new data. Such algorithms work by constructing the ML model 620 from training data to represent data-driven predictions or decisions as outputs or evaluations.

[0086] ML model 620 may be trained using supervised or unsupervised learning. Supervised learning uses prior knowledge (e.g., examples of relating inputs to outputs or results) to learn the relationships between inputs and outputs. The goal of supervised learning is to learn a function that best approximates the relationships between inputs and outputs to be trained, given some training data, so that the ML model can implement the same relationships when given inputs to produce corresponding outputs. Unsupervised learning is the training of an ML algorithm using unclassified and unlabeled information, allowing the algorithm to act on that information without guidance. Unsupervised learning is useful in exploratory analysis because it can automatically identify structures in the data.

[0087] Common tasks for supervised learning include classification and regression problems. Classification problems, also called categorization problems, aim to classify items into one of several categorical values. Regression algorithms aim to quantify several items (for example, by providing scores to several input values). Some examples of commonly used supervised ML algorithms are logistic regression (LR), naive Bayes, random forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and support vector machines (SVM). Examples of DNNs include convolutional neural networks (CNN), recurrent neural networks (RNN), deep belief networks (DBN), or hybrid neural networks that include two or more neural network models of different types or different model configurations. Some common tasks for unsupervised learning include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised learning algorithms are K-means clustering, principal component analysis, and autoencoders.

[0088] Another type of machine learning is federative learning (also known as collaborative learning), which trains algorithms across multiple distributed devices that hold local data without exchanging data. This approach contrasts not only traditional centralized machine learning techniques where all local databases are uploaded to a single server, but also older distributed methods that often assume local data samples are similarly distributed. Federative learning allows multiple actors to build a common, robust machine learning model without sharing data, thereby addressing critical concerns such as data privacy, data security, data access rights, and access to heterogeneous data.

[0089] Training of the ML model 620 can be performed continuously, periodically, or in near real-time as additional treatment data becomes available. The training process involves algorithmically adjusting one or more ML model parameters (e.g., weights or biases in any particular layer of the neural network model) until the ML model being trained meets a specific training convergence criterion. As an example, but not an limitation, the ML model 720 may be trained with weighted squared error loss (for explicit feedback) or binary cross-entropy loss (for implicit feedback). Other training techniques may be used, such as deep factorization machines, extensive and deep learning, deep structured semantic models, or autoencoder-based recommendation systems. The trained ML model 620 can establish a correspondence between endoscopic images 610 (or features extracted from the images) and target or recommendation WSLs for category "j".

[0090] The same training processes and techniques described above may be used to train other ML models using their respective training datasets, each containing groups of endoscopic images of the same colonic segment (e.g., segment "j" and segment "k" different from the above) obtained from colonoscopy procedures performed on multiple patients. The resulting multiple trained ML models can each predict the target or recommended WSL for their respective colonic segments. For example, a first ML model for predicting WSL in the cecum, a second ML model for predicting WSL in the ascending colon, a third ML model for predicting WSL in the transverse colon, a fourth ML model for predicting WSL in the descending colon, a fifth ML model for predicting WSL in the sigmoid colon, and a sixth ML model for predicting WSL in the rectosigmoid colon.

[0091] Figure 6B shows the WSL specific to category "j". j The inference phase shows how live endoscopic images 630 of a specific segment are applied to a trained ML model 620 to automatically determine 640. The live endoscopic images 630 are WSL j 640 is obtained during the insertion phase of the endoscope so that it can be determined at the end of the insertion phase. Alternatively, live endoscopic images 630 are taken intermittently before the colonoscope is withdrawn beyond section "j" WSL. j 640 can be determined during the withdrawal phase before the endoscope is withdrawn beyond section "j". WSL j 640 may be used to construct individual colonoscopy withdrawal rate maps 510, along with WSLs estimated for other sections, by applying images of each colon segment to a pre-trained ML model.

[0092] During withdrawal, the position of the endoscope can be determined in virtually real time, for example, using the endoscope positioning circuit 323. The position of the endoscope can be displayed on the colonoscopy withdrawal speed map 510 to create an overlay plot 520 that can be displayed to the endoscopist during the procedure. The endoscope withdrawal speed can be tracked and measured in virtually real time, for example, using the real-time endoscope withdrawal tracking circuit 327. When the endoscope is withdrawn to section "j", the actual withdrawal speed S j 650 is measured. j 650 is WSL j To determine if it is among the 640 specific differences, WSL j It can be compared to 640. As stated above, the withdrawal rate is considered appropriate if the measured rate is within the range of the WSL recommendation. When the measured rate exceeds the range of the WSL recommendation, an alarm may be generated to warn the user (e.g., endoscopist) of an inappropriate withdrawal rate. Recommendations may be provided to the user to adjust the withdrawal rate to substantially match the WSL.

[0093] The ML models described above are trained to estimate target or recommended segment-specific endoscopic withdrawal parameter values ​​(e.g., WSL or WTL) for any specific segment of a biological structure. In some examples, multiple ML models may be separately trained, enabled, and used (in the inference phase) for other applications such as anomaly detection or anatomical landmark recognition. In the example, an ML model may be trained and used by an anomaly detection circuit 322 to detect anomalies from input endoscopic images of a specific colon segment, and another ML model may be trained and used by an endoscopic localization circuit 323 to detect anatomical landmarks from input endoscopic images of a specific colon segment.

[0094] Figures 7A and 7B show illustrative graphs of guided withdrawal and speed tracking in a colonoscopy procedure using individual section-specific endoscopic withdrawal plans. Actual colonoscopy withdrawal speeds (such as those tracked and measured by a real-time endoscopic withdrawal tracking circuit 327) and target or recommended WSLs for each of multiple (e.g., N) sections can be plotted on the same graph and displayed to the user during withdrawal. Figure 7A shows the actual withdrawal speeds 710 measured in each of the N sections, along with the respective section-specific WSLs (e.g., WSL1720A for section 1, WSL2720B for section 2, WSL3720C for section 3, WSL4720D for section 4, WSL5720E for section 5, ..., WSL for section N) N This shows a “good” endoscopic procedure that remains below 720N. Figure 7B shows withdrawal rate tracking in another procedure. The actual withdrawal rates 730 in sections 1-4 are below the respective section-specific WSLs (i.e., WSL1-WSL4). However, in section 5, the actual measured withdrawal rate 732 exceeds the rate limit WSL5720E. Accordingly, an alarm may be triggered, and the endoscopist is warned to slow the withdrawal in that section. When the endoscope has examined a particular portion of section 5 at a rate of 732, in order to re-examine that portion at a slower rate, the endoscope is reinserted to the beginning of section 5 (or the end of the previous section 4), and from there withdrawn through section 5 at a slower rate 734 below WSL5720E.

[0095] Figure 8 is a flowchart illustrating an exemplary method 800 for creating a segment-specific endoscopic withdrawal plan and using the plan to guide endoscopic withdrawal in an endoscopic procedure such as a colonoscopy. The segment-specific endoscopic withdrawal plan may be created using images or video streams of individual segments of a biological structure. Method 800 may be performed in an endoscopic system 300. While the processes of Method 800 are depicted in a single flowchart, they do not need to be performed in a specific order. In various examples, some of the processes may be performed in an order different from that shown herein.

[0096] In reference numeral 810, an imaging system associated with an endoscope may be used to obtain images or video streams of individual segments of a target biological structure. In the example of a colonoscopy, individual segments (e.g., one or more of the cecum, ascending, transverse, descending, sigmoid colon, or rectosigmoid colon) may be imaged separately to create a segmental scheme or segment-specific image or video stream. Images or video streams may be acquired when the imaging system is set to one of several available imaging modes. The imaging modes refer to one or more of the zoom settings, contrast settings, exposure levels, field of view, or lighting or illumination conditions toward the target biological structure. In the example, images or video streams may be acquired during the insertion phase of the endoscopic procedure, and segmental schemes or segment-specific insertion images or video streams may be acquired from each of several segments (e.g., colon segments).

[0097] In reference 820, the obtained segmentation image or video stream may be analyzed to generate endoscopic image or video features for each individual segment of the biological structure. Examples of image or video features include statistical features of pixel values, or morphological features such as angles, edges, clumps, curvature, Speeded Up Robust Feature (SURF), or Scale Invariant Feature Transform (SIFT). In some examples, segmentation-specific images or video streams may be pre-processed, such as by filtering, resizing, orientation, or color or grayscale correction, and endoscopic image features may be extracted from the pre-processed images or video streams. In some examples, endoscopic image or video features may be post-processed to improve feature quality, such as by edge interpolation or edge extrapolation, to create continuous and smooth edges.

[0098] In reference numeral 830, individual segmental endoscope withdrawal plans can be generated, for example, using the withdrawal plan generation device 324 as shown in Figure 3, based at least in part on endoscopic image features or endoscopic video features. The segmental endoscope withdrawal plan may include target or recommended segment-specific endoscope withdrawal parameter values ​​for each individual segment of the biological structure. As an example but not an limitation, as discussed above with respect to Figure 3, the target or recommended segment-specific endoscope withdrawal parameters may include a target or recommended segment-specific endoscope withdrawal speed limit (WSL) or a target or recommended segment-specific endoscope withdrawal time limit (WTL) for each individual segment. WSL represents the speed limit or acceptable withdrawal speed range for each individual segment of the biological structure. WTL represents the maximum acceptable time or acceptable time range for the endoscope to remain in each individual segment during the endoscope withdrawal process. In the case of colonoscopy, segment-specific endoscopic withdrawal parameters may have target or recommended values, such as WSL or WTL, for one or more of the cecal, ascending, transverse, descending, sigmoid, or rectosigmoid segments. Other segments or sub-segments may be defined by the user, and the corresponding target or recommended withdrawal parameter values ​​may be determined similarly.

[0099] In some examples, target or recommended segment-specific endoscopic withdrawal parameter values ​​may be estimated using information about abnormalities detected in any particular segment of the biological structure. Abnormalities in any particular segment can be detected from at least segment-specific endoscopic images or videos using template matching techniques, artificial intelligence (AI) techniques, or machine learning (ML)-based techniques. Abnormality detection involves detecting and / or recognizing one or more of the presence, type, size, shape, location, or quantity of pathological or anatomical structures among many in the biological structure's environment. In the example of colonoscopy, detected abnormalities may include pathological segments such as mucosal abnormalities (e.g., polyps, inflammatory bowel disease, Meckel's diverticulum, lipoma, bleeding, neovascular mucosa) or obstructed mucosa (e.g., segments with poor bowel preparation, dilated colon). In some examples, target or recommended segment-specific endoscopic withdrawal parameter values ​​may be further estimated using the virtually real-time location of the endoscope when abnormalities are detected. The location of the endoscope can be determined using electromagnetic tracking, or using anatomical landmarks recognized from images or video streams obtained during the insertion phase of the endoscope, or from image or video stream features extracted therefrom. In an example of landmark-based endoscope localization, the landmarks may be recognized using AI or ML techniques, such as a trained ML model trained to establish a correspondence between the images or video streams of the endoscope (or features extracted therefrom) and landmark recognition, as described above in relation to Figure 3.

[0100] For example, the endoscopic withdrawal parameter values ​​specific to a target or recommended category may be further estimated using an auxiliary input unit, including, but not limited to, image or video streams and clinical data from previous endoscopic procedures performed on the patient, patient information and medical history, or pre-procedure imaging study data, user (e.g., endoscopist) profile (including user experience, work environment, compatibility with new technologies, preference for specific endoscopic protocols, etc.), endoscope and instrument information, or "AI results" including the state of AI or ML algorithms for detecting abnormalities, landmarks, or other features or structural elements of interest in the target's biological structure, as described above in relation to Figure 4.

[0101] Target or recommended section-specific endoscopic withdrawal parameter values ​​(e.g., WSL or WTL for the target) for individual items within each section can be estimated using AI or ML methods. In an example, the results of anomaly detection, marker detection and real-time endoscopic location information, and auxiliary inputs may be applied to a trained ML model. The ML model may be trained to establish a correspondence between the combined inputs and target or recommended values ​​for section-specific endoscopic withdrawal parameters (e.g., WSL or WTL), as described above in relation to Figures 6A-6B.

[0102] Target or recommended segment-specific endoscopic withdrawal parameter values ​​(e.g., segment-specific WSL or WTL) may be used to generate an endoscopic withdrawal map that graphically depicts the endoscopic withdrawal plan, such as a colonoscopy withdrawal speed map 510 shown in Figure 5. The colonoscopy withdrawal speed map may include depictions of individual segments of biological structures, each coded in color or grayscale to indicate the respective target or recommended segment-specific endoscopic withdrawal parameter values. The endoscopic withdrawal map may be displayed in the user interface as a reference to guide the endoscopist during the endoscopic procedure. In some examples, information about previously detected anomalies, such as during the insertion phase of the endoscope or from a previous endoscopic procedure, may be displayed on the endoscopic withdrawal map to serve as an additional layer of caution to prevent excessive withdrawal speed in anomaly segments during endoscopic withdrawal.

[0103] In reference 840, during the withdrawal phase, endoscopic withdrawal parameters (e.g., withdrawal speed or total withdrawal time consumption) in a specific segment of the biological structure can be tracked and measured. Various techniques can be used to track and measure the endoscopic withdrawal speed. In one example, the endoscopic withdrawal speed can be estimated using an optical flow-based method, and the endoscopic withdrawal speed can be estimated using the rate of change of endoscopic image frames. In another example, the endoscopic withdrawal speed may be estimated using electromagnetic (EM) tracking techniques. In yet another example, the endoscopic withdrawal speed may be estimated in relation to landmarks, such as those recognized from endoscopic images, as described above. By using landmarks as reference or reference points, the endoscopic withdrawal speed can be estimated based on changes in relative position with respect to the recognized landmarks. In some examples, during the endoscopic withdrawal phase, the location of the endoscope in one of the individual segments of the biological structure can be detected in substantially real time, at least in part, based on endoscopic images or video streams, or features extracted therefrom. The measured real-time endoscopic withdrawal parameter values ​​(e.g., withdrawal speed) may be associated with the currently recognized segment to create segment-specific withdrawal parameter values ​​for the measured segment.

[0104] The endoscope withdrawal plan generated in step 830 and the actual endoscope withdrawal parameter values ​​measured in step 840 may be provided to the user or process to facilitate manual or robotic withdrawal of the endoscope. In reference numeral 852, the endoscope withdrawal plan, such as an endoscope withdrawal map, may be displayed to the user as shown in Figure 5. The measured section-specific withdrawal parameter value (e.g., withdrawal rate) may be compared to a target or recommended section-specific withdrawal parameter value to determine whether the measurement is within a specific difference of the target value. The withdrawal rate is considered appropriate if the measured withdrawal rate falls within a specific difference of the rate limit for that section. When the measured withdrawal rate deviates from the rate limit by a specific difference, an alarm may be generated in reference numeral 854 to warn the user (e.g., endoscopist) about an inappropriately fast or inappropriately slow withdrawal rate. Recommendations may be provided to the user to adjust the withdrawal rate to substantially match the rate limit for that section. In some examples, the virtually real-time position of the endoscope in the segmentation of the biological structure may be displayed together with the endoscope withdrawal map (e.g., adjacent or superimposed), as shown in Figure 5. This provides the user with direct visual feedback during endoscope withdrawal. Additionally or alternatively, in some examples, a control signal may be generated in reference numeral 856 to a robotic system that can adjust the endoscope withdrawal speed according to a segmentation-based withdrawal plan.

[0105] Figure 9 provides a general block diagram of an exemplary machine 900 that can be implemented by any one or more of the techniques (e.g., methodologies) discussed herein. This portion can be applied to the computer framework of various parts of the endoscopic system 300.

[0106] In alternative examples, machine 900 can operate as a standalone device or can be connected to other machines (e.g., via a network). In a networked deployment, machine 900 can operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 900 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. Machine 900 could be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions that specify the actions to be taken by that machine. Furthermore, although only a single machine is shown, the term “machine” should be interpreted to include any set of machines that individually or collectively execute a set (or more sets) of instructions to implement one or more of the methodologies considered herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0107] Examples may include, or be operated by, logical or several components or mechanisms, as described herein. A circuit set is a collection of circuits implemented in tangible entities, including hardware (e.g., simple circuits, gates, logic, etc.). The components of a circuit set may be flexible with respect to time and basic hardware variability. A circuit set includes components that can perform specific operations individually or in combination when operating. In an example, the hardware of a circuit set may be designed to be immutable to perform a specific operation (e.g., connected by wiring). In an example, the hardware of a circuit set may include variable-connected physical components (e.g., execution units, transistors, simple circuits, etc.) that include a computer-readable medium that is physically modified (e.g., magnetically, electrically, or the movable location of an invariant mass particle) to code instructions for a specific operation. When physical components are connected, the underlying electrical properties of the hardware components are changed, for example, from an insulator to a conductor or vice versa. Instructions allow embedded hardware (e.g., execution units or loading mechanisms) to create components of a circuit set in the hardware via variable connections to perform specific parts of operation while in operation. Thus, a computer-readable medium is communicably connected to other components of the circuit set components while the device is in operation. In the example, any of the physical components can be used by two or more components of two or more circuit sets. For example, during operation, an execution unit may, at some point, be used by a first circuit in a first circuit set, and at a different time, be used again by a second circuit in the first circuit set, or by a third circuit in the second circuit set.

[0108] The machine (e.g., a computer system) 900 may comprise hardware processors 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof) that can communicate with each other, in part or in whole, via an interlink (e.g., a bus) 908, main memory 904, and static memory 906. The machine 900 may further comprise a display unit 910 (e.g., a raster display, a vector display, a holographic display, etc.), an alphanumeric input device 912 (e.g., a keyboard), and a user interface (UI) guidance device 914 (e.g., a mouse). In this example, the display unit 910, the input device 912, and the UI guidance device 914 may be touchscreen displays. The machine 900 may further include a storage device (e.g., a drive unit) 916, a signal generating device 918 (e.g., a speaker), a network interface device 920, and one or more sensors 921 such as a Global Positioning System (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 900 may also include an output control device 928, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless connection (e.g., infrared (IR), near-field communication (NFC), etc.)) for communicating with or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0109] The storage device 916 may include, or comprise, a machine-readable medium 922 on which one or more sets of data structures or instructions 924 (e.g., software) that embody or are utilized by any one or more of the technologies or functions described herein are stored. The instructions 924 may also reside entirely or at least partially in the main memory 904, static memory 906, or hardware processor 902 during execution by the machine 900. In the example, one or any combination of the hardware processor 902, main memory 904, static memory 906, or storage device 916 may constitute the machine-readable medium.

[0110] Although the machine-readable medium 922 is shown as a single medium, the term “machine-readable medium” may include a single or multiple mediums configured to store one or more instructions 924 (e.g., a centralized or distributed database, and / or associated caches and servers).

[0111] The term “machine-readable medium” may include any medium on which instructions for execution by machine 900 can be stored, encoded, or carried, causing machine 900 to perform one or more of the technologies of this disclosure, or any medium on which data structures used by such instructions, or data structures associated with such instructions, can be stored, encoded, or carried. Examples of non-limiting machine-readable mediums may include solid memory, optical media, and magnetic media. In the example, a high-capacity machine-readable medium comprises a machine-readable medium with a plurality of particles having an immutable (e.g., stationary) mass. Thus, a high-capacity machine-readable medium is not a transiently propagating signal. Specific examples of high-capacity machine-readable mediums may include non-volatile memories such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPSOM)) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, CD-ROM disks, and DVD-ROM disks.

[0112] Instruction 924 may be further transmitted or received through a communication network 926 using a transmission medium via a network interface device 920 that utilizes one of several transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Illustrative communication networks may include, among many others, local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), analog telephone service (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 standard family known as Wi-Fi®, the IEEE 802.16 standard family known as WiMAX®), the IEEE 802.15.4 standard family, and peer-to-peer (P2P) networks. In the example, the network interface device 920 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communication network 926. In the example, the network interface device 920 may include multiple antennas for wireless communication using at least one of the following technologies: single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO). The term “transmission medium” should be interpreted to include any intangible medium on which instructions for execution by machine 900 can be stored, encoded, or carried, including digital or analog communication signals or other intangible mediums to facilitate communication of such software.

[0113] Additional notes The above detailed description includes references to the accompanying drawings, which form part of the detailed description. The drawings illustrate, as examples, specific embodiments in which the present invention may be carried out. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those illustrated or described. However, the inventors also consider examples in which only those illustrated or described elements are provided. Furthermore, the inventors also consider examples using any combination or arrangement of those illustrated or described elements (or one or more of their embodiments) with respect to any specific example (or one or more of their embodiments) or any other example (or one or more of their embodiments) illustrated or described herein.

[0114] In this document, the term “one” ("a" or "an") is used to include one or more, independently of any other examples or uses of “at least one” or “one or more,” as is common in patent literature. In this document, the term “or” is used to refer non-exclusively unless otherwise indicated, or to include “A or B” as “A but not B,” “B but not A,” and “A and B.” In this document, the terms “including” and “in which” are used as the plain English equivalents of the terms “comprising” and “wherein.” Furthermore, in the following claims, the terms “including” and “comprising” are open-ended, meaning that any system, device, article, composition, formulation, or process that includes elements in addition to those listed after such terms in a claim is still considered to be within the scope of that claim. Furthermore, in the following claims, terms such as “first--,” “second--,” and “third--” are used merely as labels and are not intended to impose numerical requirements on those objects.

[0115] The above descriptions are intended to be illustrative and not limiting. For example, the above examples (or one or more of their embodiments) may be used in combination with one another. Other embodiments may be used by those skilled in the art when considering the above descriptions. The abstract is provided to allow the reader to quickly grasp the essence of the technical disclosure. The abstract is presented with the understanding that it is not to be used to interpret or limit the claims or their meaning. Also, in the above detailed description, various features may be grouped together for the sake of simplification of the disclosure. This should not be interpreted as meaning that any disclosed feature not claimed is essential to any claim. Rather, the subject matter of the invention may lie less than all the features of a particular disclosed embodiment. Accordingly, the following claims are incorporated herein as examples or embodiments, with each claim standing as a distinct embodiment in itself, and it is considered that such embodiments can be combined with one another in various combinations or rearrangements. The scope of the invention should be determined with reference to the appended claims, along with the equivalent full scope given to such claims. [Explanation of symbols]

[0116] 10 Endoscopy Systems 12. Imaging control system 14 Endoscopy 16 Control Unit 18 Output unit, display unit 20 Input Units 22 Light source units 24 Fluid supply source 26 Suction pump 28 Insertion area 30 functional areas 32 Handle area 34 Cable Area 36 coupler area 38 Control knob 40A, 40B ports 41 Cart 42 Image Processing Unit 44 Game generation device 46 Drive Unit 47 Cables 48 Surgical instruments 300 Endoscopy System 310 Endoscope 312 Imaging System 314 Lighting System 315 Auxiliary Input Section 320 Control circuits 321 Image Processing Device 322 Anomaly detection circuit 323 Endoscope positioning circuit 324 Extraction Plan Generation Device 325 Extraction parameter estimation device 326 Endoscope Withdrawal Map 327 Real-time Endoscope Retraction Tracking Circuit 328 Endoscope withdrawal control device 330 User Interfaces 340 storage devices 350 Robot Systems 410 Previous Endoscopic Procedures 420 Patient information and medical history 430 Pre-treatment imaging study data 440 User Profiles 450 Endoscope and Equipment Information 460 AI results 500 Endoscopic Procedures 510 Colonoscopy Removal Speed ​​Map 520 Overlay plot 522 Endoscope location 524 Marker 610 Endoscopic images 620 pre-trained ML models 630 Live Endoscopic Images 640 Category-Specific WSL, WSL j 650 Actual pull-out speed S j 720A, 720B, 720C, 720D, 720E, 720N WSL 710, 730, 732 Pull speed 900 machines 902 Hardware Processor 904 Main memory 906 Static Memory 908 Interlink 910 Display Unit 912 Alphanumeric input device 914 User Interface (UI) Guidance Devices 916 Storage Devices 918 Signal Generating Devices 920 Network Interface Devices 921 Sensor 922 Machine-readable media 924 Data Structures, Instructions 926 Communication Network 928 Output control device P Endoscopic withdrawal parameter

Claims

1. An endoscope equipped with an imaging system, wherein the imaging system is configured to obtain images or video streams of individual segments of a patient's biological structure during an endoscopic procedure including insertion and subsequent withdrawal of the endoscope into the patient's biological structure, A control circuit, To generate endoscopic image features or endoscopic video features for each of the individual divisions of the biological structure, the obtained images or video streams are analyzed, Based at least partially on the endoscopic image features or the endoscopic video features, an endoscopic withdrawal plan is generated for each of the individual segments of the biological structure, including target or recommended segment-specific endoscopic withdrawal parameter values. To facilitate manual or robotic withdrawal of the endoscope, the endoscope withdrawal plan is provided to the user or robotic system. A control circuit and An endoscopic system equipped with the following features.

2. The endoscopic system according to claim 1, wherein the target or recommended segment-specific endoscopic withdrawal parameter values ​​include a target segment-specific endoscopic withdrawal speed limit (WSL) for each of the individual segments of the biological structure.

3. The endoscopic system according to claim 1 or 2, wherein the target or recommended segment-specific endoscopic withdrawal parameter values ​​include a target segment-specific endoscopic withdrawal time limit (WTL) for each of the individual segments of the biological structure.

4. The endoscope is a colonoscope for use in colonoscopy procedures. The endoscope system according to any one of claims 1 to 3, wherein the imaging system is configured to obtain images or video streams from each of the individual colonic segments during the colonoscopy procedure.

5. The endoscopic system according to any one of claims 1 to 4, wherein determining the endoscopic withdrawal parameter values ​​specific to the target or recommended segment comprises determining the endoscopic withdrawal parameter specific to the first target segment for the first segment of the biological structure, using at least the endoscopic image features or endoscopic video features generated from the image or video stream obtained during the manual or robotic withdrawal of the endoscope before the endoscope is withdrawn beyond the first segment.

6. The endoscope system according to claim 5, wherein the image or video stream obtained before the endoscope reaches the first section includes an image or video stream obtained during the insertion of the endoscope into the biological structure.

7. The aforementioned control circuit is Anomaly detection is performed, which includes detecting one or more anomalies in each of the individual sections based at least partially on the endoscopic image features or the endoscopic video features. Based at least partially on the results of the anomaly detection, determine the target or recommended endoscope withdrawal parameter value for each of the individual categories. An endoscope system according to any one of claims 1 to 6, configured to perform the following:

8. The endoscopic system according to claim 7, wherein the abnormality detection includes recognizing one or more of the presence, type, size, shape, location, or amount of pathological tissue or occluded mucosa.

9. The endoscope system according to claim 7 or 8, wherein the control circuit is configured to detect the one or more anomalies using a first trained machine learning (ML) model, the first ML model being trained to establish a correspondence between (i) an endoscope image or video stream, or features extracted therefrom, and (ii) one or more anomaly characteristics.

10. The endoscope system according to any one of claims 7 to 9, wherein the control circuit is configured to determine target or recommended endoscope withdrawal parameter values ​​for each of the individual segments by applying the results of anomaly detection to a second trained machine learning (ML) model, the second ML model being trained to establish a correspondence between (i) one or more anomaly characteristics and (ii) target or recommended endoscope withdrawal parameter values.

11. The aforementioned control circuit is Determining an anomaly score based on the type, size, shape, location, or quantity of the anomaly, By applying the determined abnormality score to the second trained ML model, the target or recommended endoscopy withdrawal parameter values ​​specific to each of the individual categories are determined. The endoscopic system according to claim 10, configured to perform the following:

12. The aforementioned control circuit is Images or video streams and clinical data from previous endoscopic procedures, Patient information and medical history, or, Pre-treatment imaging research data The endoscopic system according to any one of claims 1 to 11, further configured to use one or more of the following to generate the endoscopic withdrawal plan.

13. The endoscope system according to any one of claims 1 to 12, wherein the control circuit is configured to display a graph of the endoscope withdrawal plan on a user interface, the graph display includes a depiction of individual segments of the biological structure, each coded in color or grayscale to indicate endoscope withdrawal parameter values ​​specific to each target or recommended segment.

14. The aforementioned control circuit is Determining the location of the endoscope in one of the individual segments of the biological structure in substantially real time, based at least partially on the aforementioned endoscopic image features or the aforementioned endoscopic video features, The location of the endoscope is registered in a pre-generated template of the biological structure, The location of the endoscope in one of the individual sections is displayed on the user interface so as to be superimposed on the graph display of the endoscope withdrawal plan. The endoscopic system according to claim 13, configured to perform the following:

15. The aforementioned control circuit is The method involves measuring the endoscope withdrawal speed in one of the individual sections of the biological structure, When the measured endoscope withdrawal speed deviates by a specific difference from the target-specific endoscope WSL for one of the individual sections, an alarm is generated to the user, and a recommendation is provided to adjust the withdrawal speed to substantially match the target-specific endoscope WSL. The endoscopic system according to claim 2, configured to perform the following:

16. The robot system is configured to withdraw the endoscope using a robot, and the control circuit is: The method involves measuring the endoscope withdrawal speed in one of the individual sections of the biological structure, When the measured endoscope withdrawal speed deviates by a specific difference from the target-specific endoscope WSL for one of the individual divisions, a control signal is generated to the robot system in order to automatically adjust the withdrawal speed to substantially match the target-specific endoscope WSL. The endoscopic system according to claim 2 or 15, configured to perform the following:

17. The aforementioned control circuit is Identifying abnormal classifications from the individual classifications based at least partially on the aforementioned endoscopic image features or endoscopic video features, During manual or robotic withdrawal of the endoscope, before the endoscope reaches the identified abnormality section, a visual indication of the abnormality section is displayed on the user interface, and a warning is generated to the user to withdraw the endoscope at a speed lower than the target section-specific endoscope WSL for the identified abnormality section. An endoscope system according to any one of claims 2, 15, or 16, configured to perform the following:

18. A method for planning the withdrawal of an endoscope from the patient's biological structure during endoscopic procedures, The steps include obtaining images or video streams of individual segments of the biological structure during the insertion phase of the endoscopic procedure using an imaging system associated with the endoscope, The steps include analyzing the obtained image or video stream in order to generate endoscopic image features or endoscopic video features for each of the individual divisions of the biological structure, The steps of generating an endoscopic withdrawal plan for each of the individual segments of the biological structure, including target or recommended segment-specific endoscopic withdrawal parameter values, based at least partially on the endoscopic image features or the endoscopic video features, To facilitate manual or robotic withdrawal of the endoscope, the steps include providing the user or robotic system with the endoscope withdrawal plan, and Methods that include...

19. The method according to claim 18, wherein the target or recommended segment-specific endoscopic withdrawal parameter values ​​include a target segment-specific endoscopic withdrawal speed limit (WSL) or a target segment-specific endoscopic withdrawal time limit (WTL) for each of the individual segments of the biological structure.

20. The method according to claim 18 or 19, comprising the step of determining endoscopic withdrawal parameters specific to a first target segment for a first segment of the biological structure, using at least the endoscopic image features or endoscopic video features generated from the image or video stream obtained before the endoscope is withdrawn beyond the first segment during the manual or robotic withdrawal of the endoscope.

21. The method according to claim 20, wherein the image or video stream obtained before the endoscope reaches the first section includes an image or video stream obtained during the insertion of the endoscope into the biological structure.

22. The step of performing anomaly detection, which includes detecting one or more anomalies in each of the individual sections based at least partially on the endoscopic image features or the endoscopic video features, A step of determining, at least in part, the target or recommended endoscope withdrawal parameter value for each of the individual categories based on the results of the anomaly detection; The method according to any one of claims 18 to 21.

23. The method according to claim 22, wherein the anomaly detection comprises using a first trained machine learning (ML) model, the first ML model being trained to establish a correspondence between (i) an endoscope image or video stream, or features extracted therefrom, and (ii) one or more anomaly characteristics.

24. The method according to claim 22 or 23, wherein the step of determining a target or recommended endoscope withdrawal parameter value for each of the individual categories comprises applying the results of anomaly detection to a second trained machine learning (ML) model, the second ML model being trained to establish a correspondence between (i) one or more anomaly characteristics and (ii) a target or recommended endoscope withdrawal parameter value.

25. The step of generating the endoscope withdrawal plan is: Images or video streams and clinical data from previous endoscopic procedures, Patient information and medical history, or, Pre-treatment imaging research data The method according to any one of claims 18 to 24, further comprising using one or more of the following.

26. The method according to any one of claims 18 to 25, further comprising the step of displaying a graph of the endoscope withdrawal plan on a user interface, wherein the graph includes a depiction of individual segments of the biological structure, each coded in color or grayscale to indicate endoscope withdrawal parameter values ​​specific to each target or recommended segment.

27. The steps include determining the location of the endoscope in virtually real time, The steps include registering the location of the endoscope in a pre-generated template of the biological structure, The steps include displaying the location of the endoscope in one of the individual sections on the user interface so as to be superimposed on the graph display of the endoscope withdrawal plan, and The method according to claim 26, further comprising:

28. A step of measuring the endoscope withdrawal speed in one of the individual sections of the biological structure, When the measured endoscope withdrawal speed deviates by a specific difference from the target-section-specific endoscope WSL for one of the individual sections, the steps include: generating an alarm to the user and providing a recommendation to adjust the endoscope withdrawal speed to substantially match the target-section-specific endoscope WSL; The method according to claim 19, further comprising:

29. A step of measuring the endoscope withdrawal speed in one of the individual sections of the biological structure, When the measured endoscope withdrawal speed deviates by a specific difference from the target-specific endoscope WSL for one of the individual divisions, the robot system is given the step of generating a control signal to automatically adjust the withdrawal speed to substantially match the target-specific endoscope WSL. The method according to claim 19 or 28, further comprising:

30. A step of identifying an abnormality from the individual classification based at least partially on the endoscopic image features or the endoscopic video features, During manual or robotic withdrawal of the endoscope, before the endoscope reaches the identified abnormality section, a step of displaying a visual indication of the abnormality section on the user interface and generating a warning to the user to withdraw the endoscope at a speed lower than the target section-specific endoscope WSL for the identified abnormality section. The method according to any one of claims 19, 28, or 29, further comprising:

Images