Tracking during endoscopic medical procedures

Abstract

A system for determining one or more procedure locations during an endoscopic medical procedure may comprise processing circuitry and memory with instructions stored thereon that when performed by the processing circuitry cause the processing circuitry to receive an input corresponding to the endoscopic medical procedure. The input may include an endoscopic video stream, a patient history, one or more organizational guidelines, or a current position of the endoscope in the anatomy of the patient. The processing circuitry may determine, based on the input one or more procedure locations in association with a map of a portion of the anatomy of the patient and display the map on a graphical user interface. The processing circuitry may further display graphical elements that indicate the one or more procedure locations on the displayed map and dynamically update the graphical elements during the endoscopic medical procedure to indicate whether the one or more procedure locations correspond to one of a completed or uncompleted procedure there.

Description

TRACKING DURING ENDOSCOPIC MEDICAL PROCEDURESPRIORITY CLAIM

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 570,917, filed March 28, 2024, the contents of which are incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure relates to a system to identify and track procedure locations (e.g., biopsy locations) during an endoscopic medical procedure.BACKGROUND

[0003] Therapeutic and diagnostic endoscopic medical procedures, such as those performed in the Gastrointestinal (GI) Tract often require performing procedures such as biopsies. Various techniques and tools are used to aid endoscopists in these procedures. For example, endoscopic ultrasound combined with fine-needle aspiration is a method in which ultrasound is used to guide a biopsy needle to a desired biopsy location. In other examples, an endoscope can be equipped with a position sensor to provide information about the location of the tip of the endoscope within the body.SUMMARY

[0004] A system for determining one or more procedure locations during an endoscopic medical procedure may provide immediate guidance (e.g., in realtime or near real-time) based on analysis of one or more inputs. The inputs may include endoscopic video (from an imaging sensor or camera attached to or coupled to the endoscope), patient history, such as information regarding prior medical procedures or pre-existing conditions of the patient, best practices or other guidelines, such as medical standards from a medical board, regulations from a local jurisdiction, or the like. The system may utilize one or more artificial intelligence (Al) or machine-learning (ML) algorithms or a similar deep learning model to analyze inputs such as a current endoscopic video stream or a current endoscope position (e.g., determined using electromagnetic orradiofrequency tracking, Fiber Bragg Grating (FBG) shape sensing elements, or other surgical navigation technologies) in the body of a patient (e.g., the intestine, the colon, an organ, or the like). The system may also utilize a non-AI or non-ML deterministic algorithm or process. Additionally, or alternatively, the system may utilize a hardware-based feedback loop or feedback control. In an example, the system may utilize the Al or ML algorithm(s) in conjunction with the deterministic algorithm(s) or hardware-based feedback to perform the operations discussed herein.

[0005] A system for determining one or more procedure locations during an endoscopic medical procedure may comprise processing circuitry to receive an input (e.g., an endoscopic video stream) corresponding to the endoscopic medical procedure and generate, based on the input, a map of a portion of an anatomy of a patient. The system may include a scope guide or a tracking member (e.g., an electromagnetic or radiofrequency tracking device, or the like) to ensure accurate tracking of the scope as it is guided through the anatomy of the patient. The processing circuitry may cause a map of the patient's anatomy to be generated and displayed on a graphical user interface. One or more procedure locations (e.g., biopsy locations) may be displayed on the map using information received from the input. The processing circuitry may indicate, on the graphical user interface, whether the one or more procedure locations corresponds to a completed or uncompleted procedure at the one or more locations. For example, as a biopsy is taken at a particular location, the map may be updated to show that a biopsy was taken there. Conversely, when the scope is withdrawn and a biopsy has not been taken at a particular location, the map may be updated to show that a biopsy was not taken at that location. The system may include local or cloudbased computing infrastructure for computing, generating, and displaying the map or comparing a current map against a previously generated map (e.g., a map generated during a prior endoscopic medical procedure).

[0006] The system may receive a single input, such as the endoscopic video stream, or multiple inputs, such as the video stream and one or more of a patient history (e.g., information from a patient's medical file), a current position of the endoscope in the anatomy, one or more organizational guidelines, or the like. In such an example, different inputs may be used to determine differentprocedure locations on the map of the patient's anatomy. For example, the system may indicate one or more procedure locations using a first input (e.g., the endoscopic video stream) and may indicate one or more additional procedure locations using a second input, such as the patient history. Use of patient history as an input may be especially useful for a patient dealing with a condition such as Inflammatory Bowel Disease (IBD) or similar conditions that require biopsies to be taken routinely (e.g., yearly) at particular locations in the anatomy. The system can not only identify locations to be biopsied at the outset based on the type of procedure being performed but can determine whether a change in the anatomy has occurred and update the biopsy plan in real-time based on the detected change. The changes in the anatomy may include whether a lesion has grown, a change in the amount of inflammation of the bowel, the presence of a tumor, or the like. In some implementations, the system may analyze an endoscopic video stream in real-time during an insertion phase to identify appropriate biopsy locations (e.g., suspected cancerous polyps or lesions) and populate a biopsy map with the identified biopsy locations to guide an endoscopist during withdrawal phase. For example, in the context of a colonoscopy, the system may deploy a computer-aided detection (CADe) algorithm and / or a computer-aided diagnosis (CADx) algorithm to analyze an endoscopic video stream while an endoscopist is advancing the endoscope from the anus to the cecum of the patient. Then, when the endoscopist reaches the cecum, the system may identify the cecum in the video stream (which may serve as an indication that the withdrawal phase is about to begin) and in response to identifying the cecum display a biopsy map as a graphical representation of the colon populated with the identified biopsy locations. Additionally, or alternatively, the biopsy map may be populated with standard recommended biopsy sites corresponding to patient-specific disease indications (e.g., patient medical history data).

[0007] Thus, the system presently disclosed can track the endoscopy in real-time, offering immediate guidance based on the video analysis, patient history, and best practice guidelines. Such a system may not only enhance or increase the precision and reliability of biopsy sampling but may aid in training and standardizing procedures and help to optimize the biopsy process and reducemiss rates for biopsy locations. This may help to ensure comprehensive, accurate sampling and adherence to established recommendations while streaming the procedure for the medical personnel. More comprehensive and accurate the sampling during an endoscopic procedure may help medical personnel make better, more informed decisions regarding future medical procedures. For example, results from biopsies during one procedure may determine how much intestine or bowel needs to be cut in a subsequent surgery, and / or whether biopsy samples should be collected from the same locations again during subsequent medical procedures.

[0008] While the focus of the present disclosure is on biopsies performed in the gastrointestinal (GI) tract, the systems and methods discussed herein may be extended to other types of procedures in other areas of a body of a patient. Therefore, the term "biopsy" and "procedure" may be used interchangeably herein. The term "procedure" refers to any type of medical procedure or operation performed internally using any type of medical scope capable of being inserted into the body of a patient. The term patient refers to any human or animal subject undergoing the medical procedure, and the term anatomy refers to any interior portion of the body of the patient.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.

[0010] FIG. 1 A illustrates an example flow diagram for biopsy tracking.

[0011] FIG. IB illustrates an example of a pre-withdrawal target biopsy map.

[0012] FIG. 1C illustrates an example of a post-withdrawal target biopsy map.

[0013] FIG. 2 illustrates an example of a flow diagram for generation of a pre-withdrawal biopsy plan during endoscopic insertion.

[0014] FIG. 3 illustrates an example of a flow diagram for updating the biopsy plan generated in FIG. 2 during endoscopic withdrawal.

[0015] FIG. 4 is an example of a method for determining one or more procedure locations during an endoscopic medical procedure.

[0016] FIG. 5 illustrates an example of a machine upon which one or more embodiments may be implemented.

[0017] FIG. 6 illustrates an example of a schematic diagram of an exemplary computer-based clinical decision support system (CDSS).DETAILED DESCRIPTION

[0018] Endoscopic procedures such as the biopsy techniques described above involve several challenges that can limit their efficacy and precision. Specifically, many techniques predominantly rely on operator or user-dependent practices. The reliance on human expertise can introduce variability in outcomes and can result in suboptimal procedures such as suboptimal biopsy sampling (including failing to collect biopsy samples at all appropriate locations). The risk of suboptimal procedures is especially high in complex cases performed by less experienced operators. Additionally, many advanced endoscopic techniques require specialized training, which can be a barrier to widespread adoption of such techniques. Examples of advanced endoscopic techniques include endoscopic ultrasound procedures, endoscopic submucosal dissection, confocal laser endomicroscopy, endoscopic retrograde cholangiopancreatography (ERCP), or the like. The lack of real-time feedback, guidance, and adaptability based on conditions that evolve during the procedure can complicate the procedure.

[0019] Thus, the outcome and quality of procedures such as biopsies largely depend on the operator's expertise. This can lead to inconsistencies in procedures across different practitioners and health care training. Furthermore, advanced endoscopic techniques can demand rigorous training that can deter medical professionals from adopting newer, more effective techniques. Furthermore, the lack of real-time feedback or guidance during an endoscopic procedure can lead to missed biopsy sites (e.g., missed lesions) or non- compliance with medical guidelines or standards. Additionally, even whenbiopsy procedures include a pre-planning component, this plan may need to be adjusted based on real-time findings or factors specific to the patient including, but not limited to, CADe and / or CADx outputs generated during the medical procedure in real-time (e.g., during an insertion and / or withdrawal phase).

[0020] A system for determining procedure locations (e.g., biopsy locations) during an endoscopic medical procedure is disclosed herein. The system may be designed for real-time scene analysis during endoscopic procedures, such as procedures in the upper and lower gastrointestinal (GI) endoscopy. The system may assist endoscopists or other medical professionals in ensuring compliance with guidelines during GI biopsies (e.g., the National Health Service (NHS) guidelines highlighting the importance of accuracy in biopsy procedures, American Medical Association (AMA) guidelines, or the like). Various guidelines call for a different number of biopsies to be performed depending on the type of procedure being performed. For example, for procedures in the Upper GI Tract, the National Health Service Endoscopy (NHSE) guidelines have recommendations based on particular diagnoses, such as:

[0021] Oesophagitis: Depending on the suspected cause, the recommendation varies from at least six biopsies from ulcers to no routine biopsies for certain conditions.

[0022] Barrett’s Oesophagus: Biopsies should be taken from all visible abnormalities and routine four-quadrant biopsies every 2 centimeters (cm) within the Barrett’s segment.

[0023] Upper GI Cancer & Early Neoplasia: Recommendations include taking at least two biopsies for suspected advanced cancer.

[0024] Dyspepsia & Gastritis: Rapid urease testing is suggested for Helicobacter pylori infection, and further biopsies based on the infection.

[0025] Gastric Polyps: Specific recommendations are based on the size and type of the polyp.

[0026] Coeliac disease: At least 6 biopsies from the duodenum, including samples from the bulb.

[0027] Similarly, for procedures in the Hepatopancreatobiliary Tract, the NHS guidelines have recommendations based on particular diagnoses, such as:

[0028] Solid pancreatic mass: Use of fine-needle aspiration and fine- needle biopsy is advised.

[0029] Indeterminate Biliary stricture: Different diagnostic methods are preferred based on the structure's location.

[0030] Preparation of EUS-FNA material: Specific methods of handling the material after biopsy are recommended.

[0031] For procedures in the Lower GI Tract, the NHS biopsy guidelines include:

[0032] Clinical / endoscopic signs of colitis: Segmental biopsies are advised from various sections of the colon.

[0033] Microscopic colitis: Recommendations include 2 biopsies from right hemi-colon and 2 from left hemi-colon.

[0034] IBD Surveillance: Various protocols depending on the patient's history and presenting condition.

[0035] Pouch surveillance: Specific recommendations based on the pouch's anatomy and condition.

[0036] Ulcerative Colitis: Depending on the severity of inflammation, biopsies from affected areas are advised.

[0037] Crohn’s Disease: Biopsies should be taken if the mucosa appears abnormal or inflamed.

[0038] Premalignant Colorectal Polyps: Recommendations include either excision or biopsy based on the polyp's assessment.

[0039] Potential malignant lesion: Procedures vary based on the suspicion of malignancy.

[0040] These example guidelines and recommendations underscore the importance of precision and personalization in biopsy procedures. The present inventor has recognized, among other things, a need for patient-specific biopsy tracking using imaging analysis and endoscopic tracking.

[0041] FIG. 1 A illustrates an example flow diagram 100 for biopsy tracking. At 102, upon insertion of an endoscope into a portion of anatomy (e.g., the GI tract) of a human or animal subject (hereinafter referred to as "a patient" or "the patient"), a system for determining or recommending procedure locations (hereinafter "the system") may activate or otherwise turn on. The system maytrack the progress of the endoscope as the endoscope is moved, advanced, or the like through the upper GI tract and the lower GI tract. The endoscope may include or be coupled to one or more sensors such as an imaging sensor, an RF or other electromagnetic tracking sensor or device (or other surgical navigation device), or any combination of sensors capable of monitoring and tracking the progress of the scope through the anatomy of the patient.

[0042] At 104, upon reaching a certain portion of the patient's anatomy, such as the Cecum, the system may generate a biopsy plan. Additionally, or alternatively, the biopsy plan may be generated prior to reaching the certain portion of the patient’s anatomy (e.g., during an insertion phase) and / or may be updated after the endoscope is withdrawn away from the patient anatomy (e.g., during a withdrawal phase). Generating the biopsy plan may include generating a pre-withdrawal biopsy map 106. The pre-withdrawal biopsy map 106 may be generated on a graphical user interface (GUI) such as a monitor or screen located in, or located remotely from, the procedure room or operating room. As discussed in more detail below, the pre-withdrawal biopsy map 106 may include one or more biopsy locations in the GI tract which may be recommended by the system based on one or more inputs such as the location of the endoscope in the anatomy of the patient, information from the patient history or chart such as the medical history of the patient (prior procedures, diagnoses, or the like), information about the type of procedure to be performed, guidelines from a medical organization, jurisdictional guidelines or requirements, or the like. In an example, the pre-withdrawal biopsy map 106 may be updated based on conditions that arise during the procedure such as observations and analysis made of the endoscopic video feed at 108, detection of possible target tissue (e.g., tumors or the like) at 110. Based on any observations or detections at 108 or 110, or as diagnoses are made and treatment such as biopsies are performed, at 112, an updated biopsy map 114 may be generated. The updated biopsy map 114 may indicate procedure locations where a biopsy is confirmed to have been made, or new procedure locations where a biopsy is recommended based on any observations, detections, and / or diagnoses made at 108, 110, or 112. As the scope is removed or withdrawn from the anatomy, the updated biopsy map 114A may also be updated to indicate locations at which a procedure wasrecommended but not performed. When the scope is removed from the patient at 116, a post- withdrawal target biopsy map 118 may be generated and displayed for the physician or user on the GUI.

[0043] In some implementations, the biopsy plan may be populated with biopsy locations identified by the system during the insertion and / or withdrawal phase by analyzing an endoscopic video stream in real-time with one or more CADe and / or CADx modules. In this context, “CADe” is an acronym for Computer Aided Detection, which is an Al model to identify abnormal tissue; whereas “CADx” is an acronym for Computer Aided Diagnosis, which is an Al model configured to classify detected abnormal tissue. Computer-Aided Detection (CADe) systems are designed to locate and highlight potential abnormalities or lesions in medical images, such as X-rays, mammograms, or CT scans. Their primary purpose is to assist radiologists by flagging suspicious areas that might otherwise be overlooked. However, CADe systems do not provide detailed characterization of the lesions; they focus on detection. Computer-Aided Diagnosis (CADx) systems go beyond detection and aim to characterize the detected lesions. These systems provide information about the nature of the abnormality, such as whether it is benign or malignant.

[0044] In some implementations, the system may analyze an endoscopic video stream in real-time during an insertion phase to identify appropriate biopsy locations (e.g., suspected cancerous polyps or lesions) and populate a biopsy map with the identified biopsy locations to guide an endoscopist during withdrawal phase. For example, in the context of a colonoscopy, the system may deploy a CADe module and / or CADx algorithm to analyze an endoscopic video stream while an endoscopist is advancing the endoscope from the anus to the cecum of the patient. These finds of the system obtained via CADe and / or CADx during insertion may be used to graphically populate biopsy map 106 with identified biopsy locations. Additionally, or alternatively, the biopsy map may be populated with standard recommended biopsy sites corresponding to patientspecific disease indications (e.g., patient medical history data).

[0045] FIG. IB illustrates an example of the pre-withdrawal target biopsy map 106 discussed above in FIG. 1A. As illustrated in FIG. IB, the prewithdrawal biopsy map 106 may include one or more recommended procedurelocations 120. The one or more recommended procedure locations 120 may be graphical elements generated on the pre-withdrawal biopsy map 106 as it is displayed on the GUI based on the inputs and may correspond to locations within the anatomy of the patient in which the system determines a need for or recommends that a procedure (e.g., biopsy and / or resection) be performed within the anatomy. The pre-withdrawal biopsy map 106 may be generated during an insertion phase in which the endoscope navigates the anatomy of the patient (e.g., the colon) while gathering real-time video data and while initializing the system for procedure tracking (as discussed below). In an example, the GUI may play back the endoscopic video footage augmented with an interactive layer. The interactive layer may flag potential procedure areas. The potential procedure areas may be color-coded, or shaded differently, shaped differently, or the like, to indicate a level of urgency or priority, and these potential procedure areas may then be reflected on the pre-withdrawal biopsy map 106 displayed on the GUI.

[0046] FIG. 1C illustrates an example of the post-withdrawal target biopsy map 118 discussed above in FIG. 1 A. As biopsies are performed at the recommended locations during the medical procedure, the pre-withdrawal biopsy map 106 may be updated to show which of the one or more recommended procedure locations 120 were biopsied or not. For example, on the pre-withdrawal biopsy map 106 the one or more recommended procedure locations 120 may be displayed in an initial color (e.g., blue). When a biopsy is completed at a particular procedure location of the one or more recommended procedure locations 120, the pre-withdrawal biopsy map 106 may be updated to display a different color (e.g., green) at that particular location. Conversely, when a biopsy is not performed at a different particular procedure location of the one or more recommended procedure locations 120, the pre-withdrawal biopsy map 106 may be updated to display another different color (e.g., red) at that different particular procedure location.

[0047] When the medical procedure is complete and the scope is removed or withdrawn from the patient's anatomy, the post-withdrawal target biopsy map 118 may display completed procedure locations 122 and uncompleted procedure locations 124, indicating where biopsies were and were not taken during the procedure. In such an example, post-procedure, a user suchas an endoscopist may annotate the post-withdrawal target biopsy map 118 such as to add a note to the uncompleted procedure locations 124 to provide an explanation or reasoning for why a biopsy was not taken at a particular location. Additionally, or alternatively, the system may generate a summary for the postwithdrawal target biopsy map 118. The summary may be displayed on the GUI, such as on a split screen display, with the post-withdrawal target biopsy map 118 or may be generated and displayed separately from the post-withdrawal target biopsy map 118. The summary may include a chart enumerating the locations and exact coordinates of the completed procedure locations 122, images from the endoscopic video (e.g., still images), or any other data from the medical procedure.

[0048] Thus, the system may include a procedure plan generation phase, a guidance phase, and a summarization phase. The procedure plan generation phase may include a patient specific biopsy plan generated based on one or more of prior patient history, such as prior procedures the patient has undergone, guidelines or recommendations from medical societies or associations, such as the NHSE London Gastrointestinal Endoscopy Biopsy Recommendations, or analysis of the endoscopic video taken during insertion into the anatomy of the patient. The guidance phase may occur while the endoscope is being withdrawn from the anatomy (such as the intestines, colon, or the like) and may include displaying the biopsy plan on the GUI and adjusting the plan in real-time based on an analysis of scenes from the endoscopic video, electromagnetic tracking of the endoscope, or the like. The guidance phase may assist the endoscopist in finding the biopsy locations and may alert or warn the endoscopist if a recommended biopsy location is missed. The alert may be a visual warning, such as an indication on the pre-withdrawal biopsy map 106. The indication may include causing the one or more recommended procedure locations 120 to blink or may include a pop up message indicating that a location has been missed and the coordinates of the missed location. In another example, the alert may be an audible alert such as a beep or other similar sound as the scope passes a procedure location without a biopsy being taken. In another example, the alert may be a haptic alert such as causing a handle of the endoscope to vibrate or pulse.

[0049] In some embodiments, the recommended procedure locations 120, the completed procedure locations 122, and / or the uncompleted procedure locations 124 may be graphically represented in distinctive manners. For example, these respective procedure locations may be represented as different colors as described above with recommended procedure locations 120 being a first color, the completed procedure locations 122 being a second color, and / or the uncompleted procedure locations 124 being a third color. Additionally, or alternately, a particular status (e.g., recommended, completed, uncompleted) of these respective procedure locations may be identifiable via non-color based graphical characteristics. For example, recommended procedure locations 120 may be identified as a solid dot, the completed procedure locations 122 may be identified via the solid dot being replaced with or updated to include a check mark, while the uncompleted procedure locations 124 may be identified via the solid dot being replaced with or updated to include an “X” mark.

[0050] FIG. 2 illustrates an example of a flow diagram 200 for generation of a pre-withdrawal biopsy plan during endoscopic insertion. The insertion phase may include navigating the scope through a portion of the anatomy of the patient, such as navigating a colonoscope through the intestine and colon of the patient. Initially, the biopsy tracking of the system may be set to insertion mode. The setting may be made by a manual input by an endoscopist or may trigger automatically upon insertion of the scope. Once the system is set to the insertion mode, as the scope advances through the anatomy of the patient, real-time endoscopic video 202 can be recorded and displayed. Simultaneously, or substantially simultaneously with the advancement of the scope, the system can monitor the location of the scope (e.g., using electromagnetic tracking circuitry, a tracking module or component) and mapping the location of the scope against a standardized colon template. Such a process may utilize electromagnetic tracking technology (e.g., a scope guide) or utilize detection of visual landmarks detected from the real-time endoscopic video 202, such as the anus, various colon sections, or the terminal ileum. In an example, the mapping process may use a combination of electromagnetic or RF tracking and detection of visual landmarks. The landmark identification may leverage various machinelearning techniques and utilize one or more deep learning models. The deeplearning models may include a sub-system for determining anomalous sections 204 and a sub-system for determining scope position during insertion 206.

[0051] On reaching a location in the anatomy such as the caecum, the system may wait for a manual input or use a detection algorithm to trigger the biopsy planning system 208 (or subsystem). The recorded insertion video (from the real-time endoscopic video 202) may be segmented and analyzed to identify anomalous sections of the anatomy using the sub-system for determining anomalous sections 204. The anomalies may include pathological tissue formations, obstructions in the mucosa, or the like. In an example, the system may utilize an Al model tailored for endoscopic video analysis. Several Al techniques ranging from Convolutional Neural Networks (CNN), Conditional Random Fields, or the like may be employed. The CNNs may be specifically tailored for real-time analysis using extensive training datasets from various gastrointestinal cases. Important features such as color disparities, tissue texture, and morphological changes may be scrutinized to detect abnormalities such as ulcers, tumors, polyps, or the like. In an example, an added layer of temporal video analysis may be used to track dynamic changes and patterns throughout the insertion of the scope, which may aid in capturing comprehensive data.

[0052] The image analysis component or circuitry may seamlessly collaborate with the electromagnetic tracking component or circuitry. This collaboration may help ensure that the visually identified area(s) of interest are a spatially accurate representation of the patient's anatomy. In an example, data from the sub-system for determining anomalous sections 204 and the sub-system for determining scope position during insertion 206 may be combined with additional data 210 and entered into the biopsy planning system 208. The additional data 210 may include prior patient indicators or patient history. The prior patient indicators may include electronic health records of the patient with information about the patient's surgical history, previous biopsy results (e.g., indicating locations in the patient's anatomy that have been previously biopsied or locations that have been flagged by a physician to be subsequently checked), known risk factors, or any similar relevant information regarding the patient's medical history (e.g., allergies, reaction to anesthesia, or the like). The additional data 210 may also include recommendations or guidelines from medicalsocieties or organizations which may be periodically updated to reflect the most recent guidelines.

[0053] The biopsy planning system 208 may compile and assimilate the information or analysis from the sub-system for determining anomalous sections 204, the sub-system for determining scope position during insertion 206, and the additional data 210 to generate the pre-withdrawal biopsy map 106 or another visual representation of the biopsy plan. Once the plan is generated, the endoscopist may confirm, reject, or modify the plan, as necessary or desired.

[0054] FIG. 3 illustrates an example of a flow diagram 300 for updating the biopsy plan generated in FIG. 2 during endoscopic withdrawal. A withdrawal phase may include retracting the scope while ensuring thorough mucosal inspection and readying the system for a biopsy or intervention based on realtime interventions. During the withdrawal phase, the biopsy tracking application may be triggered or activated (if inactive) as soon as withdrawal of the scope commences. Having the tracking application active during withdrawal of the scope can help ensure accurate tracking of potential biopsy sites during retraction.

[0055] During retraction of the scope, a withdrawal video stream 302 may be recorded or monitored to record video segments of the withdrawal of the scope. The withdrawal process may be evaluated using a variety of applications or methods, for example magnetic tracking, optical flow video processing, or one or more Al techniques, Al models, systems or the like. The withdrawal phase may include a workflow phase. The workflow phase may include the withdrawal of the scope, cleaning (e.g., tissue), biopsy, injection, or other treatment techniques. The workflow phase, and action items to be taken during the workflow phase, may be determined using a workflow sub-system 304, which may include an Al or ML learning model. Similar to the insertion phase utilizing the sub-system for determining scope position during insertion 206, the withdrawal phase may include a sub-system for determining scope position during withdrawal 306. The sub-system for determining scope position during withdrawal 306 may register or track the scope's location allowing the location of the scope to be aligned with a standard colon template and may leverageelectromagnetic or radiofrequency tracking or visual landmark detection using deep learning and machine-learning techniques.

[0056] As items in the workflow are performed, the pre-withdrawal biopsy map 106 may be updated to show the progress of the procedure locations determined and proposed on the pre-withdrawal biopsy map 106. For example, when a procedure is performed, the proposed location indication may change color on the pre-withdrawal biopsy map 106, such as from blue (indicating a proposed location) to green (indicating a procedure has been completed). Similarly, if a procedure is not performed at a proposed location, its location on the pre-withdrawal biopsy map 106 may change color from blue to red (indicating a skipped or uncompleted procedure). Thus, the system can generate an updated biopsy map 312.

[0057] Data or information from the workflow sub-system 304, the updated biopsy map 312, or the sub-system for determining scope position during withdrawal 306 may be transmitted, entered, fed, or the like, to a recommendation sub-system 308 (e.g., recommendation engine). The recommendation sub-system 308 may generate an alert when a user is approaching a potential procedure zone as the scope is being withdrawn. The alert may be a haptic alert, a visual alert, an audible alert, or a combination thereof. As discussed above, when a procedure action (e.g., a biopsy action) is detected, or when the user acknowledges the alert, the pre-withdrawal biopsy map 106 may be updated to reflect this activity. Conversely, when a user dismisses an alert (e.g., cancels the alert without performing a procedure action) the decision may be recorded to inform future analysis, to train an Al or ML model, to be entered into a feedback loop, or the like. Along with the alerts, the system may integrate auditory cues, tactile feedback, or the like which may offer the endoscopist or other user a more immersive experience and clearer guidance and feedback. For example, a feedback loop may be utilized allowing the user to rate the accuracy of usefulness of the alerts, prompts, or feedback indications. The data from this feedback loop may be used in refining and training the system to help ensure better accuracy in the future.

[0058] Upon completion of the overall medical procedure, for example, a detection of the scope at the rectum, the recommendation sub-system 308 maygenerate the post-withdrawal target biopsy map 118. The recommendation subsystem 308 may, along with generating and displaying the post-withdrawal target biopsy map 118, also generate post-procedure analytics. The analytics data may include a summary of the procedure, a chart enumerating every biopsy location, the exact coordinates of the biopsy location, an importance ranking of each biopsy, and a still image from the video at each biopsy location. The chart may be displayed on the GUI discussed above (e.g., a GUI in the procedure room) or a different GUI and may allow the physician to review the chart, make annotations or corrections to the chart, or the like.

[0059] Thus, the system may operate on an on-premises computer device or stream data to a cloud-based data center for computation and relay of the postwithdrawal target biopsy map 118 and analytic chart which can be accessed by a network-connected computing device. Additionally, or alternatively, the system may operate in a closed loop with a feedback loop for continuous learning. The continuous learning may include feedback from the endoscopist, such as satisfaction with recommendations made by the system during the procedure via a "like" button, or the like. Alternatively, to avoid a cold start, the system may operate in a shadow mode when a procedure is performed by an experienced endoscopist to monitor their biopsy actions and correct system output through reinforcement feedback. While the procedure discussed above involved a colonoscopy, the system may be extended to other types of procedures such as anoscopy, arthroscopy, bronchoscopy, colposcopy, cystoscopy, esophagoscopy, gastroscopy, laparoscopy, laryngoscopy, neuroendoscopy, proctoscopy, sigmoidoscopy, thoracoscopy, or the like.

[0060] FIG. 4 illustrates an example of a method for determining one or more procedure locations during an endoscopic medical procedure. The method 400 can include or comprise a number of Operations or Steps. The Operations described herein are examples only, and the method can omit one or more of the listed Operations, can repeat Operations, can include other Operations, or can execute the Operations concurrently, substantially simultaneously, or in another order, as appropriate or desired.

[0061] At 402, the method 400 may include receiving an input corresponding to an endoscopic medical procedure. The input may include oneor more of an endoscopic video stream, a current position of the endoscope in the anatomy of a patient, an organizational or jurisdictional guideline or requirement, patient demographics or patient history. The history may include information specific to the particular patient such as medical history, allergies, age, weight, or the like. The demographics may include information regarding people of the same sex, age, weight as the patient, or have the same or similar medical conditions, as the patient. The organizational guidelines may include guidelines from medical organizations or associations. The jurisdictional guidelines may include regulations from a governmental regulatory agency or authority.

[0062] Using the one or more inputs, one or more procedure locations within the anatomy of the patient may be determined. At 404, a map of a portion of the anatomy of the patient may be generated and at 406, the map may be displayed on a graphical user interface. The graphical user interface may be displayed on a monitor, the screen of a mobile device, or the like. At 408 one or more procedure locations may be determined using or based at least in part on the input and indicated on the displayed map. For example one or more procedure locations may be determined based at least in part on analysis of an endoscopic video stream from an imaging device such as a camera located at or near the tip of the endoscope. The analysis of the endoscopic video stream may include image segmentation analysis by a trained learning model. Additionally, the one or more procedure locations may be determined using one of the other inputs alone or in conjunction with analysis of the endoscopic video stream. Stated differently, when multiple procedure locations are determined, one procedure location may be determined using analysis of the endoscopic video stream (e.g., by way of CADe and / or CADx analysis) and another may be determined using a different input such as patient history. Yet another procedure location may be determined using a combination of the video stream or the patient history (or any combination of any of the inputs). On the generated map of the anatomy the recommended procedure locations may be marked with a dot, an "x" or any suitable marker. The procedure locations may include a pathology location or a non-pathology location. Additional procedure locations (pathologyor non-pathology) may be selected by a trained learning model based at least in part on analysis of the input.

[0063] As diagnostic or treatment procedures are performed the procedure locations may be updated. 410 may include indicating whether the one or more procedure locations correspond to a completed or uncompleted procedure. In an example in which a plurality of procedure locations are determined and indicated on the map at 408, when a procedure is performed at one of the locations, an indication may be made on the map. For example, a marker indicating the procedure location may change color, such as from blue (indicating a procedure location) to green (indicating that a procedure has been performed at the location). Additionally, or alternatively the marker indicating a procedure location may change shape or form, such as from a circle to a checkmark. Similarly, when a procedure is not performed at a procedure location, the color or shape of the marker may change from blue to red, from a circle to an x, or the like. When a procedure location is skipped or missed, the method 400 may include generating an alert. The alert may include a visual alert, an audible alert, a haptic alert, or a combination thereof.

[0064] When the endoscopic medical procedure is complete, at 412 a recommendation may be generated and the patient file may be updated based on the recommendation. The recommendation may include a recommendation for a subsequent medical procedure, such as a surgical procedure, a recommendation to have the patient follow up with a specialist, recommended biopsy locations during the next endoscopic procedure, or the like.

[0065] FIG. 5 is a block diagram of an example of an apparatus, device, or machine 500 upon which any one or more of the techniques (e.g., methodologies) discussed herein may perform. In alternative embodiments, the machine 500 may operate as a standalone device or may be connected (e.g., networked) to other machines. The machine 500 may be connected to the endoscope, such as connected to the medical device via the VO connection 5) and / or the accessory port 6). In a networked deployment, the machine 500 may operate in the capacity of a server machine, a client machine, or both in serverclient network environments. In an example, the machine 500 may act as a peer machine in peer-to-peer (P2P) (or other distributed) network environment. Themachine 500 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations.

[0066] Examples, as described herein, may include, or may operate by, logic or a number of components, or mechanisms. Circuit sets are a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuit set membership may be flexible over time and underlying hardware variability. Circuit sets include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuit set may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuit set may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer readable medium physically modified (e.g., magnetically, electrically, moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuit set in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer readable medium is communicatively coupled to the other components of the circuit set member when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuit set. For example, under operation, execution units may be used in a first circuit of a first circuit set at one point in time and reused by a second circuit in the first circuit set, or by a third circuit in a second circuit set at a different time.

[0067] Machine (e.g., computer system) 500 may include a hardware processor 502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, field programmable gate array (FPGA), or any combination thereof), a main memory 504 and a static memory 506, some or all of which may communicate with each other via an interlink (e.g., bus) 530. The machine 500 may further include a display unit 510, an alphanumeric input device 512 (e.g., a keyboard), and a user interface (UI) navigation device 514 (e.g., a mouse). In an example, the display unit 510, input device 512 and UI navigation device 514 may be a touch screen display. The machine 500 may additionally include a storage device (e.g., drive unit) 508, a signal generation device 518 (e.g., a speaker), a network interface device 520 connected to a network 526, and one or more sensors 516, such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor. The machine 500 may include an output controller 528, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

[0068] The storage device 508 may include a machine readable medium 522 on which is stored one or more sets of data structures or instructions 524 (e.g., software) embodying or used by any one or more of the techniques or functions described herein. The instructions 524 may also reside, completely or at least partially, within the main memory 504, within static memory 506, or within the hardware processor 502 during execution thereof by the machine 500. In an example, one or any combination of the hardware processor 502, the main memory 504, the static memory 506, or the storage device 508 may constitute machine readable media.

[0069] While the machine readable medium 522 is illustrated as a single medium, the term "machine readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store the one or more instructions 524. The term “machine readable medium” may include any medium that is capable of storing, encoding, or carrying instructions for execution by the machine 500 and that cause the machine 500 to perform any one or more of the techniques of thepresent disclosure, or that is capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine readable medium examples may include solid-state memories, and optical and magnetic media. In an example, a massed machine readable medium comprises a machine readable medium with a plurality of particles having invariant (e.g., rest) mass. Accordingly, massed machine-readable media are not transitory propagating signals. Specific examples of massed machine readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks.

[0070] FIG. 6 shows a schematic diagram of an exemplary computer- based clinical decision support system (CDSS) 600 that is configured to determine the procedure locations based on the analysis of the inputs and generate a recommendation. In various embodiments, the CDSS 600 includes an input interface 602 through which information about a patient and / or the medical procedure the patient is scheduled for, which are specific to a patient, are provided as input features to an artificial intelligence (Al) model 604, a processor, such as processor 502, which performs an inference operation in which the information is applied to the Al model to generate the one or more changed laser settings, and a user interface (UI) through which one or more changed laser settings is communicated to a user, e.g., a clinician.

[0071] In some embodiments, the input interface 602 may be a direct data link between the CDSS 600 and one or more medical devices that generate at least some of the input features. For example, the input interface 602 may transmit information about the procedure and / or information from (e.g., a signal from) a sensor coupled to a medical device or scope directly to the CDSS 600 during a therapeutic and / or diagnostic medical procedure. Additionally, or alternatively, the input interface 602 may be a classical user interface that facilitates interaction between a user and the CDSS 600. For example, the input interface 602 may facilitate a user interface through which the user may manually enter information about the procedure that are specific to the patient.Additionally, or alternatively, the input interface 602 may provide the CDSS 600 with access to an electronic patient record from which one or more input features may be extracted. The electronic patient record may be stored in a retrieved from a database 606. In any of these cases, the input interface 602 can be configured to collect one or more of the following input features in association with a specific patient on or before a time at which the CDSS 600 is used to assess:

[0072] Information about the medical procedure;

[0073] Information about the medical device(s) to be used during the procedure;

[0074] Information from a video sensor 610;

[0075] Information from additional inputs 612;

[0076] Based on one or more of the above input features, the processor 502 can perform an inference operation using the Al model to generate an anatomy map, determine the procedure locations, and display the procedure locations on the anatomy map, and make recommendations for the current or a future medical procedure. . For example, input interface 602 may deliver the information about the procedure, the devices, organizational or jurisdictional guidelines or regulations, images from the endoscopic video, or the like into an input layer of the Al model which can propagate these input features through the Al model to an output layer. The Al model can provide a computer system the ability to perform tasks, without explicitly being programmed, by making inferences based on patterns found in the analysis of data. The Al model can explore the study and construction of algorithms (e.g., machine-learning algorithms) that may learn from existing data and make predictions about new data. Such algorithms operate by building an Al model from example training data in order to make data-driven predictions or decisions expressed as outputs or assessments.

[0077] Two modes for machine learning (ML) include supervised ML and unsupervised ML. Supervised ML may use prior knowledge (e.g., examples that correlate inputs to outputs or outcomes) to learn the relationships between the inputs and the outputs. The goal of supervised ML is to learn a function that, given some training data, best approximates the relationship between the training inputs and outputs so that the ML model can implement the same relationshipswhen given inputs to generate the corresponding outputs. Unsupervised ML includes the training of an ML algorithm using information that is neither classified nor labeled and allowing the algorithm to act on that information without guidance. Unsupervised ML may be useful in exploratory analysis because it can automatically identify structure in data.

[0078] Tasks for supervised ML include classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a score to the value of some input). Some examples of supervised-ML algorithms are Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural networks (NN), deep neural networks (DNN), matrix factorization, and Support Vector Machines (SVM).

[0079] Tasks for unsupervised ML include clustering, representation learning, and density estimation. Some examples of commonly used unsupervised-ML algorithms are K-means clustering, principal component analysis, and autoencoders. Another type of ML is federated learning (also known as collaborative learning) that trains an algorithm across multiple decentralized devices holding local data, without exchanging the data. This approach stands in contrast to traditional centralized machine-learning techniques where all the local datasets are uploaded to one server, as well as to more classical decentralized approaches which often assume that local data samples are identically distributed. Federated learning enables multiple actors to build a common, robust machine learning model without sharing data, thus allowing to address critical issues such as data privacy, data security, data access rights and access to heterogeneous data.

[0080] In some examples, the Al model may be trained continuously or periodically prior to performance of the inference operation by the processor 502. Then, during the inference operation, the patient specific input features provided to the Al model may be propagated from an input layer, through one or more hidden layers, and ultimately to an output layer that corresponds to the procedure locations or the recommendations on a map of a portion of the patient's anatomy and / or a recommendation for the current or a future medicalprocedure . For example, based on the inputs such as the information about the medical procedure, or other patient specific information, information about the medical device to be used during the procedure, or the like, one or more procedure locations can be propagated to the output layer such as to be displayed on a map of a portion of the patient's anatomy. Further, one or more graphical elements can be generated and displayed on the map indicating the one or more procedure locations. During and / or subsequent to the inference operation, the one or more graphical elements can be dynamically updated to indicate whether a procedure has been completed or remains uncompleted, at the particular location. During and / or after completion of the procedure recommendations and / or post-procedure analytics can be propagated to the output layer. A recommendation during the procedure can include a recommendation to return to a biopsy site from a prior procedure for follow-up analysis. A recommendation for a subsequent procedure can include a recommendation for a patient to follow up with a specialist, a recommendation for a type of subsequent procedure, or the like.ADDITIONAL NOTES & EXAMPLES

[0081] Example l is a system for determining one or more procedure locations during an endoscopic medical procedure, the system comprising: processing circuitry; and memory, including instructions stored thereon that when performed by the processing circuitry, cause the processing circuitry to: receive an input corresponding to the endoscopic medical procedure; determine, based the input, one or more procedure locations in association with a map of a portion of an anatomy of a patient; display the map on a graphical user interface; display graphical elements that indicate the one or more procedure locations on the displayed map; and dynamically update the graphical elements during the endoscopic medical procedure to indicate whether the one or more procedure locations corresponds to one of a completed or uncompleted procedure there.

[0082] In Example 2, the subject matter of Example 1 optionally includes subject matter wherein the input includes at least one of an endoscopicvideo stream, a patient history, a current position of an endoscope in the anatomy of the patient, or one or more organizational guidelines.

[0083] In Example 3, the subject matter of Example 2 optionally includes subject matter wherein the input includes the one or more organizational guidelines and wherein the processing circuitry is to: select the one or more organizational guidelines based at least in part on a type of the endoscopic medical procedure or a physical location at which the endoscopic medical procedure is performed.

[0084] In Example 4, the subject matter of Example 3 optionally includes subject matter wherein the input includes the endoscopic video stream, wherein the one or more procedure locations are determined based at least in part on the one or more organizational guidelines.

[0085] In Example 5, the subject matter of Example 4 optionally includes subject matter wherein analysis of the endoscopic video stream includes image segmentation analysis by employing a trained learning model, wherein the endoscopic video stream is analyzed in real-time during an endoscopic insertion phase to determine the one or more procedure locations, and wherein the trained learning model includes one or more of a computer-aided detection algorithm or a computer-aided diagnosis algorithm.

[0086] In Example 6, the subject matter of any one or more of Examples 4-5 optionally include subject matter wherein the instructions further cause the processing circuitry to: indicate one or more additional procedure locations on the displayed map using a second input, wherein the one or more additional procedure locations are determined based at least in part on an analysis of the endoscopic video stream.

[0087] In Example 7, the subject matter of Example 6 optionally includes subject matter wherein the one or more procedure locations or the one or more additional procedure locations includes a pathology location.

[0088] In Example 8, the subject matter of any one or more of Examples 6-7 optionally include subject matter wherein the one or more procedure locations or the one or more additional procedure locations includes a nonpathology location, and wherein the non-pathology location is selected by a trained learning model based at least in part on an analysis of the input.

[0089] In Example 9, the subject matter of any one or more of Examples 6-8 optionally include subject matter wherein responsive to a completed procedure at the one or more procedure locations or the one or more additional procedure locations the instructions further cause the processing circuitry to: generate a completed procedure indication on the displayed map.

[0090] In Example 10, the subject matter of any one or more of Examples 6-9 optionally include subject matter wherein responsive to an uncompleted procedure at the one or more procedure locations or the one or more additional procedure locations the instructions further cause the processing circuitry to: generate an alert.

[0091] In Example 11, the subject matter of any one or more of Examples 1-10 optionally include subject matter wherein the instructions further cause the processing circuitry to: generate a recommendation to a user; and update a patient file based on the recommendation.

[0092] In Example 12, the subject matter of Example 11 optionally includes subject matter wherein the recommendation is made in real-time as an endoscope is withdrawn from the portion of anatomy.

[0093] In Example 13, the subject matter of any one or more of Examples 11-12 optionally include subject matter wherein the recommendation includes an action to be taken during a later medical procedure.

[0094] In Example 14, the subject matter of Example 13 optionally includes subject matter wherein the action to be taken during the later medical procedure is based on a change in the portion of anatomy detected during the endoscopic medical procedure from a previous medical procedure.

[0095] In Example 15, the subject matter of any one or more of Examples 11-14 optionally include subject matter wherein to update the patient file includes indicating a risk associated with the patient determined during the endoscopic medical procedure.

[0096] Example 16 is a system for determining one or more procedure locations during an endoscopic medical procedure, the system comprising: processing circuitry; and memory, including instructions stored thereon that when performed by the processing circuitry, cause the processing circuitry to: receive one or more inputs corresponding to the endoscopic medical procedure;determine, based on the one or more inputs, one or more procedure locations in association with a map of a portion of an anatomy of a patient, wherein a first input includes an endoscopic video stream, wherein the endoscopic video stream is analyzed in real-time during an endoscopic insertion phase to determine the one or more procedure locations; display the map on a graphical user interface; display graphical elements that indicate the one or more procedure locations on the displayed map; display additional graphical elements that indicate one or more additional procedure locations on the displayed map; responsive to a completed procedure at the one or more procedure locations or the one or more additional procedure locations, dynamically update the graphical elements or the additional graphical elements to indicate the completed procedure on the displayed map; and responsive to an uncompleted procedure at the one or more procedure locations or the one or more additional procedure locations, generate an alert.

[0097] In Example 17, the subject matter of Example 16 optionally includes subject matter wherein a second input includes one or more of: a patient history, a current position of an endoscope in the anatomy of the patient, or one or more organizational guidelines.

[0098] In Example 18, the subject matter of Example 17 optionally includes subject matter wherein the one or more additional procedure locations are determined based at least in part on an analysis of the endoscopic video stream, wherein analysis of the endoscopic video stream includes image segmentation analysis by employing a trained learning model, and wherein the trained learning model includes one or more of a computer-aided detection algorithm or a computer-aided diagnosis algorithm.

[0099] Example 19 is a method for determining one or more procedure locations during an endoscopic medical procedure, the method comprising: receiving an input corresponding to the endoscopic medical procedure; determining, based on the input, one or more procedure locations in association with a map of a portion of an anatomy of a patient; displaying the map on a graphical user interface; displaying graphical elements that indicate the one or more procedure locations on the displayed map; and dynamically updating the graphical elements during the endoscopic medical procedure to indicate whetherthe one or more procedure locations corresponds to one of a completed procedure or an uncompleted procedure.

[0100] In Example 20, the subject matter of Example 19 optionally includes subject matter wherein the input includes an endoscopic video stream and one or more organizational guidelines, and wherein the method further includes: selecting the one or more organizational guidelines based at least in part on a type of the endoscopic medical procedure or a physical location at which the endoscopic medical procedure is performed, wherein the one or more procedure locations are determined based at least in part on the one or more organizational guidelines.

[0101] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments that may be practiced. These embodiments are also referred to herein as “examples.” Such examples may include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0102] All publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

[0103] In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In the appendedclaims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0104] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, such as by one of ordinary skill in the art upon reviewing the above description. The Abstract is to allow the reader to quickly ascertain the nature of the technical disclosure and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. The scope of the embodiments should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A system for determining one or more procedure locations during an endoscopic medical procedure, the system comprising: processing circuitry; and memory, including instructions stored thereon that when performed by the processing circuitry, cause the processing circuitry to: receive an input corresponding to the endoscopic medical procedure; determine, based the input, one or more procedure locations in association with a map of a portion of an anatomy of a patient; display the map on a graphical user interface; display graphical elements that indicate the one or more procedure locations on the displayed map; and dynamically update the graphical elements during the endoscopic medical procedure to indicate whether the one or more procedure locations corresponds to one of a completed or uncompleted procedure there.

2. The system of claim 1, wherein the input includes at least one of an endoscopic video stream, a patient history, a current position of an endoscope in the anatomy of the patient, or one or more organizational guidelines.

3. The system of claim 2, wherein the input includes the one or more organizational guidelines and wherein the processing circuitry is to: select the one or more organizational guidelines based at least in part on a type of the endoscopic medical procedure or a physical location at which the endoscopic medical procedure is performed.

4. The system of claim 3, wherein the input includes the endoscopic video stream, wherein the one or more procedure locations are determined based at least in part on the one or more organizational guidelines.

5. The system of claim 4, wherein analysis of the endoscopic video stream includes image segmentation analysis by employing a trained learning model, wherein the endoscopic video stream is analyzed in real-time during an endoscopic insertion phase to determine the one or more procedure locations, and wherein the trained learning model includes one or more of a computer- aided detection algorithm or a computer-aided diagnosis algorithm.

6. The system of claim 4, wherein the instructions further cause the processing circuitry to: indicate one or more additional procedure locations on the displayed map using a second input, wherein the one or more additional procedure locations are determined based at least in part on an analysis of the endoscopic video stream.

7. The system of claim 6, wherein the one or more procedure locations or the one or more additional procedure locations includes a pathology location.

8. The system of claim 6, wherein the one or more procedure locations or the one or more additional procedure locations includes a non-pathology location, and wherein the non-pathology location is selected by a trained learning model based at least in part on an analysis of the input.

9. The system of claim 6, wherein responsive to a completed procedure at the one or more procedure locations or the one or more additional procedure locations the instructions further cause the processing circuitry to: generate a completed procedure indication on the displayed map.

10. The system of claim 6, wherein responsive to an uncompleted procedure at the one or more procedure locations or the one or more additional procedure locations the instructions further cause the processing circuitry to: generate an alert.

11. The system of claim 1, wherein the instructions further cause the processing circuitry to:generate a recommendation to a user; and update a patient file based on the recommendation.

12. The system of claim 11, wherein the recommendation is made in realtime as an endoscope is withdrawn from the portion of anatomy.

13. The system of claim 11, wherein the recommendation includes an action to be taken during a later medical procedure.

14. The system of claim 13, wherein the action to be taken during the later medical procedure is based on a change in the portion of anatomy detected during the endoscopic medical procedure from a previous medical procedure.

15. The system of claim 11, wherein to update the patient file includes indicating a risk associated with the patient determined during the endoscopic medical procedure.

16. A system for determining one or more procedure locations during an endoscopic medical procedure, the system comprising: processing circuitry; and memory, including instructions stored thereon that when performed by the processing circuitry, cause the processing circuitry to: receive one or more inputs corresponding to the endoscopic medical procedure; determine, based on the one or more inputs, one or more procedure locations in association with a map of a portion of an anatomy of a patient, wherein a first input includes an endoscopic video stream, wherein the endoscopic video stream is analyzed in real-time during an endoscopic insertion phase to determine the one or more procedure locations; display the map on a graphical user interface; display graphical elements that indicate the one or more procedure locations on the displayed map;display additional graphical elements that indicate one or more additional procedure locations on the displayed map; responsive to a completed procedure at the one or more procedure locations or the one or more additional procedure locations, dynamically update the graphical elements or the additional graphical elements to indicate the completed procedure on the displayed map; and responsive to an uncompleted procedure at the one or more procedure locations or the one or more additional procedure locations, generate an alert.

17. The system of claim 16, wherein a second input includes one or more of: a patient history, a current position of an endoscope in the anatomy of the patient, or one or more organizational guidelines.

18. The system of claim 17, wherein the one or more additional procedure locations are determined based at least in part on an analysis of the endoscopic video stream, wherein analysis of the endoscopic video stream includes image segmentation analysis by employing a trained learning model, and wherein the trained learning model includes one or more of a computer-aided detection algorithm or a computer-aided diagnosis algorithm.

19. A method for determining one or more procedure locations during an endoscopic medical procedure, the method comprising: receiving an input corresponding to the endoscopic medical procedure; determining, based on the input, one or more procedure locations in association with a map of a portion of an anatomy of a patient; displaying the map on a graphical user interface; displaying graphical elements that indicate the one or more procedure locations on the displayed map; and dynamically updating the graphical elements during the endoscopic medical procedure to indicate whether the one or more procedure locations corresponds to one of a completed procedure or an uncompleted procedure.

20. The method of claim 19, wherein the input includes an endoscopic video stream and one or more organizational guidelines, and wherein the method further includes: selecting the one or more organizational guidelines based at least in part on a type of the endoscopic medical procedure or a physical location at which the endoscopic medical procedure is performed, wherein the one or more procedure locations are determined based at least in part on the one or more organizational guidelines.

Images