System with multiple ordered classification modules

Abstract

A clinical support system and method for real-time activation of detection and characterization modules trained for identification or diagnosis of abnormalities in the gastrointestinal tract is disclosed. A clinical support computer may be programmed to: activate a first mucosa identification module and a first plurality of detection and characterization modules based at least in part on detection of a connection to an endoscope; monitor an image stream from the endoscope using the first mucosa identification module to identify a mucosal tissue type; execute a first detection module from the first plurality of detection and characterization modules based on identifying the mucosal tissue type as the first mucosal tissue type; and use the first detection module to process an image from the image stream received from the endoscope to identify a region of interest for output to a display device, the region of interest identifying a potential abnormality in the image.

Description

[Technical field]

[0001] Priority claim This application claims priority to U.S. Provisional Application No. 63 / 363,654, filed April 27, 2022, and U.S. Provisional Application No. 63 / 387,140, ​​filed December 13, 2022, the contents of which are incorporated herein by reference. [Background technology]

[0002] Examination of a patient's gastrointestinal (GI) tract with endoscopic instruments is a fairly typical medical preventive / diagnostic medical procedure. For the upper GI tract, a doctor (or more commonly, a healthcare provider (HCP)) examines the inside of a patient's esophagus, stomach, and duodenum (the first part of the small intestine). The instruments used during such procedures are thin, lighted tubes that contain a camera to view the inner wall surfaces of the organs that make up the upper GI tract, and this instrument is commonly called an endoscope. Thus, upper endoscopy, also known as upper gastrointestinal endoscopy, is a procedure used to visually examine a patient's digestive system. Each section of the upper GI tract has a distinct tissue, called the mucosa, that can give a well-trained eye an indication of its location within the GI tract. The visual examination performed in an endoscopy procedure is intended to identify abnormalities that may require further diagnosis.

[0003] Similarly, endoscopic instruments can be used to visually examine a patient's lower gastrointestinal tract (lower GI tract). Lower GI endoscopy, also called colonoscopy or sigmoidoscopy, allows HCPs to view the mucosal lining of a patient's lower gastrointestinal tract. The procedure is usually used as a screening test in asymptomatic individuals or to help diagnose unexplained abdominal pain, rectal bleeding, or changes in bowel habits. Routine colonoscopy is recommended after a certain age to allow early detection of cancer or other GI tract problems before they progress. Identification of abnormalities in the GI tract requires a well-trained HCP and focused review of the camera image during the procedure. Endoscope manufacturers have developed several image processing techniques to assist in identifying potential abnormalities, but these techniques can be difficult for HCPs to use in practice. Summary of the Invention [Problem to be solved by the invention]

[0004] Advances in artificial intelligence (AI) capabilities have generated immense interest in exploring use cases in healthcare. From a practical standpoint, multiple different use cases may be applicable in individual medical settings. One such medical setting is an endoscopy center where an HCP may use an endoscope to inspect multiple individual internal anatomical structures. The inspection of each individual internal anatomical structure may correspond to one or more dedicated AI use cases (e.g., algorithms, trained AI models), and these dedicated AI use cases may not be applicable to other internal anatomical structures. For example, consider a gastrointestinal endoscopy (aka upper GI endoscopy) during which an HCP may begin the procedure by inserting an endoscope into a patient's esophagus to inspect the patient's esophageal mucosa, then advancing the endoscope into the patient's stomach, and ultimately into the patient's duodenum. Due to the significant differences between the physiological structure of the mucosa in each of the esophagus, stomach, and duodenum, a trained AI model that reliably identifies and / or provides a diagnosis of an abnormality (e.g., cancerous tissue) in one of these regions may not be able to identify and / or diagnose abnormalities in other regions that may be examined during a single procedure.

[0005] Training HCPs on AI tools due to their constantly evolving nature is difficult, and thus healthcare systems that require manual changes to procedure settings to switch between available AI tools create a highly error-prone environment. Furthermore, requiring manual changes to procedure settings increases overall procedure times that reduce the number of cases that individual healthcare settings can support on a daily basis. Techniques are needed to improve the quality and increase the amount of care provided at individual healthcare settings where multiple AI tools can be utilized.

[0006] This is an emerging technology area, and existing systems are essentially standalone AI models that are trained to perform specific AI functions within distinct anatomical domains. These AI models can also be developed by multiple entities, making AI capabilities more fragmented and difficult to integrate. [Means for solving the problem]

[0007] To address these challenges, the inventors have developed an end-to-end system that intelligently and seamlessly switches between distinct AI models during a procedure based on endoscope selection and the characteristics of the currently observed tissue.

[0008] In the drawings, which are not necessarily drawn to scale, the same numbers may describe similar components in different views. The same numbers with different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 illustrates an example software architecture diagram in accordance with at least one example of the present disclosure. [Figure 2A] FIG. 1 illustrates an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. [Figure 2B] FIG. 1 illustrates an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. [Figure 2C] FIG. 1 illustrates an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. [Figure 2D] FIG. 1 illustrates an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. [Figure 2E] FIG. 1 illustrates an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. [Figure 3A] FIG. 1 illustrates a machine learning model training diagram in accordance with at least one example of the present disclosure. [Figure 3B] FIG. 1 is a machine learning model inference diagram in accordance with at least one example of the present disclosure. [Figure 4] FIG. 1 illustrates a machine learning training technique in accordance with at least one example of the present disclosure. [Diagram 5] FIG. 1 illustrates a clinical decision support technique for automatic selection of detection and / or classification modules during a procedure, in accordance with at least one example of the present disclosure. [Figure 6] 1 is a block diagram of an example machine on which any one or more of the techniques discussed herein may be implemented in accordance with at least one example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In one example, the software architecture includes multiple ordered hierarchies of modules, where the output of a higher level module acts as a trigger to activate / deactivate a lower level module. In this way, the higher level module monitors the current conditions of the operating environment and selectively activates the lower level modules designed to operate in the current conditions based on these current conditions. For example, the higher level module may determine the type of instrument connected to the endoscopy workstation and initialize (e.g., bring up to operation) the lower level modules designed for use with this instrument type (e.g., colonoscope vs. esophagoscope) or with a finer instrument model (e.g., esophagoscope A vs. esophagoscope B). Additionally or alternatively, the higher level module may determine the type of tissue currently being imaged by the endoscope and activate the CADe and / or CADx modules that uniquely correspond to this tissue type. The CADe / CADx modules are artificial intelligence computer modules for detection and / or classification of abnormal tissue. In this context, "CAD" is an acronym for Computer Aided Detection, and the CADe version indicates a version of the computer aided detection configured to identify abnormal tissue. The CADx version refers to a computer aided detection system configured to identify and classify the detected abnormal tissue. Furthermore, in the context of this application, the term "higher level module" refers to a software component that generates an output that is used to determine an operating parameter corresponding to another software component, which is appropriately referred to as a "lower level module" in contrast to the above-mentioned "higher level module". Furthermore, the term "module" is used herein to define a functionally discrete piece of software code designed to perform a particular function or provide a particular service.A "module" is executable within a suitable computing system to produce a defined output based on a defined input or set of inputs. In some examples, a "module" may encompass an entire application, such as a clinical decision support system or an application executable within a mobile computing device.

[0011] The software architecture discussed herein may also allow for an "app store" approach to CADe and CADx modules. The software architecture allows the HCP to select from a variety of CADe / CADx modules compatible with a selected instrument for use in diagnosing within various portions of the GI tract. An interface may be provided that allows the HCP to associate the selected CADe / CADx module with different segments of the GI tract (detected during the procedure by a mucosal tissue identification module, as discussed in more detail below).

[0012] FIG. 1 illustrates an exemplary software architecture diagram according to at least one example of the present disclosure. The exemplary software architecture 100 includes multiple hierarchical levels, beginning with a first level 102 including an instrument identification module 104. A second level 110 in this example may include multiple mucosal identification modules 112A-112N including a first mucosal identification module 112A through an Nth mucosal identification module 112N, and the present disclosure may refer to the mucosal identification module 112 as shorthand for referring to one or more modules of the multiple mucosal identification modules 112A-112N. Additionally, the multiple mucosal identification modules 112A-112N may be referred to as multiple mucosal identification modules 112. In this example, the second level 110 may also include one or more anatomy identification modules 114 for use in the lower GI tract. The anatomy identification module 114 may also operate based on mucosal tissue identification. As described in more detail below, both the mucosal identification module 112 and the anatomy identification module 114 may use trained machine learning models for identification of their respective tissue types. Further, in some examples, the second level 110 may use a single mucosal identification module (e.g., mucosal identification module 112) that is programmed / trained to identify a variety of different mucosal tissue types. In other words, in some examples, multiple mucosal identification modules 112A-112N are encapsulated into a single mucosal identification module.

[0013] In this example, the software architecture 100 may include a third level 120 that includes CADe modules for specific areas of the upper or lower GI tract. In this example, the third level 120 is shown as including an esophageal CADe module 122 and a gastric CADe module 124 as well as a CADe module 126 that targets identification of abnormalities in the lower GI tract. The software architecture 100 may include any number of levels, represented in this disclosure by an Nth level 130. The Nth level 130 is shown as including an esophageal CADx module 132 and a gastric CADx module 134 as well as a CADx module 138 that targets characterization / classification of abnormalities in the lower GI tract (e.g., characterization of polyps). As discussed above, the various levels of the software architecture 100 provide a structure to enable a clinical decision support system (CDSS) to seamlessly switch between the use of multiple different AI tools in a single encounter based on current conditions (e.g., the current position of the endoscopic camera in the patient's GI tract as determined by the mucosa identification module).

[0014] An exemplary endoscopy software architecture, i.e., software architecture 100, may include a first level including one or more instrument identification modules (e.g., instrument identification module 104) and one or more anatomy-specific AI suites. As shown below, the exemplary endoscopy software architecture includes an upper GI AI suite (e.g., a gastrointestinal endoscopy (EGD) AI suite) and a lower GI AI suite (e.g., a colonoscopy AI suite). Each of the AI ​​suites includes a number of modules that are designed to perform a specific AI function through analysis of the image stream captured by the endoscope. Additionally, the output of a higher level module within an AI suite may act as a determining factor as to which lower level module is to operate at a given time. For example, a determination of the type and / or model of instrument selected by the user (or plugged into the endoscope rack) may cause the CDSS to initialize a particular AI suite that corresponds to the instrument type and / or model. As another example, the most recent output of a second level module may describe which third level module is currently operating, and so on.

[0015] In the illustrated example, the second level 110 of the software architecture 100 includes various modules designed to determine the type of tissue currently being imaged by the endoscope. Specifically, within the upper GI AI suite, there are mucosa identification modules (e.g., multiple mucosa identification modules 112) each designed to determine whether a particular type of mucosa is currently being imaged. Additionally or alternatively, a single mucosa identification module (e.g., a single AI-based tissue classification model) may be configured to determine which one of multiple different types of mucosa is currently being imaged. Although shown at the second level 110, these mucosa identification modules 112 (or more broadly, anatomy identification modules if the target anatomy is not mucosa) may be at the first level in alternative implementations. In other words, instrument identification within the illustrated first level is an optional part of the software architecture 100.

[0016] Further, under the anatomy identification module are various AI models that are specifically developed and trained to analyze the type of tissue currently being viewed as determined by the anatomy identification module (which may be an AI model). For example, in a situation where the most recent image analyzed in the second level 110 indicates that esophageal tissue / mucosa is imaged by the endoscope, a third level 120 AI model (e.g., esophageal CADe module 122) is used that is configured to analyze esophageal tissue / mucosa. Such AI models may include CADe and / or CADx type models, which in some implementations are separated into one or more levels as shown.

[0017] In effect, this type of tiered software architecture allows the AI-enabled healthcare system to intelligently switch between different AI models, each developed and trained for operation under a particular operating situation, in a seamless and dynamic manner throughout a procedure as operating conditions vary. As a specific example, as described in more detail below, the AI-enabled healthcare system may automatically transition from a first AI model configured to analyze a first anatomy type to a second AI model configured to analyze a second anatomy type when the endoscope advances from a first anatomical structure to a second anatomical structure.

[0018] In this manner, the concept adds significant value by creating an operating environment that, from the end user's (e.g., HCP) perspective, operates seamlessly to facilitate a single goal (e.g., identifying, diagnosing, and ultimately treating diseased tissue) by dynamically switching between various features and / or AI tools in response to changing operating conditions. At the same time, while appearing to be the only tool from the end user's perspective, the concept retains a modular approach from the developer's perspective that allows features to be easily added, modified, and constantly improved as the underlying technology advances (e.g., without performing a complete software overhaul). Additionally, the concept facilitates end users being provided with varying degrees of functionality according to their subscription level (HCP A may be provided with less or different features than HCP B).

[0019] To further convey these concepts, an example intra-procedural system flow is described below with respect to a series of figures. Figures 2A-2E show an example intra-procedural process flow layered on a software architecture in accordance with at least one example of the present disclosure. Each figure in Figures 2A-2E represents a different operation in the technique and is timed with a different timestamp (T 1 , T 2 etc.)

[0020] Figure 2A: Time=T 1 In this example, the technique 200 may begin at operation 202, which begins at time T 1 The first moment in the process occurs when an HCP interacts with an exemplary endoscopy software architecture (e.g., software architecture 100) to connect an esophagoscope to the system. The instrument identification module 104 recognizes that the instrument is an esophagoscope and responds by registering the intended anatomy, for example, by initializing the upper GI AI suite (e.g., since the most likely use of an esophagoscope is examination of the upper GI space). As shown below, because the instrument has just been connected and the examination has not yet begun (e.g., images of the upper GI space have not yet been captured), the outputs of the various modules in the upper GI AI suite may be N / A (not applicable) (e.g., null) because images of the anatomical structures to be examined have not yet been captured.

[0021] Figure 2B: Time=T 2 Time T 2At a second time instant in the esophagus, the technique 200 continues at operation 204, which involves the HCP inserting an esophagoscope into the patient's esophagus and the mucosal identification module (e.g., the mucosal identification module 112) operating to continually (or periodically at an appropriate sample rate) determine the type of tissue currently being imaged. As shown, the mucosal identification module 112 generates an output indicating with high confidence (e.g., as measured by an appropriate threshold level, such as 0.95 or greater; different confidence levels are within the scope of this disclosure) that the esophageal mucosa is currently being imaged. In some examples, the system may generate a warning or other prompt to be displayed to the HCP if it is determined that the identified tissue type does not match the registered anatomical structure. Note that the software may provide a setting that allows the HCP to set a threshold confidence level for the mucosal identification module 112. Based on this determination at the second level 110 of the software architecture 100, the technique 200 continues at operation 206, where the system executes the esophageal CADe module 122 to continually analyze the images being generated during the procedure. At time T 2 In the example, the output of the esophageal CADe module 122 indicates that the tissue being observed is normal (e.g., non-abnormal). Thus, the system continues with the first imaging modality (e.g., visible light imaging) and continues to run the esophageal CADe module 122 while simultaneously running the mucosal identification module 112 belonging to the upper GI AI suite constantly (or at an appropriate sampling frequency). In this example, multiple mucosal identification modules 112A-112N continue to run in the background to detect changes in the imaged mucosal tissue that indicate a transition to a different section of the GI tract being examined.

[0022] In an alternative example (not shown), time T 2In, the HCP may activate the NBI module to allow some CADx modules to operate. In non-precancerous conditions, and depending on the size and location within the colon, a polypectomy may be performed to remove for follow-up evaluation by a pathologist or discarded.

[0023] Figure 2C: Time=T 3 Time T 3 At a third time in the, technique 200 may continue at operation 208, which includes the HCP manipulating the esophagoscope so that abnormal tissue is imaged. As a result, the esophageal CADe module 122 generates an output indicating that an abnormality is present in the space being imaged. In an example, a user interface dialog box may be output to a display device. Additionally, an audible sound (alert) may be generated to ensure that the HCP is aware of the detected abnormal tissue. For example, it should be appreciated that various CADe models are designed to detect abnormalities such as polyps but not characterize those polyps (e.g., as benign, malignant, etc.). In some examples, graphics such as a region of interest bounding box are overlaid on the endoscopic image to alert the HCP to the detection of abnormal tissue.

[0024] Based on the output from the esophageal CADe module 122, the system may respond by taking one or more predetermined actions. For example, the system may respond to an output from the esophageal CADe module 122 indicating that an abnormality has been detected by automatically activating a function to provide additional information about the abnormal tissue being imaged. In some embodiments, in response to an output from the esophageal CADe module 122, the system may automatically deactivate a first imaging modality (e.g., full spectrum or white light imaging) and activate a second imaging modality (e.g., blue or red light narrow band imaging (NBI)) with the esophageal CADx module 132 (in the Nth level 130) using images captured by the second imaging modality to characterize the abnormal esophageal tissue. In other examples, the esophageal CADx module 132 may be designed to operate using white light, and thus the system does not activate the second imaging modality. In the immediate example, the esophageal CADe module 122 may be appropriately described as a “higher level module” with respect to the esophageal CADx module 132 because the output of the esophageal CADe module 122 causes the system to trigger or activate the functionality of the “lower level” esophageal CADx module 132.

[0025] In some embodiments, the system may respond to an output of the Esophageal CADe module 122 indicating that an abnormality is being imaged by generating a UI element (e.g., visual, auditory, tactile) to prompt the HCP for an indication as to whether images captured by a second imaging modality are desired. If the HCP provides an input indicating that such second modality images are desired, the system may respond by switching from the first imaging modality (e.g., visible light) to the second imaging modality (e.g., NBI). Then, in response to images being captured by the second modality, the system may automatically use the Esophageal CADx module 132 to analyze these images and generate an output indicative of a tissue classification and a corresponding confidence score. In some examples, the HCP may manually turn on the NBI or another modality. In these examples, information about the new modality may be captured without the need to detect a different type of image.

[0026] Figure 2D: Time=T 4 In this example, the technique 200 may continue at operation 210, which begins at time T 4 In operation 210, the second imaging modality is switched on (either automatically or in response to an input from the HCP to do so). In response to the second imaging modality being activated, the system begins providing captured images to the esophageal CADx module 132, which classifies the abnormality identified by the CADe (e.g., by the esophageal CADe module 122 executing in operation 206). For example, as shown, the esophageal CADx module 132 provides an output indicating that the abnormality is malignant stage 1 and that this classification corresponds to a confidence of 0.87.

[0027] Figure 2E: Time=T 5 In this example, the technique 200 5At a fifth moment in the process, the method may end with operations 212 and 214. In these operations (212, 214), the HCP advances the esophagoscope through the esophagus to the stomach. As a result, the esophagoscope begins to capture images of the gastric mucosa. As the mucosa identification modules (e.g., the multiple mucosa identification modules 112A-112N) continue to run throughout the technique 200, the system seamlessly determines to transition from operation of an AI module configured to analyze the esophageal mucosa to an AI module configured to analyze the gastric mucosa based on the output of these mucosa identification modules (112A-112N). Thus, as shown in FIG. 2E, the output of the esophageal CADe module 122 becomes inapplicable (or becomes null, as it may cease to operate altogether), and the gastric CADe module 124 begins to output an indication of whether the currently viewed gastric mucosa is normal or abnormal. In fact, the technique 200 continues in operation throughout the entire endoscopic procedure, with the discussion above showing a portion of such a procedure.

[0028] Alternative Use Case: Landmark Identification and Automatic Report Generation During a colonoscopy, one important consideration is that a complete colonoscopy has been performed, meaning that the HCP has advanced the colonoscope from the anus to the end of the colon. The cecum is located at the end of the colon and marks the clear transition from the colonic space to the small intestine. Current best practice is for the HCP to advance the colonoscope to the end of the colon and then begin to withdraw the colonoscope while inspecting the colonic mucosa upon withdrawal.

[0029] The software architecture 100 can also be used to interrogate the colonic space and automatically populate aspects of the procedure report based on landmark identification. For example, an anatomy identification module, such as anatomy identification module 114, may be configured to analyze images that specifically identify the cecum. Once identified, the system can automatically populate the report with instructions including: Time it reached the cecum The duration the colonoscope was advanced This can be calculated by noting the time when the anatomy identification module first identifies the colonic mucosa (indicating the moment the colonoscope enters the anus) and the time when the cecum is identified, then calculating the advancement time as the difference between these times. Duration that the colonoscope was removed (calculated inversely as above, i.e., the time that the cecum was identified minus the time that the colonoscope was removed from the anus and the colonic mucosa was no longer identified in the image)

[0030] 3A is a diagram 300A illustrating a machine learning model training diagram in accordance with at least one example of the present disclosure. Diagram 300A illustrates components and inputs for training a model 302 using machine learning.

[0031] Machine learning (ML) is an application that provides computer systems with the ability to perform tasks without being explicitly programmed by making inferences based on patterns found during the analysis of data. Machine learning investigates the study and construction of algorithms, also referred to herein as tools, that can learn from existing data and make predictions about new data. Although examples may be presented for a few machine learning tools, the principles presented herein may be applied to other machine learning tools.

[0032] Machine learning (ML) algorithms use data (e.g., action and / or dialogue primitives, goal vectors, rewards, etc.) to find correlations between identified features that affect output outcomes. A feature is an individual measurable property of the phenomenon being observed. Exemplary features for model 302 may include diagnostic data (e.g., from a physician), reported patient output outcome data, labeled mucosal tissue endoscopic images, and corresponding endoscope positions. Features that include and / or may be referred to as clinical data may be compared to input data, such as endoscopic tissue images.

[0033] The concept of feature is related to that of explanatory variables used in statistical techniques such as linear regression. Choosing informative, discriminative, and independent features is important for the effective operation of ML in pattern recognition, classification, and regression. Features may be of different types, such as numerical features, strings, and graphs.

[0034] During training, the ML algorithm analyzes the identified features and, optionally, input data (e.g., patient data such as environmental data, condition data, demographics and / or co-morbidities, etc.) based on configuration parameters defined for training. The result of training is a model 302, which is capable of receiving inputs to produce complex tasks. In this example, the model 302 will be trained to identify different types of mucosa from input endoscopic images.

[0035] In one example, the input data may be labeled (e.g., for use as features in the training phase). Labeling may include identifying mucosal tissue and the location in the GI tract where the tissue was located. The labeled training images may be weighted and / or used to generate different versions of the model 302.

[0036] Input training data for the model 302 may include clinical data, which may include patient data such as weight, height, and any other patient data that may influence risk factors associated with the targeted diagnosis.

[0037] A neural network, sometimes called an artificial neural network, is a computing system based on the study of biological neural networks of animal brains. Such systems progressively improve their performance, called learning, to perform tasks, usually without task-specific programming. For example, in image recognition, a neural network may be taught to analyze example images that are tagged with names for objects and to identify images that contain the objects by learning the objects and names, and the analysis results may be used to identify and / or classify the objects in untagged images. In FIG. 3A, for example, model 302 may be trained to identify mucosal tissue types and / or locations in the GI tract, and / or percentage likelihood and / or confidence of the tissue location.

[0038] Neural networks are based on a collection of connected units, called neurons, where each connection between neurons, called a synapse, can send a unidirectional signal with an activation strength that varies with the strength of the connection. A receiving neuron can activate and propagate a signal to downstream neurons connected to it based on whether the combined input signal, usually from potentially many sending neurons, is strong enough, where strength is a parameter.

[0039] A deep neural network (DNN) is a stacked neural network consisting of multiple layers. A layer consists of nodes, which are locations where computations occur, loosely patterned on neurons in the human brain, and the brain fires when it encounters sufficient stimuli. A node combines an input from the data with a set of coefficients and / or weights that amplify or attenuate that input, and this set assigns importance to the input for the task the algorithm is trying to learn. The products of these inputs and weights are summed, and the sum is passed through what is called an activation function for the node to determine whether and to what extent the signal will proceed further through the network to affect the final output result. DNNs use a chain of many layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Higher level features are derived from lower level features to form a hierarchical representation. The layers following the input layer may be convolutional layers that are filtering the results of the input and produce feature maps that are used by the next convolutional layer.

[0040] The DNN may be a particular type of DNN, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), etc. In some examples, other artificial neural networks may be used. In some examples, a classifier may be used instead of a neural network. A classifier may not include a hidden layer, but may classify a particular input as corresponding to a particular output. For example, for mucosal image input data, an identification of location within the GI tract may be generated by a classifier.

[0041] The input data for training the model 302 may include data captured from an endoscope along with labeled data from a physician. The model 302 may be used in the inference stage (described in more detail below with respect to FIG. 3B) to determine the location within the GI tract of a particular input mucosal tissue image.

[0042] As shown in FIG. 3A, the training data may include signal training data consisting of endoscopic images of mucosal tissue and additional procedure-related data such as endoscopic instrument type and illumination modality. In some examples, the signal training data includes annotation or labeling data provided by a physician (e.g., as image labeling data). For example, a physician may annotate each input endoscopic mucosal tissue image (e.g., each image data N). In some examples, a patient and / or a physician may also provide clinical data used to train the model 302. The clinical data may include diagnosis data and patient data.

[0043] Based on the signal training data and / or the annotation training data, model 302 may generate output weights corresponding to individual processing nodes scattered across an input layer, an output layer, and one or more hidden layers. Model 302 and the trained weights may later be used to infer an indication of GI tract location based on an endoscopic input image of mucosal tissue.

[0044] 3B illustrates a machine learning model inference diagram 300B according to at least one example of the present disclosure. In the inference diagram 300B, a model 304 (e.g., model 302 after training and / or updated, etc.) may be used to output a prediction, such as a location in the GI tract or a mucosal tissue type. A confidence level and / or weighting may be output as the prediction or in addition to the other predictions discussed above. The machine learning model inference diagram 300B may represent an example computer-based clinical decision support system (CDSS) configured to assist in predicting a position of an endoscopic instrument in a patient's GI tract.

[0045] As shown in FIG. 3B, the model 304 may receive as input signals from an endoscope, such as image data of the patient's GI tract. The input data may also include endoscopic instrument type and illumination modality, among others. The model 304 may generate outputs (e.g., inferences) including location within the GI tract or mucosal tissue type, etc. The model 304 may have a runtime that occurs while the patient is undergoing an endoscopic procedure. The model 304 may provide the physician with real-time or near real-time information about the location within the GI tract. The GI tract location information may be used by the CDSS to activate or deactivate a number of other AI / ML modules (e.g., CADe and / or CADx modules) that assist in the detection and / or characterization of tissue abnormalities.

[0046] 4 illustrates a machine learning training technique 400 according to at least one example of the present disclosure. In this example, technique 400 outlines basic steps for training a machine learning model to detect different mucosal tissue types and / or GI tract locations based on an input endoscopic image. Technique 400 may include operations such as receiving mucosal endoscopic image data at 402, labeling the image data at 404, training a machine learning model at 406, and outputting the machine learning model at 408.

[0047] In this example, technique 400 may begin at 402, where a system (as a computer system, detailed in FIG. 6 below) receives a plurality of mucosal tissue images from an endoscope. The mucosal tissue images may be from the upper GI tract and / or the lower GI tract. Images from the upper GI tract may include images of mucosa including esophageal mucosa, gastric mucosa, and duodenal mucosa, each of which has a visually distinct appearance. Image data from the lower GI tract may include images from the cecum, ascending colon, right colonic flexure, transverse colon, left colonic flexure, descending colon, sigmoid colon, rectum, and anal canal, among other distinct areas of the lower GI tract.

[0048] Technique 400 may continue at 404, where the system receives label data for each of the training mucosal endoscopic images received at operation 402. The labeling operation at 404 may also include weighting the training images according to the strength of representation of a particular mucosal tissue type. The output of operation 404 may include a set of training data including annotations associated with each image of the received image data. At 406, technique 400 continues with the training data being used to train a machine learning model that can subsequently be used to predict the location of the input endoscopic image in the GI tract. FIG. 3A above discusses additional details about training a machine learning model that can be implemented in the context of technique 400. Once trained, technique 400 may conclude at operation 408, where the system outputs the machine learning model for use in predicting GI tract location and / or mucosal tissue type. In an example, the trained machine learning model output by technique 400 can be used in technique 500, discussed below with reference to FIG. 5.

[0049] 5 illustrates a clinical decision support technique 500 for automatic selection of detection and / or classification modules during a procedure, according to at least one example of the present disclosure. In this example, technique 500 may include operations such as detecting an endoscope at 502, activating a mucosal identification module at 504, activating a first plurality of detection and characterization modules at 506, monitoring an image stream at 508, identifying a mucosal tissue type at 510, executing a first detection and / or characterization module at 512, alternatively executing a second detection and / or characterization module at 514, and outputting detection and / or characterization data at 516. Technique 500 discusses detection and characterization modules, which may be CADe and CADx modules, respectively.

[0050] In this example, technique 500 may optionally begin at 502, where a system (e.g., a CDSS) detects an endoscope type. For example, the system may detect a connection with an endoscope and determine that the endoscope belongs to an "esophagoscope" category of instruments. Additionally or alternatively, the system may more finely determine the specific model of the detected endoscope (e.g., the system may identify the endoscope as the GIF-XP190N version of the EVIS EXERA III sold by Olympus Corporation of the Americas). In this operation, the detection of the endoscope type may be triggered by the connection of the instrument to the CDSS. In other examples, the detection of the endoscope type may be triggered by the initialization of the instrument after connection with the CDSS or some other logical point in the procedure prior to the use of the instrument.

[0051] At 504, technique 500 continues with the system activating a first mucosa identification module. Activation of the mucosa identification module may be based on the type and / or model of the endoscopic instrument connected to the system. For example, based on the system determining at 502 that an instrument belonging to the "esophagoscope" category of instruments is connected, the system may begin running a mucosa identification module designed to use one or more classification type ML models to analyze the endoscopic image and output an indication of whether the image shows squamous epithelial mucosa, gastric mucosa, or villous mucosa. As discussed above, different types of endoscopes are used for different procedures, such as upper GI tract procedures versus lower GI tract procedures. Accordingly, the mucosa identification module can be tailored for upper GI tract versus lower GI tract (or based on other specific procedure features). In some examples, the mucosa tissue identification module can be trained to span the entire GI tract.

[0052] At 506, technique 500 continues with activating the first plurality of detection and characterization modules. In this example, the activation of the detection and characterization modules is also based on the type of endoscopic instrument detected. For example, in response to determining the type (e.g., esophagoscope) and / or model (e.g., GIF-XP190N) of the endoscope, the system may begin to run the upper GI AI suite shown in FIG. 1. Thus, it will be appreciated that aspects of each of operations 504 and 506 may be performed based on the output results and / or determination of operation 502. In another example, operations 504 and 506 may be based on another system input indicating where in the patient's GI tract the procedure is being performed. Another factor in determining which mucosal identification module to activate at 504 and which plurality of detection and characterization modules to activate at 506 may involve the available light modes (e.g., normal white light vs. NBI, or other illumination types) in the connected instrument.

[0053] At 508, technique 500 may continue with the system monitoring an image stream generated by the endoscope. Operation 508 may operate continuously throughout the procedure using the endoscopic instrument. At 510, technique 500 may continue with the system using an active mucosa identification module (e.g., a first mucosa identification module) to continuously (or periodically at a suitable sample rate) identify a mucosa tissue type within each processed image from the image stream monitored at operation 508. Operation 510 may run on every image in the monitored image stream or may be performed periodically on images extracted from the monitored image stream. In this example, once the first mucosa tissue type is identified at 510, technique 500 may continue with operation 512, where the system executes a first detection module and / or a first characterization module. For example, based on the occurrence of operation 510 indicating that squamous mucosa is currently being imaged, the system may perform operation 512, where the esophageal CADe module 122 and / or the esophageal CADx module 132 may be executed since the squamous mucosa delineates most of the esophagus. Executing the detection or characterization module at operation 512 may include selecting an illumination modality supported by the endoscopic instrument being used. At 516, technique 500 continues with the system outputting the detection and / or characterization data to assist the HCP in performing the ongoing procedure. Outputting the detection data at 516 may include outputting a region of interest overlaid on the endoscopic image to the HCP for identifying abnormal tissue. Operation 516 may also include displaying a user prompt to provide analysis options such as switching illumination modalities and / or executing a characterization module.

[0054] In some examples, technique 500 loops through operations 508-516 in real-time or quasi-real-time to continually update a graphical display of the detection and / or characterization data on a display device displaying the endoscopic image stream. In one example, after the first detection module generates a region of interest that identifies potentially abnormal tissue, technique 500 can loop back to operation 512 to execute a first characterization module. The first characterization module can then output a tissue classification in operation 516. In some examples, the characterization module can also output only a confidence score or indicator along with the tissue classification.

[0055] Returning to operation 510 (which may be performed continuously or periodically, as described above, simultaneously with operations 512 and 516), once the mucosal identification module identifies a second mucosal tissue type, technique 500 may continue with the system executing a second detection module and / or a second characterization module in operation 514. For example, based on an indication that operation 510 occurs that the gastric mucosa is now being imaged, the system may automatically switch to performing operation 514, where the gastric CADe module 124 and / or the gastric CADx module 134 may be executed since the gastric mucosa delineates the stomach. Technique 500 may terminate in operation 516, where the system outputs the detection and / or characterization data to assist the HCP in performing the ongoing procedure. In this example, the detection of the second mucosal tissue type in operation 510 is an indication that the procedure has transitioned to a different area of ​​the GI tract. In an example, this detection that the procedure has transitioned to a new area of ​​the GI tract, and an indication of this event, may be output to a display device coupled to the system to further assist the HCP.

[0056] Although the above invention is discussed with reference to an endoscopic application, the same concepts may be applied to endoscopic, ultrasound, and microscopy applications. The system described herein can transition between AI modules within a procedure, adding metadata that identifies which AI module is operating on different parts of the video stream. The system supports modular insertion of AI modules as features are added or updated. The system can generate reports detailing which algorithms (modules) were selected and why throughout the procedure.

[0057] FIG. 6 illustrates a block diagram of an example machine 600 on which any one or more of the techniques (processes) discussed herein may be implemented according to some embodiments. In alternative embodiments, the machine 600 may operate as a standalone device and / or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 600 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 600 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 600 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the machine. Further, although only a single machine is shown, the term "machine" shall also be taken to include any collection of machines that individually or collectively execute a set (or sets) of instructions to implement any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), other computer cluster configurations, etc.

[0058] The machine (e.g., computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 604, and a static memory 606, some or all of which may communicate with each other via an interlink (e.g., a bus) 608. The machine 600 may further include a display unit 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In an example, the display unit 610, the input device 612, and the UI navigation device 614 may be touch screen displays. The machine 600 may further include a storage device (e.g., a drive unit) 616, a signal generating device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 621, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 600 may include an output controller 628, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, for communicating with and / or controlling one or more peripheral devices (e.g., printers, card readers, etc.).

[0059] The storage device 616 may include a machine-readable medium 622 on which is stored one or more sets of data structures or instructions 624 (e.g., software) that embody or are used by any one or more of the techniques or functions described herein. The instructions 624 may reside, completely or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution by the machine 600. In an example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the storage device 616 may constitute a machine-readable medium.

[0060] Although the machine-readable medium 622 is depicted as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to store one or more instructions 624. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 600, causing the machine 600 to perform any one or more of the techniques of this disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include semiconductor memories, and optical and magnetic media.

[0061] The instructions 624 may further be transmitted or received over a communications network 626 using a transmission medium by the network interface device 620 using any one of a number of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communications networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile telephone networks (e.g., cellular networks), plain old telephone service (POTS) networks, and wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards known as Wi-Fi®, the IEEE 802.16 family of standards known as WiMax®), the IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device 620 may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas for connecting to the communications network 626. In one example, the network interface device 620 may include multiple antennas for communicating wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions for execution by the machine 600, including digital or analog communication signals or other intangible media to facilitate communication of such software.

[0062] The method examples described herein may be at least partially machine or computer implemented. Some examples may include computer-readable or machine-readable media encoded with instructions operable to configure an electronic device to perform the methods described in the above examples. Such method implementations may include code such as microcode, assembly language code, higher level language code, and the like. Such code may include computer-readable instructions for performing various methods. The code may form some part of a computer program product. Furthermore, in some examples, the code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memory (RAM), read-only memory (ROM), and the like. EXAMPLES

[0063] The following non-limiting examples detail certain aspects of the present subject matter for solving the problems and providing the benefits, among others, discussed herein.

[0064] Example 1 is a gastrointestinal (GI) tract mucosa identification system for training a machine learning model for use in a computer-based clinical decision support system for supporting real-time selection of computer-aided detection and characterization modules trained for specialized anatomical regions. In this example, the GI tract mucosa identification system may include data of mucosa images and a computing system having a processing circuit and a memory device. The memory device may include instructions that, when executed by the processing circuit, cause the processing circuit to perform operations including labeling, training a machine learning model, and storing the machine learning model. The labeling may include labeling a plurality of mucosa images from a database based on a location within the GI tract to generate training data. Training the machine learning model may include using the training data to predict a location of an endoscope within the GI tract based on real-time mucosa imaging from the endoscope.

[0065] Example 2 is a method for training a machine learning model for use in a computer-based clinical decision support system that provides real-time selection of computer-aided detection and characterization modules trained for specialized identification or diagnosis of anomalies in a particular anatomical region. In this example, the method may include operations for receiving data, labeling the received data, training a machine learning model, and outputting the machine learning model. The received data may include data captured by an endoscope, including mucosal images from various portions of the GI tract. Labeling the received data may include labeling based on a location within the GI tract to generate training data. Training the machine learning model may be performed using the training data to train the model to predict a location of the endoscope within the GI tract based on real-time mucosal imaging from the endoscope.

[0066] In Example 3, the subject matter of Example 2 can optionally include receiving the data by receiving an endoscopic image including an esophageal mucosa, a gastric mucosa, and a duodenal mucosa.

[0067] In Example 4 of any one of Examples 2 and 3, the subject matter can optionally include training a machine learning model by training an upper GI tract model and a lower GI tract model.

[0068] In Example 5, the subject matter of Example 4 can optionally include training the lower GI tract model by using training data specific to the lower GI tract.

[0069] In Example 6, the subject matter of Example 5 can optionally include training data specific to the lower GI tract, including labeled endoscopic images from a group of images selected from one or more of the following anatomical areas: cecum, ascending colon, right colic flexure, transverse colon, left colic flexure, descending colon, sigmoid colon, rectum, and anal canal.

[0070] Example 7 is an endoscopic clinical support system for real-time activation of computer-aided detection and characterization modules trained for specialized identification or diagnosis of abnormalities in a patient's gastrointestinal (GI) tract. In this example, the system may include an endoscope and a clinical support computing device. The endoscope may include a sensor and an illumination component configured to generate medical image data. The clinical support computing device is communicatively coupled to the endoscope. The clinical support computing device also includes a processing circuit, a display device configured to display the medical image data and associated user interface graphics, and a memory including instructions that, when executed by the processing circuit, cause the processing circuit to perform operations including: activating a first mucosal identification module and a first plurality of detection and characterization modules based at least in part on detecting a connection to the endoscope; monitoring an image stream from the endoscope using the first mucosal identification module to identify a mucosal tissue type; and executing a first detection module from the first plurality of detection and characterization modules based on identifying the mucosal tissue type as the first mucosal tissue type. and processing an image from the image stream received from the endoscope using the first detection module to identify a region of interest for output to a display device, the region of interest identifying a potential anomaly in the image.

[0071] In Example 8, the subject matter of Example 7 may optionally include activating the first mucosal identification module by receiving endoscope type or model information that identifies the endoscope.

[0072] In Example 9, the subject matter of Example 8 may optionally include receiving endoscope type or model information, including receiving lighting information identifying different lighting types that the lighting component is capable of producing.

[0073] In Example 10, the subject matter of any one of Examples 7 to 9 may optionally include a memory further including instructions that, when executed by the processing circuit, cause the processing circuit to perform operations such as continuously monitoring images extracted from the image stream received from the endoscope using the first mucosal identification module, and executing a second detection module from the first plurality of detection and characterization modules based on the first mucosal identification module identifying the mucosal tissue type as a second mucosal tissue type.

[0074] In Example 11, the subject matter of Example 10 may optionally include executing the second detection module by outputting an indication to a display device that the endoscope has begun imaging a different region of the GI tract.

[0075] In Example 12, the subject matter of any one of Examples 7 to 11 may optionally include monitoring an image stream from the endoscope with a first mucosal identification module, which may include implementing a machine learning model trained at least in part based on training data including a plurality of labeled mucosal images from various locations within the GI tract for predicting a position of the endoscope within the GI tract based on real-time mucosal imaging from the endoscope.

[0076] In Example 13, the subject matter of any one of Examples 7 to 12 may optionally include monitoring the image stream using a first mucosa identification module, which may include periodically processing images extracted from the image stream using the first mucosa identification module.

[0077] In Example 14, the subject matter of any one of Examples 7 to 13 may optionally include executing a first detection module, which may include selecting a first imaging modality from a plurality of imaging modalities supported by the endoscope.

[0078] In Example 15, the subject matter of Example 14 may optionally include identifying the region of interest by selecting a second imaging modality from a plurality of imaging modalities supported by the endoscope.

[0079] In Example 16, the subject matter of Example 15 can optionally include a memory further including instructions that, when executed by the processing circuit, cause the processing circuit to perform an operation including executing a first characterization module from the plurality of detection and characterization modules based at least in part on identifying the region of interest.

[0080] In Example 17, the subject matter of Example 16 can optionally include executing the first characterization module by outputting to a display device an indication of the tissue classification and corresponding confidence score associated with the region of interest.

[0081] In Example 18, the subject matter of any one of Examples 14 to 17 may optionally include identifying the region of interest including displaying a user prompt on the display device providing an option to activate a second imaging modality from multiple imaging modalities supported by the endoscope.

[0082] Example 19 is a system for real-time activation of a computer-aided detection and characterization module trained for specialized identification or diagnosis of anomalies in a patient's gastrointestinal (GI) tract. The system may include one or more processing units and a computer-readable medium encoded with computer-executable instructions for causing the one or more processing units to perform operations such as: receiving image data generated by a medical imaging device during a consultation; providing a first portion of the image data to one or more anatomy identification modules configured to output a respective indication of whether the respective portion of the image data exhibits a first anatomy type or a second anatomy type at a first time during the consultation; receiving a first indication from the one or more anatomy identification modules that the first portion of the image data exhibits the first anatomy type; analyzing the first portion of the image data utilizing a first CAD model in response to the first indication, the first CAD model configured to generate annotations in association with anomalies shown in the images of the first anatomy type; providing a second portion of the image data to the one or more anatomy identification modules at a second time during the consultation subsequent to the first time. receiving a second indication from the one or more anatomy identification modules that a second portion of the image data indicates a second anatomy type, and in response to the second indication, analyzing the second portion of the image data utilizing a second CAD model, the second CAD model configured to generate annotations in association with abnormalities indicated in the image of the second anatomy type.

[0083] In Example 20, the subject matter of Example 19 may optionally include computer-executable instructions that further cause one or more processing units to receive an output from the first CAD model indicating detection of an anomaly within the first portion of the image data, and in response to the output from the first CAD model, analyze at least a portion of the first portion of the image data utilizing a third CAD model, the third CAD model configured to generate an output that classifies the anomaly detected by the first CAD model.

[0084] In Example 21, the subject matter of Example 20 can optionally include leveraging the third CAD model by activating an alternative illumination modality on the medical imaging device.

[0085] In Example 22, the subject matter of Example 21 can optionally include an alternative illumination modality that is a blue or red light narrowband imaging modality.

[0086] In Example 23, the subject matter of any one of Examples 20 to 22 may optionally include the third CAD model being configured to output a tissue classification and a confidence score associated with the tissue classification.

[0087] In Example 24, the subject matter of any one of Examples 19 to 23 can optionally include, where the first anatomical type is squamous epithelial mucosa corresponding to an esophageal anatomical region, and the second anatomical type is gastric mucosa corresponding to a gastric anatomical region.

[0088] In Example 25, the subject matter of any one of Examples 19 to 24 may optionally include, where the image data includes a sequence of a plurality of image frames generated during an endoscopic examination.

[0089] In Example 26, the subject matter of any one of Examples 19 to 25 may optionally include utilizing a first CAD module by outputting instructions of a first anatomical type.

[0090] In Example 27, of any one of Examples 19-26, the subject matter may optionally include utilizing a second CAD model, including outputting instructions of a second anatomical type. [Explanation of symbols]

[0091] 100 Software Architecture 104 Instrument Identification Module 112 Mucosal Identification Module 114 Anatomical Identification Module 122 Esophageal CADe Module 124 Gastric CADe Module 126 CADe Module 132 Esophageal CADx Module 134 Stomach CADx Module 138 CADx Modules 600 machines 602 Hardware Processor 604 Main Memory 606 Static Memory 608 Interlink 610 Display Unit 612 Alphanumeric Input Device 614 User Interface (UI) Navigation Devices 616 Storage Devices 618 Signal Generating Device 620 Network Interface Device 621 Sensor 622 Machine-readable media 626 Communication Network 628 Output Controller

Claims

1. 1. A gastrointestinal (GI) tract mucosa identification system for training machine learning models for use in a computer-based clinical decision support system for assisting in real-time selection of computer-aided detection and characterization modules trained for specialized anatomical regions, comprising: a database of mucosal images captured from various parts of the GI tract; A processing circuit; and a memory containing instructions that, when executed by the processing circuit, cause the processing circuit to: labeling a plurality of mucosal images from the database based on their location within the GI tract to generate training data; training a machine learning model to predict a position of the endoscope within the GI tract based on real-time mucosal imaging from the endoscope based at least in part on the training data; storing the machine learning model; A GI tract mucosa identification system that performs an operation including:

2. 1. A method for training a machine learning model for use in a computer-based clinical decision support system that provides real-time selection of computer-aided detection and characterization modules trained for specialized identification or diagnosis of anomalies within a particular anatomical region, comprising: receiving data captured by the endoscope, the data including mucosal images from various portions of the GI tract; labeling the received data based on location within the GI tract to generate training data; training the machine learning model to predict a position of the endoscope within the GI tract based on real-time mucosal imaging from an endoscope based at least in part on the training data; outputting the machine learning model; A method comprising:

3. The method of claim 2 , wherein the step of receiving data includes receiving endoscopic images including esophageal, gastric, and duodenal mucosa.

4. 3. The method of claim 2, wherein training the machine learning models includes training an upper GI tract model and a lower GI tract model.

5. The method of claim 4 , wherein training the lower GI tract model includes using training data specific to the lower GI tract.

6. The lower GI tract specific training data includes data for the following anatomical areas: Appendicitis, ascending colon, right colonic flexure, transverse colon, left colonic flexure, descending colon, sigmoid colon, rectum, and anal canal The method of claim 5, further comprising labeling the endoscopic image from a group of images selected from one or more of:

7. 1. An endoscopic clinical support system for real-time activation of a computer-aided detection and characterization module trained for specialized identification or diagnosis of anomalies within a gastrointestinal (GI) tract of a patient, comprising: an endoscope including sensor and illumination components configured to generate medical image data; a clinical support computing device communicatively coupled to the endoscope; the clinical support computing device comprising: A processing circuit; a display device configured to display the medical image data and associated user interface graphics; and a memory containing instructions that, when executed by the processing circuit, cause the processing circuit to: activating a first mucosa identification module and a first plurality of detection and characterization modules based at least in part on detecting a connection to the endoscope; monitoring an image stream from the endoscope using the first mucosal identification module to identify a mucosal tissue type; executing a first detection module from the first plurality of detection and characterization modules based on identifying the mucosal tissue type as a first mucosal tissue type; processing images from the image stream received from the endoscope using the first detection module to identify and output regions of interest to the display device, the regions of interest identifying potential anomalies within the images; A system that causes a system to perform an operation including:

8. 8. The system of claim 7, wherein activating the first mucosal identification module is based on receiving endoscope type or model information that identifies the endoscope.

9. 10. The system of claim 8, wherein receiving endoscope type or model information includes receiving lighting information identifying different lighting types that the lighting component is capable of producing.

10. The memory further includes instructions that, when executed by the processing circuit, cause the processing circuit to: continuously monitoring images extracted from the image stream received from the endoscope using the first mucosa identification module; executing a second detection module from the first plurality of detection and characterization modules based on the first mucosal identification module identifying the mucosal tissue type as a second mucosal tissue type; The system of claim 7 , further comprising:

11. 11. The system of claim 10, wherein executing the second detection module includes outputting an indication to the display device that the endoscope has begun imaging a different region of the GI tract.

12. 8. The system of claim 7, wherein monitoring the image stream from the endoscope using the first mucosa identification module includes implementing a machine learning model trained at least in part based on training data including a plurality of labeled mucosa images from various locations within the GI tract to predict a position of the endoscope within the GI tract based on real-time mucosa imaging from the endoscope.

13. 8. The system of claim 7, wherein monitoring the image stream using the first mucosa identification module includes periodically processing images extracted from the image stream using the first mucosa identification module.

14. The system of claim 7 , wherein executing the first detection module includes selecting a first imaging modality from a plurality of imaging modalities supported by the endoscope.

15. The system of claim 14 , wherein identifying the region of interest includes selecting a second imaging modality from the plurality of imaging modalities supported by the endoscope.

16. 16. The system of claim 15, wherein the memory further includes instructions that, when executed by the processing circuit, cause the processing circuit to perform operations including executing a first characterization module from the plurality of detection and characterization modules based at least in part on identifying the region of interest.

17. 17. The system of claim 16, wherein executing the first characterization module includes outputting to the display device an indication of a tissue classification and a corresponding confidence score associated with the region of interest.

18. 15. The system of claim 14, wherein identifying the region of interest includes displaying a user prompt on the display device providing an option to activate a second imaging modality from the plurality of imaging modalities supported by the endoscope.

19. 1. A system comprising: one or more processing units; and a computer-readable medium encoding computer-executable instructions that cause the one or more processing units to: receiving image data generated by a medical imaging device during a medical examination; providing a first portion of the image data to one or more anatomy identification modules configured to output a respective indication of whether the respective portion of the image data is indicative of a first anatomy type or a second anatomy type at a first time during the examination; receiving a first indication from the one or more anatomical identification modules that the first portion of the image data indicates the first anatomical type; analyzing, in response to the first instruction, the first portion of the image data utilizing a first CAD model, the first CAD model configured to generate annotations associated with abnormalities shown in the first anatomical type images; providing a second portion of the image data to the one or more anatomy identification modules at a second time during the examination subsequent to the first time; receiving a second indication from the one or more anatomical identification modules that the second portion of the image data indicates the second anatomical type; and in response to the second instruction, analyzing the second portion of the image data utilizing a second CAD model, the second CAD model configured to generate annotations associated with abnormalities shown in the second anatomical type images.

20. The computer executable instructions cause the one or more processing units to: receiving an output from the first CAD model indicative of a detection of an anomaly within the first portion of the image data; 20. The system of claim 19, further comprising: analyzing, in response to the output from the first CAD model, at least a portion of the first portion of the image data utilizing a third CAD model, the third CAD model configured to generate an output that classifies the anomaly detected by the first CAD model.

21. The system of claim 20 , wherein leveraging the third CAD model includes activating an alternative illumination modality on the medical imaging device.

22. 22. The system of claim 21, wherein the alternative illumination modality is a blue or red light narrowband imaging modality.

23. 21. The system of claim 20, wherein the third CAD model is further configured to output a tissue classification and a confidence score associated with the tissue classification.

24. 20. The system of claim 19, wherein the first anatomy type is squamous epithelial mucosa corresponding to an esophageal anatomy region and the second anatomy type is gastric mucosa corresponding to a gastric anatomy region.

25. 20. The system of claim 19, wherein the image data comprises a sequence of image frames generated during the endoscopic examination.

26. 20. The system of claim 19, wherein utilizing the first CAD module includes outputting an indication of the first anatomical type.

27. 20. The system of claim 19, wherein utilizing the second CAD model includes outputting an indication of the second anatomical type.

Images