System and method for identifying polyp images

Abstract

Systems and methods for identifying images containing polyps are disclosed. An example method for identifying images includes accessing images of a gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein: each of the images is suspected to include a polyp and is associated with a probability of containing a polyp, and the images include seed images, wherein each seed image is associated with one or more of the images. The images associated with each seed image are identified as suspected to include the same polyp as the associated seed image. The method includes applying a polyp detection system to the seed images to identify seed images including polyps, wherein the polyp detection system is applied to each seed image based on the images associated with the seed image and probabilities associated with the seed image and the associated images.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 075,795, filed on September 8, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to image analysis of in vivo images of the gastrointestinal tract (GIT), and more specifically to systems and methods for identifying images of polyps in the GIT. Background Technology

[0004] Capsule endoscopy (CE) allows for the examination of the entire gut tract (GIT) under endoscopy. Some capsule endoscopy systems and methods are designed to examine specific parts of the GIT, such as the small intestine (SB) or colon. CE is a non-invasive procedure that does not require hospitalization, and patients can continue most of their daily activities while the capsule is in their body.

[0005] In a typical CE procedure, a physician refers the patient to the procedure. The patient then arrives at a healthcare facility (e.g., a clinic or hospital) to perform the procedure. A healthcare professional (e.g., a nurse or physician) supervises the patient swallowing a capsule approximately the size of a multivitamin at the facility and provides the patient with wearable devices, such as sensor straps and a recorder placed in a small pouch, as well as a strap that must be placed around the patient's shoulder. The wearable device typically includes a storage device. Instructions and / or directives can be given to the patient, who is then discharged to resume their daily activities.

[0006] The capsule captures images as it travels naturally through the gut. The images and additional data (e.g., metadata) are then transmitted to a recorder worn by the patient. The capsule is typically disposable and is expelled naturally with bowel movements. Procedural data (e.g., the captured images or portions thereof, along with additional metadata) is stored on the wearable device's storage.

[0007] Patients typically return their wearable devices, along with the protocol data stored on them, to the healthcare facility. The protocol data is then downloaded to a computing device, usually located at the facility, which has engine software stored on it. This engine then processes the received protocol data into a compiled study (or “study”). A study typically includes thousands of images (approximately 6,000). The number of images to be processed is usually in the tens of thousands, averaging around 90,000.

[0008] The radiologist (who may be the protocol supervisor, the attending physician, or the referring physician) accesses the study via the radiologist application. The radiologist then reviews the study, evaluates the protocol, and provides their input via the radiologist application. Because the radiologist needs to review thousands of images, the study reading time can typically average between half an hour and an hour, and the reading task can be cumbersome. The radiologist application then generates a report based on the compiled study and the radiologist's input. On average, it takes one hour to generate a report. This report may include, for example, images of interest selected by the radiologist, such as images identified as containing symptoms; data based on the protocol (i.e., the study); and / or recommendations for follow-up and / or treatment provided by the radiologist to assess or diagnose the patient's medical condition. The report can then be forwarded to the referring physician. The referring physician can then use the report to determine the necessary follow-up or treatment. Summary of the Invention

[0009] To the extent consistent, any or all aspects detailed herein may be used in conjunction with any or all other aspects detailed herein. Aspects of this disclosure relate to images for identifying polyps with a high degree of confidence. Due to the high degree of confidence, aspects of this disclosure relate to the automatic use of identified images without human prompting or intervention, and / or to presenting identified images to healthcare professionals when such images may have been missed during human review, and / or to decisions to veto other tools that may incorrectly designate identified images.

[0010] According to various aspects of this disclosure, a method for identifying images including polyps includes: accessing a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device during CE procedures, wherein: each of the plurality of images is suspected of including a polyp and is associated with a probability of including a polyp, the plurality of images include seed images and each seed image is associated with one or more images of the plurality of images, wherein the one or more images are associated with each seed image identified as potentially including a polyp identical to the associated seed image; and applying a polyp detection system to these seed images to identify seed images including polyps, wherein the polyp detection system is applied to each of the seed images based on one or more images associated with the seed image and the probability associated with the seed image and the one or more associated images.

[0011] In various embodiments of the method, the method includes identifying images among the plurality of images that include polyps with a size equal to or greater than a predefined size, wherein each of the plurality of images is further associated with an estimated size of a suspected polyp contained in each image, and wherein a polyp detection system is further applied to each of the seed images based on an estimated polyp size associated with a seed image and one or more images associated with the seed image.

[0012] In various embodiments of the method, the procedure is determined to be insufficient and excluded, and at least one seed image is identified as including polyps of a size equal to or greater than a predefined size or including a predefined number of polyps of a size equal to or greater than a predefined size, and the method includes vetoing the exclusion of the procedure.

[0013] In various implementations of the method, the polyp detection system includes at least one of the following: one or more positive filters, one or more negative filters, one or more classical machine learning systems, or a combination thereof.

[0014] In various embodiments of the method, the input to the one or more classical machine learning systems, the one or more positive filters, or the one or more negative filters includes at least one of the following: the probability of a seed image containing polyps, the number of images associated with the seed image, the number of images associated with a seed image having a probability of containing polyps according to a predefined threshold, or a combination thereof.

[0015] In various implementations of this method, the one or more images associated with each seed image are determined by applying a tracker to neighboring images to track suspected polyps contained in each seed image, or by using a classification system that compares the seed images with neighboring images.

[0016] In various embodiments of this method, the multiple images of the gastrointestinal tract (GIT) accessed by the method are images from CE protocol studies.

[0017] In various embodiments of the method, the method includes selecting a seed image from the plurality of images.

[0018] In various embodiments of the method, the method includes providing instructions to the CE protocol referral physician to refer the CE protocol subject to the colonoscopy protocol based on a seed image identified as including a polyp.

[0019] In various embodiments of the method, the method includes: for each of the plurality of images: applying a classical machine learning system configured to provide a probability that the image contains a polyp based on input features corresponding to the image, and accessing a soft interval of the classical machine learning system corresponding to the image; and determining, without human intervention, whether to recommend a colonoscopy based on the soft interval of the plurality of images.

[0020] In various embodiments of the method, the method includes a mapping of the probability of accessing a soft septum to an image containing a polyp, wherein the determination of whether to recommend a colonoscopy is also based on this mapping of the probability of the soft septum to an image containing a polyp.

[0021] In various embodiments of the method, the method includes: for each of the plurality of images, accessing an estimated polyp size of the image, the estimated polyp size being generated based on the image; and accessing a mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size, wherein the determination of whether to recommend a colonoscopy is also based on the estimated polyp size and the mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size.

[0022] In various embodiments of the method, the method includes displaying a seed image identified as including a polyp.

[0023] In various embodiments of the method, the method includes providing treatment recommendations based on seed images identified as containing polyps.

[0024] In various embodiments of the method, the method includes displaying a seed image and indicating a seed image identified as containing a polyp.

[0025] In various embodiments of the method, the method includes: displaying at least a seed image to a user; receiving a selection of an image from the displayed image by the user; determining at least one unselected image that was not selected by the user and is in the seed image identified as containing a polyp; and presenting the at least one unselected image to the user.

[0026] In various implementations of this method, the image selected by the user is the image chosen to be included in the CE procedure report.

[0027] In various implementations of this method, once a request to generate a report is received, the method performs the action of presenting at least one unselected image to the user.

[0028] According to various aspects of this disclosure, a method for identifying images includes: accessing a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the possibility of containing polyps; applying at least one filter to the plurality of images, wherein the at least one filter includes at least one of the following: a positive filter configured to identify images as containing polyps, or a negative filter configured to identify images as not containing polyps; and providing information based on at least one of the following: at least one image among the plurality of images identified by the at least one filter, or at least one image among the plurality of images not identified by the at least one filter.

[0029] In various implementations of this method, negative filters are configured to identify images as images of the body exit portion of the GIT based on these images, without specifying that they contain polyps.

[0030] In various embodiments of the method, the negative filter is configured to identify images based on whether these images are evaluated as images of at least one of the ileocecal valve or hemorrhoidal plexus, without specifying that they contain polyps.

[0031] In various implementations of this method, negative filters are configured to identify images based on the assumption that these images contain polyps with an estimated polyp size below a threshold size, without specifying them as containing polyps.

[0032] In various embodiments of the method, the method further includes: for each of the plurality of images, accessing the image trajectory of the image.

[0033] In various implementations of this method, the negative filter is configured to identify images without specifying them as containing polyps, based on the image trajectory of an image for which the probability of the presence of a polyp is higher than a threshold for only one image.

[0034] In various implementations of this method, a positive filter is configured to identify an image as containing a polyp based on the image trajectory.

[0035] In various implementations of this method, a positive filter is configured to identify images as containing polyps based on image trajectories of images for which the probability of polyp presence is higher than a threshold for at least a threshold number of images.

[0036] According to various aspects of this disclosure, a system for identifying images includes one or more processors and at least one memory storing instructions. When executed by the one or more processors, these instructions cause the system to: access a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the possibility of containing polyps; apply at least one filter to the plurality of images, wherein the at least one filter includes at least one of the following: a positive filter configured to identify images as containing polyps, or a negative filter configured to identify images as not containing polyps; and provide information based on at least one of the following: at least one image among the plurality of images identified by the at least one filter, or at least one image among the plurality of images not identified by the at least one filter.

[0037] In various implementations of this system, negative filters are configured to identify images as images of the body exit portion of the GIT based on these images, without specifying that they contain polyps.

[0038] In various implementations of the system, the negative filter is configured to identify images based on whether these images are evaluated as images containing at least one of the ileocecal valve or hemorrhoidal plexus, without specifying them as containing polyps.

[0039] In various implementations of the system, negative filters are configured to identify images based on the assumption that these images contain polyps with an estimated polyp size below a threshold size, without specifying them as containing polyps.

[0040] In various implementations of the system, these instructions, when executed by the one or more processors, further enable the system to access the image trajectory of each of the plurality of images.

[0041] In various implementations of the system, the negative filter is configured to identify images based on the image trajectory of an image where the probability of a polyp being present in only one image is higher than a threshold, without specifying that the image contains a polyp.

[0042] In various implementations of this system, a positive filter is configured to identify images as containing polyps based on image trajectories.

[0043] In various implementations of the system, a positive filter is configured to identify images as containing polyps based on the image trajectory of images for which the probability of polyp presence is higher than a threshold for at least a threshold number of images.

[0044] According to various aspects of this disclosure, a method for identifying images includes: accessing a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the probability of containing polyps; for each of the plurality of images: applying a classical machine learning system configured to provide an indication of whether the image contains polyps based on input features corresponding to the image; and based on at least one image presentation information of the plurality of images, the information having an indication provided by the classical machine learning system that contains polyps that meet a confidence threshold.

[0045] In various embodiments of the method, the method further includes: for each of the plurality of images: accessing the image trajectory of the image.

[0046] In various implementations of this method, the input features corresponding to the image include at least one of the following: the trajectory length of the image trajectory, or the number of images in the image trajectory with a polyp presence fraction higher than a threshold.

[0047] In various implementations of this method, the input features corresponding to the image include the index difference between the index of the image and the index of the ileocecal valve image.

[0048] In various embodiments of this method, the input features corresponding to the image include the segment number of the colon segment in which the image is captured.

[0049] In various implementations of this method, the classic machine learning classifier is a multinomial support vector machine.

[0050] According to various aspects of this disclosure, a system for identifying images includes one or more processors and at least one memory storing instructions. When executed by the one or more processors, these instructions cause the system to: access a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the probability of containing polyps; for each of the plurality of images: apply a classical machine learning system configured to provide an indication of whether the image contains polyps based on input features corresponding to the image; and based on at least one image presentation information from the plurality of images, the information having an indication provided by the classical machine learning system that contains polyps meeting a confidence threshold.

[0051] In various implementations of the system, these instructions, when executed by the one or more processors, further enable the system to access the image trajectory of each of the plurality of images.

[0052] In various implementations of the system, the input features corresponding to the image include at least one of the following: the trajectory length of the image trajectory, or the number of images in the image trajectory with a polyp presence fraction above a threshold.

[0053] In various implementations of this system, the input features corresponding to the image include the index difference between the image index and the image index of the ileocecal valve.

[0054] In various implementations of this system, the input features corresponding to the image include the segment number of the colon segment in which the image is captured.

[0055] In various implementations of this system, the classic machine learning classifier is a multinomial support vector machine.

[0056] According to various aspects of this disclosure, a method for identifying images includes: accessing a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the possibility of containing polyps; applying at least one filter to the plurality of images, wherein the at least one filter includes at least one of: a positive filter configured to identify images as containing polyps; or a negative filter configured to identify images as not containing polyps; providing at least one unfiltered image by selecting at least one image not identified by the at least one filter from the plurality of images; for each of the at least one unfiltered images: applying a classical machine learning system configured to provide an indication of whether the unfiltered image contains polyps based on input features corresponding to the unfiltered image; and based on at least one image presentation information of the at least one unfiltered image, the information having an indication provided by the classical machine learning system that contains polyps that meet a confidence threshold.

[0057] In various embodiments of the method, the method further includes: generating a capsule endoscopy report for presentation to a clinician without human intervention, the capsule endoscopy report including at least one of the following: at least one unfiltered image containing an indication of polyps that meet a confidence threshold provided by the classical machine learning system, or at least one image identified by a positive filter.

[0058] In various embodiments of the method, the method further includes: receiving a user's selection of images from the plurality of images; determining at least one unselected image from the at least one unfiltered image that was not selected by the user and has an indication provided by the classical machine learning system that includes a polyp satisfying a confidence threshold; and presenting the at least one unselected image to the user.

[0059] According to various aspects of this disclosure, a system for identifying images includes one or more processors and at least one memory storing instructions. When executed by the one or more processors, these instructions cause the system to: access a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the possibility of containing polyps; apply at least one filter to the plurality of images, wherein the at least one filter includes at least one of: a positive filter configured to identify images as containing polyps; or a negative filter configured to identify images as not containing polyps; provide at least one unfiltered image by selecting at least one image not identified by the at least one filter from the plurality of images; for each of the at least one unfiltered images: apply a classical machine learning system configured to provide an indication of whether the unfiltered image contains polyps based on input features corresponding to the unfiltered image; and based on at least one image presentation information of the at least one unfiltered image, the information having an indication provided by the classical machine learning system that contains polyps satisfying a confidence threshold.

[0060] In various implementations of the system, these instructions, when executed by the one or more processors, further cause the system to generate a capsule endoscopy report for presentation to a clinician without human intervention, wherein the capsule endoscopy report includes at least one of the following: at least one unfiltered image containing an indication of a polyp that meets a confidence threshold provided by the classical machine learning system, or at least one image identified by a positive filter.

[0061] In various embodiments of the system, when executed by the one or more processors, these instructions further cause the system to: receive a user's selection of images from the plurality of images; determine at least one unselected image from the at least one unfiltered image that was not selected by the user and has an indication provided by the classical machine learning system that includes a polyp satisfying a confidence threshold; and present the at least one unselected image to the user.

[0062] According to an aspect of this disclosure, a computer-implemented method for recommending a colonoscopy includes: accessing multiple images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the multiple images have the potential to contain polyps; for each of the multiple images: applying a classical machine learning system configured to provide an indication of whether the image contains polyps based on input features corresponding to the image, and accessing a soft interval of the classical machine learning system corresponding to the image; and determining whether to recommend a colonoscopy based on the soft interval of the multiple images without human intervention.

[0063] In various embodiments of the method, the method also includes a mapping of the probability of accessing a soft septum to an image containing a polyp, wherein the determination of whether to recommend a colonoscopy is also based on this mapping of the probability of the soft septum to an image containing a polyp.

[0064] In various embodiments of the method, the method further includes: for each of the plurality of images, accessing an estimated polyp size of the image, wherein the estimated polyp size is generated based on the image; and accessing a mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size, wherein the determination of whether to recommend colonoscopy is also based on the estimated polyp size and the mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size.

[0065] According to an aspect of this disclosure, a system for recommending a colonoscopy includes one or more processors and at least one memory storing instructions. When executed by the one or more processors, these instructions cause the system to: access a plurality of images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, wherein the plurality of images have the probability of containing polyps; for each of the plurality of images: apply a classical machine learning system configured to provide an indication of whether the image contains polyps based on input features corresponding to the image, and access a soft interval of the classical machine learning system corresponding to the image; and determine, without human intervention, whether to recommend a colonoscopy based on the soft interval of the plurality of images.

[0066] In various implementations of the system, when executed by the one or more processors, these instructions further cause the system to access a mapping of the probability of a soft septum to an image containing a polyp, wherein the determination of whether to recommend a colonoscopy is also based on this mapping of the probability of a soft septum to an image containing a polyp.

[0067] In various embodiments of the system, these instructions, when executed by the one or more processors, further cause the system to: for each of the plurality of images, access an estimated polyp size of the image, wherein the estimated polyp size is generated based on the image; and access a mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size, wherein the determination of whether to recommend a colonoscopy is also based on the estimated polyp size and the mapping of the estimated polyp size to the probability that the actual polyp size is at least a predefined size. Attached Figure Description

[0068] The above and other aspects and features of this disclosure will become more apparent when considered in conjunction with the accompanying drawings, in view of the following detailed description, wherein similar reference numerals identify similar or identical elements.

[0069] Figure 1 A diagram illustrating the gastrointestinal tract (GIT);

[0070] Figure 2 A block diagram of an exemplary system for analyzing medical images captured in vivo via capsule endoscopy (CE) procedures, according to various aspects of this disclosure;

[0071] Figure 3 A block diagram of an exemplary computing system that can be used with the system disclosed herein;

[0072] Figure 4 A diagram showing the colon;

[0073] Figure 5 A diagram illustrating an exemplary deep learning neural network according to various aspects of this disclosure;

[0074] Figure 6 A block diagram illustrating exemplary operations for selecting colon images containing colon polyps with a high degree of confidence, according to various aspects of this disclosure;

[0075] Figure 7 A diagram of selected seed images according to various aspects of this disclosure;

[0076] Figure 8 A block diagram illustrating exemplary operations for selecting an image of a colon containing polyps according to various aspects of this disclosure;

[0077] Figure 9 A block diagram of another exemplary operation for selecting a colon image containing polyps according to various aspects of this disclosure;

[0078] Figure 10 A diagram illustrating exemplary image trajectories of seed images according to various aspects of this disclosure;

[0079] Figure 11 A diagram of exemplary image trajectories processed by a positive filter according to various aspects of this disclosure;

[0080] Figure 12 Examples of display screens and user interfaces provided for clinicians to view and / or select colon images that may contain colon polyps, in accordance with various aspects of this disclosure;

[0081] Figure 13 Exemplary display screens and user interfaces for presenting suggested images containing polyps to clinicians, according to various aspects of this disclosure;

[0082] Figure 14 An exemplary display screen for a fully automated process of presenting a selected colon image containing polyps, according to various aspects of this disclosure;

[0083] Figure 15A graph illustrating the probability of an image containing polyps based on soft intervals, according to various aspects of this disclosure;

[0084] Figure 16 A graph illustrating the probability of an image containing a polyp of at least 6 mm in size, according to various aspects of this disclosure; and

[0085] Figure 17 A block diagram illustrating another exemplary operation for selecting colon images containing colon polyps with a high degree of confidence, according to various aspects of this disclosure. Detailed Implementation

[0086] This disclosure relates to systems and methods for identifying images of polyps captured in vivo by a capsule endoscopy (CE) device with a high degree of confidence. Due to the high confidence, aspects of this disclosure relate to the automatic use of the identified images without human prompting or intervention, and / or to presenting the identified images to a healthcare professional when such images might have been missed during human review, and / or to decisions to reject other tools that might incorrectly designate the identified images. In various aspects, decisions regarding subject images utilize information about the images associated with them, such as information about the image "trajectory," which will be discussed in more detail below. In various aspects, decisions regarding subject images use weights such that not all images are considered equally. Aspects of this disclosure relate to deep machine learning in classification / detection to achieve relatively high sensitivity and specificity, and aspects of this disclosure use heuristic and / or "classical" machine learning (defined later) to optimize results and increase sensitivity and / or specificity.

[0087] In the following detailed description, specific details are set forth in order to provide a thorough understanding of this disclosure. However, those skilled in the art will understand that this disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure this disclosure. Some features or elements described with respect to one system may be combined with features or elements described with respect to other systems. For clarity, the discussion of the same or similar features or elements may not be repeated.

[0088] While this disclosure is not limited in this respect, discussions using terms such as, for example, “processing,” “computing,” “calculating,” “determining,” “establishing,” “analyzing,” “checking,” etc., may refer to the operation and / or process of a computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data representing physical (e.g., electronic) quantities in the registers and / or memory of a computer into other data representing physical quantities in a non-transitory storage medium similarly represented in the registers and / or memory of a computer or that may store instructions for performing the operation and / or process. While this disclosure is not limited in this respect, the terms “multiple” and “a plurality” as used herein may include, for example, “multiple” or “two or more.” The terms “multiple” or “a plurality” may be used throughout the specification to describe two or more components, devices, elements, units, parameters, etc. This set of terms, when used herein, may include one or more. Unless explicitly stated otherwise, the methods described herein are not limited to a particular order or sequence. Furthermore, some of the methods or elements thereof may occur or be performed simultaneously, at the same point in time, or in parallel.

[0089] As used herein with respect to an image, the term “location” and its derivatives may refer to the estimated location of a capsule along a GIT (e.g., a colon) when the image is captured, or the estimated location of a portion of the GIT shown in the image along the GIT.

[0090] A type of CE protocol may be determined, in particular, based on the part of the GIT of interest to be imaged (e.g., the colon) or based on a specific purpose (e.g., for examining the status of GI diseases such as Crohn's disease, or for colon cancer screening).

[0091] The terms screen, view, and display are used interchangeably in this document and can be understood in the context of the specific text.

[0092] Unless otherwise expressly stated, the terms “around” or “adjacent” as used herein with respect to an image (e.g., an image surrounding or adjacent to another image) may refer to spatial and / or temporal characteristics. For example, an image surrounding or adjacent to another image may be an image whose estimated location is close to other images along the GIT and / or an image captured near the capture time of another image, within a certain threshold, such as within one or two centimeters, or within one, five, or ten seconds.

[0093] Depending on the context, the terms "GIT" and "part of GIT" can refer to or include each other. Therefore, the term "part of GIT" can also refer to the entire GIT, and the term "GIT" can also refer to only a part of GIT.

[0094] The terms “image” and “frame” may refer to or include the other, and may be used interchangeably in this disclosure to refer to a single capture performed by an imaging device. For convenience, the term “image” may be used more frequently in this disclosure, but it will be understood that references to an image should also apply to a frame.

[0095] The term "classical machine learning" refers to machine learning that involves feature selection or feature engineering for the inputs of machine learning.

[0096] The term "soft margin" refers to the continuous output of a classifier (e.g., a classic machine learning algorithm) relative to the distance between the example and the classifier's separating hyperplane / classification boundary. Soft margins can be used to evaluate how confident a classifier is in its decisions. The higher the absolute value of the soft margin, the further away from the classification boundary it is from, and the more confident its decision. The term "hard margin" refers to the classification decision made by applying a threshold (e.g., 0) to the soft margin and determining which category each example belongs to.

[0097] The term "clinician" can refer to any healthcare provider or practitioner, including any physician, such as a gastroenterologist, primary care physician, or referring physician.

[0098] refer to Figure 1 The diagram illustrates the GIT 100. The GIT 100 is an organ system of humans and other animals. The GIT 100 typically includes a mouth 102 for ingesting food, salivary glands 104 for producing saliva, an esophagus 106 through which food passes with the aid of contractions, a stomach 108 for secreting enzymes and gastric acid to aid digestion, a liver 110, a gallbladder 112, a pancreas 114, a small intestine 116 (“SB”) for absorbing nutrients, and a colon 400 (e.g., the large intestine) for storing water and waste as feces before defecation. The colon 400 typically includes an appendix 402, a rectum 428, and an anus 430. Food ingested through the mouth is digested by the GIT to absorb nutrients, and the remaining waste is expelled as feces through the anus 430.

[0099] Studies of different parts of the GIT 100 (e.g., colon 400, esophagus 106, and / or stomach 108) can be presented via a suitable user interface. As used herein, the term "study" refers to and includes studies from CE imaging devices (e.g., Figure 2(212) At least one set of images selected during a single CE procedure performed on a specific patient and images captured at a specific time, and optionally information other than images may also be included. The type of procedure performed determines which part of the GIT 100 is of interest. Examples of the types of procedures performed include, but are not limited to, small bowel procedures, colon procedures, small and colon procedures, procedures designed to specifically display or examine the small bowel, procedures designed to specifically display or examine the colon, procedures designed to specifically display or examine the colon and small bowel, or procedures that display or examine the entire GIT (esophagus, stomach, SB, and colon).

[0100] Figure 2 A block diagram of a system for analyzing medical images captured in vivo via CE procedures is shown. The system typically includes a capsule system 210 configured to capture images from a GIT (Glass In-Vitro Illustrated) system and a computing system 300 (e.g., a local system and / or a cloud system) configured to process the captured images.

[0101] Capsule system 210 may include an ingestible CE imaging device 212 (e.g., a capsule) configured to capture images of the GIT as the CE imaging device 212 travels through it. These images may be stored on the CE imaging device 212 and / or transmitted to a receiving device 214 (typically including an antenna). In some capsule systems 210, the receiving device 214 may be located on a patient who has ingested the CE imaging device 212 and may take the form, for example, a band worn by the patient or a patch attached to the patient.

[0102] Capsule system 210 can be communicatively coupled to computing system 300 and can transmit captured images to computing system 300. Computing system 300 can use image processing techniques, machine learning techniques and / or signal processing techniques, and other techniques to process the received images. Computing system 300 may include local computing devices located at the patient and / or the patient's treatment facility, a cloud computing platform provided by cloud services, or a combination of local computing devices and a cloud computing platform.

[0103] In cases where the computing system 300 includes a cloud computing platform, images captured by the capsule system 210 can be transmitted online to the cloud computing platform. In various embodiments, images can be transmitted via a receiving device 214 worn or carried by the patient. In various embodiments, images can be transmitted via the patient's smartphone or via any other device connected to the Internet and coupled to the CE imaging device 212 or the receiving device 214.

[0104] Figure 3A block diagram of an exemplary computing system 300 that can be used with the image analysis system of this disclosure is shown. The computing system 300 may include a processor or controller 305, which may be or include, for example, one or more central processing unit processors (CPUs), one or more graphics processing units (GPUs or GPGPUs), a chip or any suitable computing or arithmetic device, an operating system 215, memory 320, storage device 330, input device 335, and output device 340. For CE imaging device 212 ( Figure 2 Modules or equipment for collecting or receiving medical images (e.g., a receiver worn on a patient) or for displaying or selecting images for display (e.g., a workstation) may be or include Figure 3 The computing system 300 shown may be executed by the computing system. The communication component 322 of the computing system 300 may allow communication with remote or external devices, for example, via the Internet or another network, via radio, or via a suitable network protocol (such as File Transfer Protocol (FTP)).

[0105] The computing system 300 includes an operating system 315, which may be or may include any code segment designed and / or configured to perform tasks involving coordinating, scheduling, arbitrating, supervising, controlling, or otherwise managing the operations of the computing system 300 (e.g., the execution of a scheduler). The memory 320 may be or may include, for example, random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffers, short-term memory cells, long-term memory cells, or other suitable memory cells or storage units. The memory 320 may be or may include multiple possibly different memory cells. The memory 320 may store, for example, instructions for executing methods (e.g., executable code 325) and / or data such as user responses, interrupts, etc.

[0106] Executable code 325 may be any executable code, such as an application, program, process, task, or script. Executable code 325 may be executed by controller 305 under the control of operating system 315. For example, execution of executable code 325 may cause the display or selection of a medical image, as described herein. In some systems, more than one computing system 300 or components of computing system 300 may be used for the various functions described herein. One or more computing systems 300 or components of computing system 300 may be used for the various modules and functions described herein. Devices including components similar to or different from those included in computing system 300 may be used, and these devices may be connected to a network and used as a system. One or more processors 305 may be configured to perform the methods of this disclosure by, for example, executing software or code. Storage device 330 may be or may include, for example, a hard disk drive, floppy disk drive, compact disc (CD) drive, CD recordable (CD-R) drive, universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. Data such as instructions, codes, medical images, and image streams can be stored in storage device 330 and loaded from storage device 330 into memory 320, where it can be processed by controller 305. In some embodiments, this can be omitted. Figure 3 Some of the components shown.

[0107] Input device 335 may include, for example, a mouse, keyboard, touchscreen, or touchpad, or any suitable input device. It should be understood that any suitable number of input devices may be operatively coupled to computing system 300. Output device 340 may include one or more monitors, screens, displays, speakers, and / or any other suitable output devices. It should be understood that any suitable number of output devices may be operatively coupled to computing system 300, as shown in box 340. Any suitable input / output (I / O) device may be operatively coupled to computing system 300, such as a wired or wireless network interface card (NIC), modem, printer or fax machine, universal serial bus (USB) device, or external hard drive, and may be included in input device 335 and / or output device 340.

[0108] include Figure 3 Multiple computer systems 300 comprising some or all of the components shown may be used with the described system and method. For example, CE imaging equipment 212, receivers, cloud-based systems, and / or workstations or portable computing devices for displaying images may include... Figure 3 Some or all of the components of a computer system. Including, for example... Figure 3The computing system 300's cloud platform (e.g., a remote server) can receive procedural data such as images and metadata, process and generate studies, and also display the generated studies for physician review (e.g., on a web browser running on a workstation or portable computer). The "on-premise" option allows the use of workstations or local servers within the medical facility to store, process, and display images and / or studies.

[0109] According to some aspects of this disclosure, a user (e.g., a physician) can build his or her understanding of a case by reviewing images (e.g., captured by CE imaging device 212) automatically selected as potentially interesting images. In some systems of this disclosure, a relatively small number of images from the captured images are displayed according to the case for the user to review. "Relatively small number" means, in contrast to current methods, at most approximately several hundred, or at least on average, an image video stream that typically includes thousands of images (e.g., about 6,000 images) according to a case. In some systems, only a few hundred images are displayed for the user to review. In some systems, the number of images displayed for the user to review is at most about 1,000. Browsing a relatively small number of static images significantly reduces the user's review process, decreases case reading time, and leads to a better diagnosis, compared to viewing or reviewing thousands of images. Aspects of an exemplary user interface for displaying a study are described in co-pending international patent application publication number WO / 2020 / 079696 entitled “Systems and Methods for Generating and Displaying a Study of a Stream of In-VivoImages,” the entire contents of which are incorporated herein by reference. A computing system 300 and a capsule system (210) are described in co-pending U.S. provisional application number 62 / 867,050 entitled “Systems and Methods For Capsule Endoscopy Procedure.” Figure 2 Other aspects thereof, the entire contents of which are incorporated herein by reference.

[0110] refer to Figure 4 The diagram shows the colon 400. The colon 400 absorbs water, and any remaining waste is stored as feces before being expelled through defecation. For example, the colon 400 can be divided into five anatomical segments: cecum 404, right or ascending colon 410, transverse colon 416, left or descending colon 422 (e.g., left colon-sigmoid colon 424), and rectum 428.

[0111] The ileum 408 is the last part of the small intestine, leading to the cecum 404, and is separated from the cecum 404 by a muscular flap called the ileocecal valve (ICV) 406. The cecum 404 is the first part of the colon 400. The cecum 404 includes the appendix 402. The next part of the colon 400 is the ascending colon 410. The ascending colon 410 connects to the small intestine through the cecum 404. The ascending colon 410 extends upward through the abdominal cavity toward the transverse colon 416.

[0112] The transverse colon 416 is the section of the colon 400 from the hepatic flexure (also known as the right colic flexure 414) (the turn of the colon 400 through the liver) to the splenic flexure (also known as the left colic flexure 418) (the turn of the colon 400 through the spleen). The transverse colon 416 hangs over the stomach, attached to it by a large peritoneal fold called the greater omentum. Posteriorly, the transverse colon 416 connects to the posterior abdominal wall via the mesentery called the transverse middle colon.

[0113] The descending colon 422 is the portion of colon 400 that begins at the left bend of the colon 418 and extends to the sigmoid colon 426. One function of the descending colon 422 in the digestive system is to store feces that will be emptied into the rectum. The descending colon 422 is also called the distal colon because it extends further along the gastrointestinal tract than the proximal colon. Gut microbiota are typically very concentrated in this region. The sigmoid colon 426 is the portion of colon 400 that follows the descending colon 422 and precedes the rectum 428. The name sigmoid means S-shaped. The walls of the sigmoid colon 426 are muscular and contract to increase air pressure within colon 400, causing feces to move into the rectum 428. The sigmoid colon 426 is supplied with blood by several branches of the sigmoid colic artery (usually between 2 and 6).

[0114] Rectus 428 is the last part of colon 400. Rectus 428 holds the formed feces, waiting to be expelled through defecation.

[0115] CE Imaging Equipment 212 ( Figure 2 It can be used to image the interior of the colon 400. Entry into the colon 400 from the small intestine occurs via the ICV 406. Typically, after entering the colon 400 via the ICV 406, the CE imaging device 212 enters the cecum 404. However, sometimes the CE imaging device 212 misses the cecum 404 and enters directly into the ascending colon 410. The colon 400 can be wide enough to allow for virtually unrestricted movement of the CE imaging device 212. The CE imaging device 212 can rotate and roll. The CE imaging device 212 can remain in one place for a long time, or it can move very quickly through the colon 400.

[0116] Typically, the division of the GIT into multiple anatomical segments can be performed, for example, based on recognizing that the CE imaging device 212 has traversed between different anatomical segments. This recognition can be performed, for example, based on machine learning techniques. The division of a GIT image into a GIT portion of a captured image is solved in co-pending U.S. Provisional Application No. 63 / 018,890, and the division of a colon image into a colon portion of a captured image is solved in co-pending U.S. Provisional Application No. 63 / 018,878. The entire contents of each of these two co-pending patent applications are incorporated herein by reference. Other techniques for dividing a GIT image into a GIT portion or colon portion of a captured image will be understood by those skilled in the art.

[0117] The following description relates to images of the colon captured by a capsule endoscopy device. Such colon images may be part of a GIT image stream and may be selected from the GIT image stream using techniques from a common pending application or other methods that will be understood by those skilled in the art.

[0118] refer to Figure 5 A block diagram of a deep learning neural network 500 for providing classification scores for an image is shown. Image 502 is a colon image. In this disclosure, the term "classification score" or "score" may be used to describe a value or vector of values ​​generated by a machine learning system / model for a category or set of categories applicable to an image / frame. The term "classification probability" or "probability" may be used to describe the transformation of the classification score into a value reflecting the probability that each category in the set of categories is applied to the image / frame.

[0119] In some systems, deep learning neural networks 500 may include convolutional neural networks (CNNs) with "long short-term memory" (LSTM) and / or recurrent neural networks, which will be described in more detail below. In machine learning, CNNs are one of the most commonly used artificial neural networks for image analysis. The convolutional aspect of a CNN involves applying matrix processing operations (called "kernels" or "filters") to local portions of an image. During supervised training of the CNN, the kernels / filters are computationally tuned to identify features of the input image that can be used to classify the image. CNNs typically consist of convolutional layers, activation function layers, and pooling (usually max pooling) layers to reduce dimensionality without losing too much information.

[0120] The deep learning neural network 500 can use one or more CNNs to provide imaging capabilities for the CE imaging device 212 (see...). Figure 2One or more colon images captured provide classification scores for the presence of one or more feature points, colonic characteristics, colonic symptom, or colonic contents (e.g., bubbles, etc.). For example, a deep learning neural network 500 can generate classification scores for the presence of colonic polyps 510, the presence of ileocecal valves 512, hemorrhoids 514, or other feature points, characteristics, symptom, or contents 516 (e.g., colonic bleeding). The deep learning neural network 500 can be used in a computing system 300 ( Figure 3 This can be executed on [the target platform]. Those skilled in the art will understand the Deep Learning Neural Network 500 and how it can be implemented. Various deep learning neural networks can be used, including but not limited to MobileNet or Inception.

[0121] The deep learning neural network 500 can be trained based on labeled training images. For example, the images may have labels 504 indicating the presence of feature points, symptoms, characteristics, or contents, such as the presence of colonic polyps, ileocecal valves, or hemorrhoids. Labels 504 are shown in dashed lines to indicate that they are used only for training the deep learning neural network 500 and are not used outside of training, i.e., when operating the deep learning neural network 500 for inference. This training may include enhancing the training images by adding noise, changing colors, hiding parts of the training images, scaling the training images, rotating the training images, mirroring the training images, and / or stretching the training images. Those skilled in the art will understand how to train the deep learning neural network 500 and how to implement this training.

[0122] Used to provide classification scores Figure 5 The exemplary embodiments described are exemplary, and other ways of providing classification scores are contemplated within the scope of this disclosure. For example, two or more deep learning neural networks (not shown) may operate to provide classification scores 510-516 for a colon image 502. For example, one deep learning neural network may be configured to provide a classification score for the presence of a polyp 510, another deep learning neural network may be configured to provide a classification score for the presence of an ileocecal valve 512, and a third deep learning neural network may be configured to provide a classification score for the presence of a hemorrhoid plexus 514. Classification scores 510-516 may be provided by two or more deep learning neural networks in different configurations.

[0123] For example, in various embodiments, unsupervised learning or another type of learning may be used. In various embodiments, classification scores may be provided through various configurations of neural networks, through machine learning systems that are not neural networks (e.g., classic machine learning systems involving feature selection), and / or through classification techniques that will be recognized by those skilled in the art. In various embodiments, the machine learning system or classification system may provide classification probabilities instead of classification scores, or provide classification probabilities in addition to classification scores. In various embodiments, techniques such as Platt scaling, SoftMax, or other techniques that will be recognized by those skilled in the art may be used to convert classification scores into classification probabilities. Such variations are contemplated within the scope of this disclosure.

[0124] refer to Figure 6 A flowchart illustrating an exemplary operation for identifying an image containing polyps is shown. Figure 6 The operation can be performed by a computing system such as Figure 2 and Figure 3 The computing system is used to execute it. Figure 6 Some or all of the boxes may be referred to as a polyp detection system. At box 610, access is provided by a capsule endoscopy device (such as...). Figure 2 Various colon images captured by the CE imaging device 212. At box 620, an initial image selection process is applied to the colon images to select various images as seed images. The selection process accesses the polyp presence fraction or probability 622, such as by… Figure 5 The scores / probabilities provided by deep learning neural networks.

[0125] Typically, the seed image selection process 620 selects the image with the highest polyp presence score, and this selection can be performed in various ways. Exemplary selection processes are described in International Application Publication No. WO2017199258 and U.S. Provisional Application No. 63 / 018,870, the entire contents of which are incorporated herein by reference and can be applied to the initial selection process of box 620. For example, and as a brief description, the initial selection process can be an iterative process. At each iteration, the process selects the image with the highest score / probability for the presence of a polyp, and the selected image is referred to herein as a “seed image.” The scores / probabilities of images surrounding the seed image are reduced to decrease the probability of selecting an image with the same polyp in subsequent iterations. This process is repeated until one or more stopping criteria are met. For example, the iterative image selection process can terminate when no remaining image score meets a score / probability threshold. Alternatively, the iterative image selection process can terminate when a specific number of seed images (such as sixty or one hundred seed images) have been selected. Figure 7The results of the iterative selection process are illustrated in a graph, where the x-axis represents the image index / ID number and the y-axis represents the polyp presence score of the image. Images selected by the iterative process are indicated by circles at the top of the graph. The result of the initial image selection process 620 is a set of seed images with high polyp presence scores or probabilities. As mentioned above, the image selection process described is exemplary, and other image selection methods and techniques are conceivable within the scope of this disclosure.

[0126] The result of box 620 is a seed image with a high polyp presence score or probability. The operation of boxes 630-650 is described below, and these boxes can be performed based on a trade-off between sensitivity and specificity, as will be understood by those skilled in the art. In the operation of box 620, the focus can be on sensitivity, even if it requires a reduction in specificity. In the operation of boxes 630-650, the focus can be on specificity, even if it requires a reduction in sensitivity.

[0127] Continue to refer to Figure 6 At box 630, the seed image generated by the initial image selection process is processed by a negative filter and / or a positive filter. As used herein, positive filtering is the operation of positively designating a seed image that meets one or more criteria as a seed image containing polyps. On the other hand, negative filtering is the operation of identifying a seed image that meets one or more criteria as a seed image that should not be positively designated as a seed image containing polyps. In various embodiments, the negative filter does not designate the seed image as not containing polyps. In various embodiments, the negative filter may designate the seed image as not containing polyps. Box 630 may be applied with one or more positive filters and / or one or more negative filters, which will be described in more detail below. Now, it should be noted that various filters may use fractions or probabilities 632, such as those derived from... Figure 5 The classification score or probability is provided by the deep learning neural network. Additionally, various filters can be used with image trajectories 634, which will combine... Figure 10 and Figure 11 The description is as follows. Filters can be implemented using heuristics or machine learning systems (such as deep learning neural networks or classical machine learning systems). The results of box 630 may include seed images that have been specified by the positive filter as containing polyps, seed images that have been identified by the negative filter, and seed images that have not been specified by either the positive or negative filter. The last set of seed images (which have not been specified by either the positive or negative filter) will be referred to herein as “unfiltered” seed images. Unfiltered seed images are processed by box 640.

[0128] At box 640, the unfiltered seed image generated from box 630 is processed by a machine learning system, which operates to provide a classification score or probability indicating whether the unfiltered seed image contains a polyp. The machine learning system accesses input features 642 associated with the unfiltered seed image, which will be described in more detail below. In various embodiments, the machine learning system may be a classic machine learning system and may be trained by supervised learning, unsupervised learning, or another type of learning. In various embodiments, the machine learning system may be a soft-margin multinomial support vector machine with dimension n, which may be dimension 2, dimension 3, or another dimension. As mentioned above, the output of the machine learning system is a classification score or probability indicating whether the unfiltered seed image contains a polyp. Those skilled in the art will understand how to implement such a machine learning system and how to train such a machine learning system based on input features.

[0129] At box 650, the process identifies images with a high confidence level of containing polyps based on classification scores or probabilities provided by a machine learning system. Various thresholds can be applied to the classification scores or probabilities. For example, in various embodiments, images with a classification probability of containing polyps greater than 99% can be selected at box 650. The result of box 650 is an image not designated as containing polyps by the positive filter but with a high confidence level of containing polyps based on the machine learning classification score or probability. Such images selected by box 650 can be used in various ways described below. In various embodiments, images designated as containing polyps by the positive filter in box 630 can also be used in various ways, as described below.

[0130] Figure 6 The implementation scheme is exemplary, and variations are contemplated within the scope of this disclosure. For example, in various implementations, the process does not need to perform blocks 640 and 650, but may end at block 630, as... Figure 8 As shown. In Figure 8 In one implementation, the result of block 630 can be a seed image specified by a positive filter as containing polyps and / or an unfiltered seed image. Figure 6 Another variation, in various implementation schemes, may omit box 630, such as... Figure 9 As shown. In Figure 9 In this implementation, the machine learning system will be applied to all seed images 640 and will access the input features associated with seed image 642. Such variations and other variations are conceivable within the scope of this disclosure.

[0131] The following will describe what is possible. Figure 6 and Figure 8 Various positive and negative filters are applied in box 630.

[0132] like Figure 6 and Figure 8 As shown, various filters access and utilize the image trajectory of seed image 632. As used herein, "trajectory" refers to a set of consecutive images in which a sequential image tracker tracks polyps in the seed images. As mentioned above, the phrase "consecutive images" refers to images that are adjacent to each other in a sequence when arranged sequentially. "Sequential image tracker" refers to an object tracking technique designed to identify small variations in objects between consecutive images / frames, and it can identify whether seed images that are close to each other may contain the same polyps. Such tracking techniques include, for example, optical flow techniques. Those skilled in the art will understand how optical flow techniques are implemented. Other techniques for tracking objects in consecutive images are conceivable within the scope of this disclosure.

[0133] Figure 10 An example of applying a continuous image tracker to a seed image to identify the trajectory of the seed image is shown. Starting with seed image 1010, the continuous image tracker processes adjacent images to track polyp 1012. In the illustrated example, polyp 1012 is tracked across five frames before and three frames after seed image 1010. Tracking ends at the fourth frame 1020 after seed image 1010 through the operation of the tracking technique. A graphical representation 1030 of the tracking technique shows that the expected position 1032 of the polyp deviates from the actual position 1034 of the polyp. Therefore, polyp 1012 is not tracked to frame 1020. The trajectory of seed image 1010 is... Figure 10 The set of consecutive frames in the image (excluding frame 1020), the polyp 1012 in seed image 1010 is tracked across these consecutive frames by a continuous image tracker. This trajectory includes seed image 1010. Therefore, in Figure 6 and Figure 8 In this process, a trajectory is accessed for each seed image 632, and this trajectory can be used by various positive and / or negative filters. Figure 10 The implementation described is exemplary. In various implementations, other techniques for comparing two images may be used to identify “trajectories,” such as techniques for comparing two images using a classification system, described in co-pending U.S. Provisional Application No. 63 / 073,544, filed September 2, 2020. The entire contents of that provisional application are incorporated herein by reference.

[0134] As described above, positive filtering is the operation of positively designating a seed image that meets one or more criteria as a seed image containing a polyp. According to various aspects of this disclosure, positive filtering may have the following criteria: a seed image having a polyp presence score or probability 622 greater than or equal to a threshold will be designated as a seed image containing a polyp. In various embodiments, the polyp presence score may be normalized to a value between 0 and 1. The polyp presence probability naturally lies between 0 and 1. In various embodiments, the threshold may be 0.999999 or 0.9999999 or another value that provides a high degree of certainty that the seed image contains a polyp.

[0135] In various implementations, the positive filter may have additional criteria: the trajectory of the seed image includes at least a specific number of consecutive images whose polyp presence score or probability is greater than or equal to a threshold. In various implementations, the threshold for the seed image and the images in the trajectory may be the same value. In various implementations, the threshold for the seed image and the images in the trajectory may be different values. As an example, the positive filter may designate the seed image as containing polyps when the seed image has a polyp presence score / probability of at least 0.99999 and at least five consecutive frames adjacent to the seed image also have a polyp presence score / probability of at least 0.9999. Figure 11 An example of such a seed image and trajectory is shown, where the seed image is identified by frame number 171571. The seed image has a polyp presence score of 0.99999, and the five consecutive frames adjacent to the seed image have a polyp presence score of at least 0.9999. Therefore, Figure 11 The seed image is specified by a positive filter to contain polyps.

[0136] The positive filter described above is exemplary. Other positive filters for positively identifying an image as containing a polyp are conceivable within the scope of this disclosure. For example, trajectory information can be used in other ways to form a positive filter. As mentioned above, the trajectory comprises a set of images, and such images are obtained over time by a capsule endoscopy device (e.g., 212, Figure 2 Capture. Long Short-Term Memory (LSTM) can be used to process temporal information. Deep learning neural networks (such as...) Figure 5 A deep learning neural network (500) can be configured to receive image trajectories as input. The deep learning neural network can be trained to provide a classification score or probability of a seed image based on the image trajectories received by the deep learning neural network. The classification score or probability can be, for example, a probability score indicating that the seed image contains a polyp.

[0137] As described above, negative filtering is the operation of identifying seed images that meet one or more criteria as seed images that should not be positively designated as containing polyps. In various embodiments, negative filtering does not designate seed images as not containing polyps. In various embodiments, negative filtering may designate seed images as not containing polyps.

[0138] like Figure 6 and Figure 8 As shown, the negative filter can access classification scores or probabilities 632, such as those derived from... Figure 5 The classification score or probability provided by the machine learning system. According to various aspects of this disclosure, the negative filter can access the ileocecal lobe (ICV) existence score or probability (e.g., 512, ...). Figure 5 This mechanism can identify seed images with ICV scores or probabilities higher than a threshold, such as above 0.99999 or another threshold. The ileocecal valve is an anatomical feature at the transition from the small intestine to the colon and can visually resemble a large colonic polyp, allowing seed images to have a sufficiently high polyp presence score or probability of being seed images, while also having an ICV presence score above a predetermined threshold. Such seed images can be identified as meeting the criteria by a negative filter. The negative filter can specify seed images as not containing polyps.

[0139] According to various aspects of this disclosure, the negative filter can access the fraction or probability of the presence of hemorrhoids (e.g., 514, Figure 5 Hemorrhoids are anatomical features located at the end of the colon around the rectum and may resemble colonic polyps in appearance. In various implementations, a negative filter can operate to identify seed images with a hemorrhoid score or probability higher than a threshold, such as above 0.99999 or another threshold. Seed images may have a sufficiently high polyp presence score or probability of being a seed image, while also having a hemorrhoid presence score higher than a predetermined threshold. Such seed images can be identified by the negative filter as meeting the criteria. The negative filter can also designate seed images as not containing polyps.

[0140] In various implementations, instead of accessing a score or probability of hemorrhoid presence, a negative filter can work to determine the proximity of the seed image to the body exit / gastrointestinal exit. The proximity of the seed image to the body exit can be determined in various ways. For example, a negative filter can access a colon image (e.g., in...) Figure 6The proximity of the seed image to the body exit can be determined by whether the seed image is within the last portion of the colon image, such as whether the seed image is within the last 0.5% of the colon image or within another last percentage of the colon image. If the seed image is within the last portion of the colon image, the negative filter can identify the seed image as meeting the criteria. In various embodiments, the negative filter can specify the seed image as not containing polyps. In various embodiments, the negative filter can identify the seed image as meeting the criteria, but may not specify the seed image as not containing polyps.

[0141] According to various aspects of this disclosure, the negative filter can access the image trajectory of the seed image, such as combining... Figure 10 The image trajectory is described. The negative filter may have a criterion for identifying a seed image when it is the only image in the trajectory with a polyp presence score or probability higher than a threshold. For example, a seed image may be identified as meeting the criterion when the seed image of the image trajectory has a polyp presence probability of at least 0.998 and every other image in the image trajectory has a polyp presence probability of less than 0.998. Other thresholds may be used. In various embodiments, the negative filter may specify a seed image as not containing polyps. In various embodiments, the negative filter may identify a seed image as meeting the criterion, but may not specify a seed image as not containing polyps.

[0142] According to aspects of this disclosure, a negative filter can access an estimated polyp size of a seed image. The negative filter can have a criterion for identifying the seed image when the estimated polyp size of the seed image is less than a threshold, such as when the estimated polyp size is less than 3.5 mm or less than another threshold. Various techniques can be used to generate the estimated polyp size accessed by the negative filter. Examples of the techniques are disclosed in a co-pending U.S. patent application with file number A0004997US01 (2851-17PRO), the entire contents of which are incorporated herein by reference. Other techniques for estimating the polyp size in an image will be understood by those skilled in the art. Such other techniques are contemplated within the scope of this disclosure.

[0143] Therefore, various positive and negative filters have been described above. Such filters can be applied to… Figure 6 and Figure 8 Within frame 630. Figure 6 In the operation, seed images that are neither specified by the positive filter nor recognized by the negative filter (i.e., unfiltered seed images) can be processed by the machine learning system of box 640, as described above. Figure 8 In the operation, box 630 marks the end of the operation and provides a seed image specified by a positive filter, and in some embodiments, an unfiltered seed image may also be provided.

[0144] The following will describe Figure 6 and Figure 9 The exemplary input features of the machine learning system accessed in box 642. As described above, the machine learning system works based on input features to provide a classification score or probability indicating whether an unfiltered seed image contains a polyp. In various embodiments, the machine learning system may be a soft-margin multinomial support vector machine with dimension n. In various embodiments, the machine learning system may be based on another classic machine learning model that those skilled in the art will recognize, such as decision trees, Naive Bayes, or logistic regression. As described below, some input features may be based on the image trajectory of the seed image, such as... Figure 10 and Figure 11 The image trajectory shown.

[0145] According to various aspects of this disclosure, one of the input features to a machine learning system may be derived from a polyp detector (such as...). Figure 5 The detector shown provides the seed polyp score / probability.

[0146] According to various aspects of this disclosure, one of the input features to the machine learning system may be a seed polyp score / probability, which is based on a set of polyp detectors (e.g., Figure 5 The seed polyp score is determined by a vote or operation provided by the polyp detector set, which takes an image as input and outputs the probability that the image contains a polyp. For example, the seed polyp score could be the mean of the polyp scores / probabilities provided by the polyp detector set, or it could be provided by another operation (such as median).

[0147] According to various aspects of this disclosure, one of the input features to the machine learning system may be the number of images in the image trajectory of the seed image, which may be referred to as the trajectory length.

[0148] According to various aspects of this disclosure, one of the input features to the machine learning system may be the number of images in the image trajectory of the seed image that have a polyp presence score or probability greater than a threshold (such as a polyp presence probability greater than 0.998 or greater than another threshold).

[0149] According to various aspects of this disclosure, one of the input features to the machine learning system may be the difference between the index / ID number of the seed image and the index / ID number of the image at the colon origin. In various embodiments, the image at the colon origin may be an image of the ICV. The image of the colon origin can be determined in various ways. For example, the ICV may have a fraction or probability (e.g., 512, ...). Figure 5This technique can be used to determine images of the ICV. For example, and as mentioned above, the division of a GIT image into GIT portions of a captured image is solved in co-pending U.S. Provisional Application No. 63 / 018,890, and the division of a colon image into colon portions of a captured image is solved in co-pending U.S. Provisional Application No. 63 / 018,878. This technique for dividing a GIT image into multiple portions can be used to identify images of the colon's origin. Other techniques for identifying images of the colon's origin are conceivable within the scope of this disclosure.

[0150] According to various aspects of this disclosure, one of the input features to the machine learning system may be location information (represented as numbers) regarding the colonic segment in which the seed image was captured. (As combined with...) Figure 4 The colon 400 comprises five anatomical segments: the cecum, right or ascending colon, transverse colon, left or descending colon, and rectum. These five segments can be numbered 1 to 5. The segment number of the colon segment in which a seed image is captured can be an input feature to a machine learning system. As mentioned above, co-pending U.S. Provisional Application No. 63 / 018,878 addresses the division of a colon image into colonic portions of the captured image. This technique for dividing a colon image into colonic portions in which the image is captured can be used to identify the number of colonic segments in which a seed image is captured. Other techniques for identifying the number of colonic segments in which a seed image is captured will be understood by those skilled in the art and are conceived within the scope of this disclosure.

[0151] Therefore, various input features to machine learning systems have been described. Those skilled in the art will understand how to train and implement machine learning systems based on such input features. In various embodiments, it is not necessary to use all of the described input features, and various combinations of these input features can be used. In various embodiments, all of the described input features can be used. Some or all of these input features can be normalized in various ways. The described input features are exemplary, and other input features are contemplated within the scope of this disclosure.

[0152] Refer again Figure 6 and Figure 8 The machine learning system in box 640 processes the input features and provides a classification score or probability indicating whether each seed image contains a polyp. The classification probability can be used directly to determine which seed images have a sufficiently high probability of being selected as seed images containing polyps. Classification scores can be converted to classification probabilities in various ways, such as through Platt scaling, SoftMax, or other techniques that those skilled in the art will recognize. Seed images designated as containing polyps can be used in various ways, as combined below. Figure 12 The explanation given.

[0153] Based on the aspects of this disclosure, various methods can be used to enhance... Figure 6 , Figure 8 and Figure 9 This involves adjustments to the procedures. For example, additional rules based on polyp size estimates (e.g., adjusted polyp detector score thresholds) can be added, for instance, to align with local medical guidelines / practice / strategies related to polyp size. For example, US medical practice typically bases on at least one polyp of a specific size or larger, while European medical practice typically bases on multiple polyps of any size. Other countries may have different medical practices and may develop additional rules based on their specific national medical practices.

[0154] Therefore, the above description provides a system and method for identifying images containing polyps with a high degree of confidence. Exemplary use of the identified images is described below.

[0155] Now for reference Figure 12 The illustration shows an exemplary display screen for presenting polyp images to clinicians. A GUI (or study viewing application) can be used to display studies for user review and generate study reports (or CE procedure reports). Figure 12 The screen displays a set of still images included in the study. These images can be used, for example, in... Figure 6 , Figure 8 or Figure 9 The user selects a seed image in box 620. The user can review the images and select one or more images of interest (e.g., displaying one or more polyps). These study images are displayed according to their location in the colon. This location can be any of the following five anatomical colonic segments: cecum, ascending colon, transverse colon, descending-sigmoid colon, and rectum. The screen displays study images identified as being located in the descending-sigmoid colon. The user can switch between the display of images located in different segments. The user (e.g., a clinician) can use the displayed screen to select images to include in the study report. In some embodiments, the study may also include trajectories associated with the seed image (i.e., the study seed image). In this case, the user can request (via user input) to display trajectories (not shown) associated with the displayed images. By reviewing the associated trajectories, the clinician receives additional information related to the seed image, which can help the clinician determine whether the seed image (or optionally any other trajectory image) is of interest.

[0156] Continue to refer to Figure 12Clinicians can add bounding boxes around polyps observed in images. The user-added bounding boxes 1210 can appear in a specific color, such as green or another color. According to aspects of this disclosure, an image containing polyps identified by the systems and methods of this disclosure can be presented to a clinician, and bounding boxes 1220 can be automatically added to such images to indicate the location of the polyps. The bounding boxes 1220 added by the systems and methods of this disclosure can appear in a different color than the user-added bounding boxes, such as red or another color. In this way, the user can easily see which bounding boxes were added by the user and which were added automatically.

[0157] Now for reference Figure 13 An exemplary display screen is shown for suggesting polyp images to a clinician. The display screen shows images 1310, 1312 that have been selected by the clinician as containing polyps, which the clinician wants to include in the final capsule endoscopy procedure report. Before the clinician completes the selection, the systems and methods of this disclosure can display suggested images of polyps 1320 that the clinician may have missed or not selected. Suggested image 1320 may be an image designated as containing a polyp by a positive filter (e.g., box 630, ...). Figure 6 and Figure 8 The image or in frame 650 ( Figure 6 and Figure 9 The image selected at the location has a sufficiently high classification score or probability of containing a polyp. In various embodiments, the suggested image 1320 may also include an unfiltered seed image. In various embodiments, the suggested polyp image 1320 may be limited to a GIT segment that does not contain any polyp frames selected by the clinician, or limited to a GIT segment where the clinician only identifies smaller polyps (e.g., less than 6 mm). In various embodiments, if a GIT segment includes an image of a polyp (e.g., 6 mm) selected by the clinician and the system of this disclosure identifies an image of a smaller polyp (e.g., 5 mm), the system of this disclosure may not suggest the smaller polyp to the clinician. Therefore, in various embodiments, images that provide additional clinical values ​​(e.g., according to medical practice guidelines) may be suggested, while images that do not provide additional clinical values ​​may not be suggested.

[0158] refer to Figure 14 This illustrates an exemplary display screen that can be automatically generated by the systems and methods of this disclosure without any human intervention or input. Figure 13 Compared to the display screen (which contains images 1310, 1312 selected by the user), Figure 14 Images of polyps can be automatically selected without manual input. The automatically selected images can be those specified by a positive filter as containing polyps (e.g., box 630). Figure 6 and Figure 8The image or in frame 650 ( Figure 6 and Figure 9 The image selected at the location has a sufficiently high classification score or probability of containing a polyp. In various embodiments, the display screen may always show the page with all suggested polyps, regardless of any clinician selection or decision, and such a display screen may be available to the user or clinician before or after the clinician views any images. In various embodiments, Figure 14 The automatic selection may exclude unfiltered seed images. In various embodiments, the systems and methods of this disclosure can skip... Figure 14 The display screen can automatically generate and finalize capsule endoscopy procedure reports without any input or intervention from clinicians.

[0159] Figures 12 to 14 It shows the result of Figure 6 , Figure 8 and Figure 9 The possible uses of the images selected during the process. Figures 12 to 14 The embodiments described are exemplary, and such a display screen does not limit the scope of this disclosure. Other uses are contemplated. In various embodiments, the systems and methods of this disclosure can be used to veto decisions by other tools to exclude CE procedures, such as those described in a co-pending U.S. Provisional Application with Dossier No. A0003746US01(2851-7PRO), which is incorporated herein by reference in its entirety. Such tools provide adequacy measures that indicate the effectiveness of CE procedures when capturing predefined events in multiple images. In various embodiments, the adequacy measure of the procedure is determined based on characteristic measures, which may include multiple measures indicating the probability of capturing or not capturing at least one of the predefined events. These multiple measures may include: (i) a segment adequacy probability based on at least two of the following: motility score, cleanliness level of each segment, or passage time; (ii) a global adequacy measure based on at least one of the following: average cleanliness score across all segments, patient demographics, the last segment of the GIT reached by the CE device, or the absolute time spent by the CE device in that segment of the GIT; and / or (iii) at least one of the following: anatomical colonic segment associated with the image, passage pattern of the CE device, CE device communication errors, anatomical features in multiple images, or coverage of GIT tissue in multiple images. Such and other uses are contemplated within the scope of this disclosure.

[0160] While this disclosure provides systems and methods for identifying polyp images with high confidence, not all polyps require follow-up procedures. In particular, the size of the polyp is crucial in determining whether follow-up procedures are necessary. If the polyp is large enough, such as at least 6 mm, clinicians typically wish to examine it via colonoscopy. According to aspects of this disclosure, the systems and methods of this disclosure can determine whether colonoscopy or follow-up procedures are recommended within a specific number of months or years. Such determination can be made by a computational system (e.g., Figure 3 The computing system executes this.

[0161] refer to Figure 15 and Figure 16 This determination can be made based on the probability of the presence of at least one polyp with a size of 6 mm or larger. This determination uses data obtained from... Figure 6 , Figure 8 and Figure 9 The images identified in the process can be seed images designated by a positive filter as containing polyps (box 630) or seed images with a sufficiently high classification score or probability of containing polyps (boxes 640, 650). Assuming there are n such images, P i(TP&尺寸≥6[mm]) Let represent the probability that image i contains a polyp and the polyp is at least 6 mm in size. This probability includes two element images: whether the image contains a polyp and whether the polyp is at least 6 mm in size. Assuming these two elements are independent, the probability can be expressed as:

[0162] P i (TP & size ≥ 6 [mm]) = P i (TP)P i (Dimensions ≥ 6 mm).

[0163] P i (TP) represents the probability that image i contains a polyp. i (Size ≥ 6 [mm]) indicates the probability that the polyp in image i is 6 mm or larger. To determine whether a colonoscopy procedure should be recommended, only one candidate image needs to have a sufficiently high probability of including a polyp of at least 6 mm in size.

[0164] Figure 15 It shows that it can be used to determine P i (TP) curve. The x-axis represents machine learning systems (such as...) Figure 6 , Figure 8 and Figure 9The soft margin of a machine learning system (as described in box 640). As those skilled in the art will understand, a soft margin can refer to the continuous output of a classifier (e.g., a classical machine learning algorithm) relative to the distance between the example and the separating hyperplane / classification boundary of the classifier. The soft margin indicates how confident the machine learning system is that it has correctly classified an image as containing polyps. The higher the absolute value of the soft margin, the further away from the classification boundary, and the more confident its decision. The soft margin can be empirically derived by accessing the soft margins associated with the labeled training set. Figure 15 The curve graph.

[0165] As an example, Figure 15 The x-axis can be divided into intervals of 0.1 or another size. For each soft margin interval, the number of training inputs corresponding to the presence of polyps and having soft margins within that interval can be counted, and the number of training inputs corresponding to the absence of polyps and having soft margins within that interval can be counted. These two counts can be used empirically to calculate the percentage of inputs with polyps and soft margins within that interval. If an input's soft margin falls within that interval, this percentage can be used as a proxy for the probability that the input has a polyp. Figure 15 An exemplary result of this calculation is shown. These probabilities are quite noisy because they are determined empirically. Regression analysis can be performed to fit curve 1502 to empirical probabilities to provide a basis for determining P based on the soft margin. i (TP) smoothing estimator 1502. The above is about... Figure 15 The described implementation scheme is exemplary and is conceivable for determining P. i Other methods of (TP).

[0166] Figure 16 It shows that it can be used to determine P i A graph (size ≥ 6 [mm]). The x-axis represents the polyp size estimate determined for capsule endoscopy (CE) images. As described above, the polyp size can be estimated in the manner described in the co-pending U.S. patent application A0004997US01 (2851-17PRO), or by other techniques that will be understood by a person skilled in the art. Each training CE image with an actual polyp size known (e.g., at least 6 mm or less) can be processed to determine its estimated polyp size. The x-axis can be divided into intervals for the estimated polyp size, such as intervals of 0.1 mm or intervals of another size. Training inputs with estimated polyp sizes falling within the intervals can be counted. The counts within the intervals can be used to calculate the percentage of training inputs with actual polyp sizes of 6 mm or greater for that interval, and an empirical percentage can be used as a substitute for the probability that the input in that interval has a polyp of 6 mm or greater. Figure 16An exemplary result of this calculation is shown. These probabilities are quite noisy because they are determined empirically. Regression analysis can be performed to fit curve 1602 to empirical probabilities to provide a basis for determining P based on the estimated polyp size. i Smoothing estimator 1602 (size ≥ 6 mm). The above is about... Figure 16 The described implementation scheme is exemplary and is conceivable for determining P. i Other methods (for dimensions ≥ 6 mm).

[0167] As mentioned above, the probability that a seed image contains a polyp, and that the polyp is at least 6 mm in size, can be determined by the following formula: P i (TP)P i (Size ≥ 6 mm). If any probability generated by the calculation is greater than a threshold, such as 0.999 or another threshold, the calculation can determine the presence of an image of a polyp 6 mm or larger, and a colonoscopy can be recommended based on this.

[0168] The implementation using 6mm as the polyp size boundary can be applied to another polyp size boundary, such as 5mm or 7mm or another polyp size boundary.

[0169] The above text is about Figure 15 and Figure 16 The described implementation is exemplary. Other methods are conceivable for determining whether a colonoscopy should be recommended. Such variations and other modifications are conceivable within the scope of this disclosure.

[0170] refer to Figure 17 It shows that it can be used Figure 15 and Figure 16Exemplary operation of the system and method. At box 1710, the operation involves accessing multiple images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device during CE procedures. Each of the multiple images is suspected of containing a polyp and is associated with a probability of containing a polyp. Additionally, the multiple images include seed images, wherein each seed image is associated with one or more images of the multiple images, and the one or more images associated with each seed image are identified as suspected of containing the same polyp as the associated seed image. At box 1720, the operation involves applying a polyp detection system to the seed images to identify seed images containing polyps. The polyp detection system is applied to each of these seed images based on the one or more images associated with the seed image and the probability associated with the seed image and the one or more associated images. At box 1730, the operation involves identifying images of the multiple images that include polyps of a size equal to or larger than a predefined size. Each of the plurality of images is further associated with an estimated size of a suspected polyp contained in each image, and the polyp detection system is further applied to each of these seed images based on the estimated polyp size associated with the seed image and the one or more images associated with the seed image. At block 1740, when at least one seed image is identified as including a polyp of size equal to or greater than a predefined size, or including a predefined number of polyps of size equal to or greater than a predefined size, if the procedure is determined to be insufficient and excluded, the operation involves excluding the procedure.

[0171] Regarding box 1740, and as mentioned above, techniques for determining procedural inadequacy are disclosed in a co-pending U.S. provisional application with case number A0003746US01(2851-7PRO). As stated above, such a tool provides an adequacy measure that indicates the effectiveness of a CE procedure when capturing predefined events in multiple images, and the adequacy measure of the procedure can be determined based on characteristic measures that may include multiple measures indicating the probability of capturing or not capturing at least one of the predefined events.

[0172] Continuing with reference to box 1740, the exclusion operation of a rejection procedure may be based on heuristics, such as a threshold for polyp detection probability and / or optionally, polyp size or a minimum number of images. In various embodiments, the rejection operation may be based on (e.g., based on the probability that images in the seed image set include polyps of at least a predefined size) for each procedure. Figure 15 and Figure 16 ). Figure 17 The operation is exemplary, and variations are conceivable within the scope of this disclosure.

[0173] Therefore, the above description provides systems and methods for identifying images containing polyps with a high degree of confidence, and provides various uses for such identified images. The aspects and embodiments described herein are exemplary and do not limit the scope of this disclosure.

[0174] While several embodiments of this disclosure have been shown in the accompanying drawings, it is not intended to limit this disclosure, as it is intended to be as broad as permitted by the art and this specification should be read in the same manner. Therefore, the foregoing description should not be construed as restrictive, but rather as illustrative of particular embodiments only. Those skilled in the art will be able to conceive of other modifications within the scope and spirit of the appended claims.

Claims

1. A system for recommending colonoscopy, comprising: One or more processors; and At least one memory storing instructions that, when executed by the one or more processors, cause the system to: Access multiple images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, the multiple images having the potential to contain polyps; For each of the plurality of images: A classic machine learning system is applied, which is configured to provide an indication of whether the image contains polyps based on input features corresponding to the image; Access the soft interval of the classical machine learning system corresponding to the image; Access the mapping from soft intervals to the probability that an image contains polyps; Access the estimated polyp size of the image, which is generated based on the image; Access the mapping from estimated polyp size to the probability that the actual polyp size is at least a predefined size, and Without human intervention, the decision to recommend a colonoscopy is made based on the mapping from the soft septum to the probability that the image contains a polyp and the mapping from the estimated polyp size to the probability that the actual polyp size is at least a predefined size.

2. A computer-based method for recommending colonoscopy, comprising: Access multiple images of the gastrointestinal tract (GIT) captured by a capsule endoscopy device, the multiple images having the potential to contain polyps; For each of the plurality of images: A classic machine learning system is applied, which is configured to provide an indication of whether the image contains polyps based on input features corresponding to the image; Access the soft interval of the classical machine learning system corresponding to the image; Access the mapping from soft intervals to the probability that an image contains polyps; Access the estimated polyp size of the image, which is generated based on the image; Access the mapping from estimated polyp size to the probability that the actual polyp size is at least a predefined size, and Without human intervention, the decision to recommend a colonoscopy is made based on the mapping from the soft septum to the probability that the image contains a polyp and the mapping from the estimated polyp size to the probability that the actual polyp size is at least a predefined size.

Images