Estimating adequacy of a procedure

Abstract

A computer-implemented method for estimating adequacy of a capsule endoscopy (CE) procedure includes accessing a plurality of images of at least a portion of a gastrointestinal tract (GIT) captured by a CE imaging device during a CE procedure, accessing a plurality of characteristic metrics associated with the plurality of images, determining an adequacy metric of the CE procedure based on the plurality of characteristic metrics, wherein the adequacy metric provides a measure of whether an imaging coverage provided by the plurality of images is adequate to capture an event of interest in at least the portion of the GIT regardless of whether such an event of interest actually exists in at least the portion of the GIT, and displaying an adequacy indication of the CE procedure based on the adequacy metric.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application Serial No. 63 / 075,778, filed September 8, 2020, and U.S. Provisional Patent Application Serial No. 63 / 228,937, filed August 3, 2021, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to image analysis methods and systems, and more specifically to systems and methods for analyzing image streams captured via capsule endoscopy to estimate the adequacy of the procedure. Background Technology

[0004] Capsule endoscopy (CE) allows for the examination of the entire gastrointestinal tract (GIT) through an endoscope. Several capsule endoscopy systems and methods are designed to examine specific parts of the GIT, such as the small intestine (SB) or colon. CE is a non-invasive procedure that does not require hospitalization, and patients can continue most of their daily activities while the capsule is inside their body.

[0005] In a typical CE procedure, the patient is referred by a physician. The patient then arrives at a medical facility (e.g., a clinic or hospital) for the procedure. Under the supervision of a healthcare professional (e.g., a nurse or physician), the patient swallows a capsule approximately the size of a multivitamin and is provided with a wearable device, such as a sensor strap and recorder placed in a bag and worn on the patient's shoulder. The wearable device typically includes a storage device. The patient can then receive instructions and / or guidance before being discharged to resume their daily activities.

[0006] The capsule captures images as it passes naturally through the GIT. The images and additional data (e.g., metadata) are then transmitted to a recorder worn by the patient. The capsule is typically disposable and is expelled naturally with defecation. Surgical data (e.g., captured images or portions thereof, along with additional metadata) is stored on the wearable device's storage.

[0007] Wearable devices are typically returned to the medical facility by the patient along with the surgical data stored on them. The surgical data is then downloaded to a computing device, usually located at the medical facility, which stores engine software. The engine then processes the received surgical data into a compiled research report (or "research report"). A typical research report includes thousands of images (approximately 8,000 to 10,000). The number of images to be processed is usually in the tens of thousands, averaging around 100,000.

[0008] The reader (who may be the supervising surgeon, specialist, or referring physician) can access the research report via a reader application. The reader then reviews the report, evaluates the surgery, and provides their input via the reader application. Since the reader needs to review thousands of images, reading a single research report can typically take an average of half an hour to an hour, and the reading task can be tedious. The reader application then generates a report based on the compiled research report and the reader's input. On average, generating a report takes one hour. The report may include, for example, images of interest, such as images selected by the reader and identified as including pathology; an assessment or diagnosis of the patient's medical condition based on surgical data (i.e., the research report) and / or follow-up and / or treatment recommendations provided by the reader. The report can then be forwarded to the referring physician. The referring physician can then determine the necessary follow-up or treatment based on the report. Summary of the Invention

[0009] This disclosure relates to systems and methods for analyzing image streams of the gastrointestinal tract (GIT). More specifically, this disclosure relates to systems and methods for analyzing image streams after a capsule endoscopy (CE) procedure to estimate the adequacy of the CE procedure used to capture the event of interest, such as estimating whether the imaging coverage of the image stream is sufficient to visualize at least one polyp (regardless of whether any polyp is actually present). In various respects, at least one polyp may include a significant polyp, such as a polyp of approximately 6 mm or larger. As described herein, when the adequacy of the CE procedure cannot be determined by constructing a three-dimensional view of the GIT or a portion of the GIT using images captured in vivo by a CE imaging device, other metrics and / or indicators are used to determine the adequacy of the CE procedure. When the imaging coverage of the image stream is estimated to be insufficient to visualize at least one polyp (regardless of whether any polyp is actually present), this disclosure can exclude (e.g., by clinicians and / or automatically) the CE procedure and can therefore significantly reduce the percentage of people with polyps who are mistakenly removed by capsule endoscopy as if the polyp was not visualized. This disclosure can also more confidently rule out cases where there are no polyps if the CE procedure is deemed sufficient to visualize at least one polyp (regardless of whether any polyps are actually present).

[0010] Although examples have been shown and described with respect to images captured in vivo by a capsule endoscope device, the disclosed techniques can be applied to images captured by other devices or institutions. Furthermore, to the fullest extent, any or all aspects detailed herein may be used in conjunction with any or all other aspects detailed herein.

[0011] According to various aspects of this disclosure, a computer-implemented method for estimating the adequacy of a capsule endoscopy (CE) procedure is provided, comprising: accessing multiple images of at least a portion of the gastrointestinal tract (GIT) captured by a CE device during the CE procedure; determining an adequacy measure of the CE procedure, the adequacy measure indicating a measure of the effectiveness of the CE procedure in terms of predefined events captured in the multiple images; and displaying the adequacy measure on a display.

[0012] In one aspect of this disclosure, the adequacy measure of the procedure can be determined based on predefined characteristics of the CE procedure.

[0013] In one aspect of this disclosure, predefined events may include pathology types, at least one occurrence of a pathology type, all occurrences of a pathology type in a predefined portion of a GIT, at least one occurrence of a pathology type of a predefined size, all occurrences of a pathology type of a predefined size in a predefined portion of a GIT, polyps, at least one occurrence of polyps, all occurrences of polyps in the colon, at least one occurrence of polyps in the colon larger than a predefined size, all occurrences of polyps in the colon larger than a predefined size, parasites, disease indicators, and / or disease appearance.

[0014] In another aspect of this disclosure, a predefined event may be a periodic event, a temporary event, and / or a constant event.

[0015] In another aspect of this disclosure, the method may further include providing instructions on whether to exclude the generation of research reports based on CE surgery, wherein the determination is based on a determined adequacy measure.

[0016] In another aspect of this disclosure, the method may further include excluding the generation of research reports based on the determined sufficiency measure.

[0017] In another aspect of this disclosure, the method may further include, in the case of excluding CE surgery: receiving a probability score indicating whether a predefined event is included in the accessed image; and generating a study report of the previously excluded CE surgery based on the probability score of the event being higher than a predetermined threshold.

[0018] In one aspect of this disclosure, sufficiency measures can be determined based on classical machine learning techniques, deep learning techniques, and / or heuristics.

[0019] In one aspect of this disclosure, characteristic metrics may include segmental characteristics or per-operative global characteristics.

[0020] In another aspect of this disclosure, each of the multiple images of the GIT can be associated with one of the multiple consecutive segments of the GIT. The method may further include determining a segmental sufficiency measure for each of the multiple consecutive segments of the GIT based on one or more segmental characteristics.

[0021] In another aspect of this disclosure, segmental characteristics may be selected from a group consisting of: movement scores or scores indicating the average cleanliness level of each segment.

[0022] In another aspect of this disclosure, a sufficiency metric can be further determined based on a segment sufficiency probability, which is based on multiplying at least two of the following: movement score, cleanliness level of each segment, and / or transit time.

[0023] In one aspect of this disclosure, the global adequacy measure per procedure may be based on the average cleanliness score across all segments, patient demographics, the last segment of the GIT reached by the CE device, and / or the absolute time spent by the CE device in the GIT segment.

[0024] In another aspect of this disclosure, characteristic metrics may include anatomical colonic segments associated with images, transport patterns of the capsule endoscope device, CE device communication errors, anatomical landmarks in multiple images, and / or coverage of GIT tissue in multiple images.

[0025] According to various aspects of this disclosure, a system for estimating the adequacy of capsule endoscopy (CE) surgery is provided. The system includes a display, at least one processor, and at least one memory. The memory includes instructions stored thereon that, when executed by the at least one processor, cause the system to: access multiple images of at least a portion of the gastrointestinal tract (GIT) captured by the CE device during CE surgery; determine an adequacy measure of the CE surgery, the adequacy measure indicating a measure of the effectiveness of the CE surgery in terms of predefined events captured in the multiple images; and display the adequacy measure on the display.

[0026] In another aspect of this disclosure, the adequacy measure of the procedure can be determined based on predefined characteristics of the CE procedure.

[0027] In one aspect of this disclosure, the instructions, when executed by at least one processor, can further cause the system to provide an indication of whether to exclude the generation of a study report based on capsule endoscopy (CE) surgery, wherein the determination is based on a determined adequacy metric. According to aspects of this disclosure, a computer-implemented method for estimating the adequacy of capsule endoscopy (CE) surgery is provided, comprising: accessing multiple images of at least a portion of the gastrointestinal tract (GIT) captured by a CE device during CE surgery; determining an adequacy metric indicating the effectiveness of the CE surgery in terms of predefined events captured in the multiple images; and excluding the generation of a study report based on CE surgery, the exclusion being based on the adequacy metric being below a predetermined threshold.

[0028] In another aspect of this disclosure, the method may further include instructing the user that CE surgery has been ruled out.

[0029] In another aspect of this disclosure, the method may further include: receiving event scores; and including previously excluded CE procedures in the study report based on the received event scores being higher than a predetermined threshold.

[0030] According to various aspects of this disclosure, a computer-implemented method for estimating the adequacy of a capsule endoscopy (CE) procedure includes: accessing a plurality of images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging device during the CE procedure; accessing a plurality of characteristic measures associated with the plurality of images; determining an adequacy measure of the CE procedure based on the plurality of characteristic measures, wherein the adequacy measure provides a measure of whether the imaging coverage provided by the plurality of images is sufficient to capture an event of interest in at least said portion of the GIT regardless of whether such an event of interest is actually present in at least said portion of the GIT; and displaying an indication of the adequacy of the CE procedure based on the adequacy measure.

[0031] In various embodiments of the computer-implemented method, the method includes processing the plurality of images to identify a plurality of image groups, wherein in each of the plurality of image groups, each image of the corresponding image group captures the same tissue region.

[0032] In various embodiments of the computer-implemented method, for each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the number of images in the respective image group, and the adequacy measure of the CE procedure is determined based on the number of images in each of the plurality of image groups.

[0033] In various embodiments of the computer-implemented method, for each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the average cleanliness ratio of the corresponding image group, and the adequacy measure of the CE procedure is determined based on the average cleanliness ratio of each of the plurality of image groups.

[0034] In various embodiments of the computer-implemented method, the method includes determining an average cleanliness ratio for each image group by: accessing a mapping between cleanliness scores and cleanliness ratios; and for each of the plurality of image groups: accessing a cleanliness score for each image in the respective image group, determining a cleanliness ratio for each image in the respective image group based on the mapping between cleanliness scores and cleanliness ratios, and determining an average cleanliness ratio for the respective image group as the average of the cleanliness ratios of the images in the respective image group.

[0035] In various embodiments of the computer-implemented method, at least the portion of the GIT comprises multiple segments. Determining the adequacy measure of the CE procedure includes determining an adequacy measure for each of the multiple segments, and determining an adequacy measure of the CE procedure based on the adequacy measure of each of the multiple segments.

[0036] In various embodiments of the computer-implemented method, the method may further include determining that the adequacy metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture events of interest in at least said portions of the GIT, regardless of whether such events of interest are actually present in at least said portions of the GIT. The adequacy indication of the CE procedure includes at least one reason why the CE procedure is determined to be insufficient.

[0037] In various embodiments of the computer-implemented method, determining the adequacy measure of the CE procedure based on an adequacy measure of each of the plurality of segments includes: accessing a prior probability of the occurrence of the event of interest in each of the plurality of segments, the prior probability being determined empirically based on a patient population; and determining the adequacy measure of the CE procedure based on the prior probability and based on the adequacy measure of each of the plurality of segments.

[0038] In various embodiments of the computer-implemented method, the method includes: accessing at least one quality metric associated with the plurality of images; determining the sufficiency indication based on a first set of sufficiency rules when the at least one quality metric is satisfied; and determining the sufficiency metric based on a second set of sufficiency rules when any of the at least one quality metric is not satisfied.

[0039] According to various aspects of this disclosure, a system for estimating the adequacy of capsule endoscopy (CE) surgery includes a display device, at least one processor, and at least one memory, the at least one memory including instructions stored thereon. These instructions, when executed by the at least one processor, cause the system to: access a plurality of images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging device during CE surgery; access a plurality of characteristic measures associated with the plurality of images; determine an adequacy measure of the CE surgery based on the plurality of characteristic measures, wherein the adequacy measure provides a measure of whether the imaging coverage provided by the plurality of images is sufficient to capture events of interest in at least said portion of the GIT regardless of whether such events of interest are actually present in at least said portion of the GIT; and display an adequacy indication of the CE surgery on the display device based on the adequacy measure.

[0040] In various embodiments of the system, the instructions, when executed by the at least one processor, further cause the system to process the plurality of images to identify a plurality of image groups, wherein in each of the plurality of image groups, each image of the corresponding image group captures the same tissue region.

[0041] In various embodiments of the system, for each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the number of images in the corresponding image group, and the adequacy measure of the CE procedure is determined based on the number of images in each of the plurality of image groups.

[0042] In various embodiments of the system, for each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the average cleanliness ratio of the corresponding image group, and the adequacy measure of the CE procedure is determined based on the average cleanliness ratio of each of the plurality of image groups.

[0043] In various embodiments of the system, when executed by the at least one processor, the instructions further cause the system to determine the average cleanliness ratio of each image group by: accessing a mapping between cleanliness scores and cleanliness ratios; and for each of the plurality of image groups: accessing the cleanliness score of each image in the respective image group, determining the cleanliness ratio of each image in the respective image group based on the mapping between cleanliness scores and cleanliness ratios, and determining the average cleanliness ratio of the respective image group as the average of the cleanliness ratios of the images in the respective image group.

[0044] In various embodiments of the system, at least the portion of the GIT includes multiple segments, and determining the adequacy measure of the CE procedure includes: determining an adequacy measure for each of the multiple segments, and determining an adequacy measure of the CE procedure based on the adequacy measure of each of the multiple segments.

[0045] In various embodiments of the system, when determining the adequacy measure of the CE procedure based on an adequacy measure of each of the plurality of segments, the instructions, when executed by the at least one processor, cause the system to: access the prior probability of the occurrence of the event of interest in each of the plurality of segments, the prior probability being determined empirically based on a patient population; and determine the adequacy measure of the CE procedure based on the prior probability and based on the adequacy measure of each of the plurality of segments.

[0046] In various embodiments of the system, the instructions, when executed by the at least one processor, further cause the system to: access at least one quality metric associated with the plurality of images; determine the adequacy indication based on a first set of adequacy rules when the at least one quality metric is satisfied; and determine the adequacy metric based on a second set of adequacy rules when any of the at least one quality metric is not satisfied.

[0047] In various embodiments of the system, the instructions, when executed by the at least one processor, can cause the system to determine that the adequacy metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture events of interest in at least said portions of the GIT, regardless of whether such events of interest actually exist in at least said portions of the GIT, and the adequacy indication of the CE procedure includes at least one reason why the CE procedure is determined to be insufficient.

[0048] In various embodiments of the system, the event of interest is a significant polyp, and the instruction, when executed by the at least one processor, causes the system to determine that the adequacy metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture the event of interest in at least said portions of the GIT, regardless of whether such an event of interest is actually present in at least said portions of the GIT, and to determine that a significant polyp is detected in the plurality of images by a polyp detector processing the plurality of images. The adequacy indication of the CE procedure may include an indication that the CE procedure is determined to be insufficient, but such determination is rejected by the polyp detector.

[0049] According to various aspects of this disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause to perform a method comprising: accessing a plurality of images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging apparatus during a CE procedure; accessing a plurality of characteristic measures associated with the plurality of images; determining an adequacy measure of the CE procedure based on the plurality of characteristic measures, wherein the adequacy measure provides a measure of whether the imaging coverage provided by the plurality of images is sufficient to capture an event of interest in at least the portion of the GIT regardless of whether such an event of interest is actually present in at least the portion of the GIT; and displaying an adequacy indication of the CE procedure based on the adequacy measure.

[0050] In various embodiments of the non-transitory computer-readable medium, the instructions, when executed by the processor, cause further execution of a method including: accessing at least one quality metric associated with the plurality of images; determining the adequacy indication based on a first set of adequacy rules when the at least one quality metric is satisfied; and determining the adequacy metric based on a second set of adequacy rules when any of the at least one quality metric is not satisfied.

[0051] Further details and aspects of exemplary embodiments of this disclosure are described below with reference to the accompanying drawings. Attached Figure Description

[0052] The above and other aspects and features of this disclosure will become more apparent when considered in conjunction with the following detailed description, in which similar reference numerals denote similar or identical elements.

[0053] Figure 1 It is a simplified diagram showing the gastrointestinal tract (GIT);

[0054] Figure 2 This is a block diagram of an exemplary system for analyzing medical images captured in vivo via capsule endoscopy (CE) surgery, according to various aspects of this disclosure.

[0055] Figure 3 This is a block diagram of an exemplary computing device that can be used with the system disclosed herein;

[0056] Figure 4 It is a simplified diagram showing the large intestine;

[0057] Figure 5 This is a block diagram of an exemplary deep learning neural network and the inputs and outputs of the deep learning neural network, based on various aspects of this disclosure;

[0058] Figure 6 Based on all aspects of this disclosure Figure 5 A simplified diagram of the layers of a deep learning neural network;

[0059] Figure 7 This is a block diagram of an exemplary classic machine learning classifier based on various aspects of this disclosure;

[0060] Figure 8 Based on all aspects of this disclosure. Figure 2 The CE device captures an exemplary image that will have a poor cleaning score;

[0061] Figure 9A This is an exemplary graph showing the output of the motion detector for the cecum according to various aspects of this disclosure;

[0062] Figure 9B This is an exemplary graph showing the output of a motion detector for an image of the ascending colon according to various aspects of this disclosure;

[0063] Figure 9C This is an exemplary graph showing the output of a motion detector for an image of the transverse colon according to various aspects of this disclosure;

[0064] Figure 9D This is an exemplary graph showing the output of a motion detector for an image of the descending colon, based on various aspects of this disclosure;

[0065] Figure 9E This is an exemplary graph showing the output of a motion detector for an image of the rectum according to various aspects of this disclosure;

[0066] Figure 10 This is a flowchart of an exemplary method for estimating the adequacy of capsule endoscopy, based on various aspects of this disclosure;

[0067] Figure 11 This is a flowchart of another exemplary method for estimating the adequacy of capsule endoscopy, based on various aspects of this disclosure;

[0068] Figure 12 This is a flowchart of an exemplary method for identifying groups of images capturing the same tissue region, based on various aspects of this disclosure;

[0069] Figure 13 This is a simplified diagram of an exemplary set of images based on various aspects of this disclosure;

[0070] Figure 14 This is a flowchart of an exemplary method for estimating the average cleanliness ratio based on various aspects of this disclosure;

[0071] Figure 15AThis is an exemplary histogram of the number of polyp images with various cleanliness scores according to various aspects of this disclosure;

[0072] Figure 15B This is an exemplary histogram of the number of colon images with various cleanliness scores according to various aspects of this disclosure;

[0073] Figure 16 Based on all aspects of this disclosure Figure 15A and Figure 15B An example histogram of the cleaning ratio is a graph.

[0074] Figure 17 This is a flowchart of an exemplary method for estimating a measure of adequacy of capsule endoscopy based on multiple segment scores, according to various aspects of this disclosure.

[0075] Figure 18 This is a diagram that provides an exemplary mapping of various aspects of this disclosure;

[0076] Figure 19 It is a diagram illustrating exemplary sufficiency rules for classifying surgeries, based on various aspects of this disclosure; and

[0077] Figure 20 This is a diagram illustrating another set of exemplary rules for classifying surgeries, based on various aspects of this disclosure. Detailed Implementation

[0078] This disclosure relates to systems and methods for analyzing medical images, and more specifically, to systems and methods for estimating the adequacy of a capsule endoscopy (CE) procedure after its completion, such as estimating whether the imaging coverage provided by the image stream captured in vivo via the capsule endoscopy (CE) procedure is sufficient to capture at least one polyp or other event of interest (regardless of whether any polyp or other event of interest is actually present). The estimated adequacy of the procedure can be used by clinicians to understand whether the CE procedure was sufficient to capture the event of interest (regardless of whether any event of interest is actually present). The estimated adequacy of the CE procedure can also be used to automatically exclude a procedure from the study report when the procedure is estimated to be inadequate. Although examples are shown and described with respect to images captured in vivo by a CE device, the disclosed techniques can be applied to images captured by other devices or institutions.

[0079] The term “adequacy” and its derivatives, as used in this article in relation to surgery, can refer to a measure of whether the imaging coverage provided by the image stream captured by surgery is sufficient to capture the event of interest (regardless of whether any event of interest actually exists).

[0080] As used herein with respect to the term "exclusion" and its derivatives, it may include indications that the CE procedure provided was insufficient to capture the event of interest and / or that the quality of the CE procedure outcome was below a threshold level. For example, the images may be very blurry and / or the results may have resulted in the loss of a significant number of images due to connectivity issues between the CE device and the system. Depending on the circumstances, no CE study report will be generated for excluded procedures.

[0081] As used herein with respect to surgery, the terms “predefined event” and “event of interest” and their derivatives can be and include periodic events (such as contraction), transient events (such as fresh bleeding), or constant events (such as polyps since their first appearance), as well as other events. The terms “predefined event” and “event of interest” can also include, but are not limited to, the following: pathological type, at least one occurrence of a pathological type, all occurrences of a pathological type in a portion of the GIT, at least one occurrence of a pathological type of a certain size, all occurrences of a pathological type of a certain size in a portion of the GIT, polyps, at least one occurrence of polyps, all occurrences of polyps in the colon, at least one occurrence of polyps larger than a certain size in the colon, all occurrences of polyps larger than a certain size in the colon, parasites, disease indicators or appearance, contraction, fresh bleeding, stenosis and / or disease, and other events.

[0082] As used herein with reference to surgery, the term "characteristic measure" and its derivatives may be or may include a measure of the presence or absence of a characteristic, or the degree to which such a characteristic may be present or absent. In various embodiments, the value of a characteristic measure may be determined by processing the image stream captured by the CE surgery. It is conceivable that some characteristic measures may be binary (e.g., retention: its presence or absence), and some characteristic measures may be scores (e.g., gastrointestinal cleansing score).

[0083] In the following detailed description, specific details are set forth to provide a full understanding of this disclosure. However, those skilled in the art will understand that this disclosure can be practiced without these specific details. In other instances, well-known methods, processes, and components have not been described in detail so as not to obscure the content of this disclosure. Some features or elements described with respect to one system may be combined with features or elements described with respect to other systems. For clarity, discussion of the same or similar features or elements may not be repeated.

[0084] While this disclosure is not limited in this respect, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “establishing,” “analyzing,” and “checking” can refer to multiple operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or transform physical (e.g., electronic) quantities of data represented in the registers and / or memory of a computer into other information represented in the registers and / or memory of a computer or in a non-transitory storage medium that may store instructions for performing operations and / or processes. While this disclosure is not limited in this respect, the terms “plurality” and “a plurality” as used herein can include, for example, “multiple” or “two or more.” Throughout this specification, the terms “plurality” or “a plurality” can be used to describe two or more components, devices, elements, units, parameters, etc. The set of terms used herein can include one or more items. Unless explicitly stated otherwise, the methods described herein are not limited to a particular order or sequence. Furthermore, some of the described methods or elements thereof may occur or be executed simultaneously, at the same point in time, or in parallel. Throughout this specification, the term "classification" may be used to refer to the decision to assign one of a set of categories to an image / frame. Throughout this specification, the term "classification score" may be used to describe a vector of values ​​generated by a machine learning system / model for a set of categories applicable to an image / frame. Throughout this specification, the term "classification probability" may be used to describe transforming the classification score into a value reflecting the probability that each category in that set is applicable to the image / frame. This transformation may involve the use of other factors, values, or functions, and may use one or more algorithms including a machine learning system / model.

[0085] As used in this article with respect to the image, the term "position" and its derivatives may refer to the estimated position of the capsule along the GIT when the image is captured, or the estimated position of the portion of the GIT shown in the image along the GIT.

[0086] The type of CE procedure can be determined, in particular, based on the part of the GIT of interest and to be imaged (e.g., the colon or small intestine (“SB”)) or based on the specific purpose (e.g., for examining the status of GI diseases (such as Crohn’s disease) or for colon cancer screening).

[0087] Unless otherwise specifically indicated, the terms “around” or “adjacent” as used herein with respect to images (e.g., images surrounding another image or adjacent to other images) may refer to spatial and / or temporal characteristics. For example, an image surrounding another image or adjacent to other images may be an image estimated to be located near other images along the GIT, and / or an image captured near the capture time of another image, within a certain threshold (e.g., within one or two centimeters, or within one, five, or ten seconds).

[0088] The terms “GIT” and “part of GIT” can refer to or include the other, depending on their context. Therefore, the term “part of GIT” can refer to the entire GIT, while the term “GIT” can refer to only a portion of the GIT. As used herein, the term “segmentation” can refer to identifying one or more transition points in an image stream.

[0089] As used herein, the term “segmentation” or “division” can refer to one or more transition points between segments or portions of the gastrointestinal tract (GIT) in an image stream.

[0090] As used in this article, the term "distal" refers to the part of the GIT that is further away from the mouth, while the term "proximal" refers to the part of the GIT that is closer to the mouth.

[0091] The terms “image” and “frame” may refer to or include the other, and may be used interchangeably in this disclosure to refer to a single capture by an imaging device. For convenience, the term “image” may be used more frequently in this disclosure, but it should be understood that references to an image should also apply to a frame.

[0092] Throughout this specification, the terms "(multiple) classification scores" or "(multiple) scores" may be used to indicate a value or value vector applicable to one class or set of classes of an image / frame. In various embodiments, the values ​​or value vectors of one or more classification scores may or may reflect probabilities. In various embodiments, the model may output classification scores that are probabilistic. In various embodiments, the model may output classification scores that are not probabilistic.

[0093] The term "classification probability" can be used to describe a classification score as a probability, or to describe a transformation of a classification score that is not a probability into a value reflecting the probability of each of a set of classes applicable to an image / frame. From the context, it should be understood that various references to "probability" refer to classification probability and are abbreviations for classification probability.

[0094] As used herein, "machine learning system" means and includes any computational system that implements any type of machine learning. As used herein, "deep learning neural network" means and includes neural networks with several hidden layers that do not require feature selection or feature engineering. In contrast, a "classical" machine learning system is a machine learning system that requires feature selection or feature engineering.

[0095] refer to Figure 1 The diagram illustrates the GIT 100. The GIT 100 is an organ system in humans and other animals. The GIT 100 typically includes a mouth 102 for ingesting food, salivary glands 104 for producing saliva, an esophagus 106 through which food passes with the aid of contractions, a stomach 108 for secreting enzymes and gastric acid to aid digestion, a liver 110, a gallbladder 112, a pancreas 114, a small intestine 116 (“SB”) for absorbing nutrients, and a colon 400 (e.g., the large intestine) for storing water and waste as feces before defecation. The colon 400 typically includes an appendix 402, a rectum 428, and an anus 430. Food ingested through the mouth is digested by the GIT to absorb nutrients, and the remaining waste is expelled as feces through the anus 430.

[0096] Study reports for different parts of the GIT 100 (e.g., SB), colon 400, esophagus 106, and / or stomach 108 can be presented via a suitable user interface. As used herein, the term "study report" refers to and includes reports from images produced by a CE imaging device (e.g., Figure 2 (212) At least one set of images selected from images captured during a single CE procedure performed on a specific patient at a specific time, and optionally may include information other than images. The type of procedure performed can determine which part of the GIT100 is of interest. Examples of types of procedures performed include, but are not limited to, SB procedure, colon procedure, SB and colon procedure, procedure designed specifically to expose or examine the SB, procedure designed specifically to expose or examine the colon, procedure designed specifically to expose or examine the colon and SB, or procedure used to expose or examine the entire GIT (i.e., esophagus, stomach, SB, and colon).

[0097] Figure 2 A block diagram of a system for analyzing medical images captured in vivo via CE surgery is shown. The system typically includes a capsule system 210 configured to capture images of GIT and a computing system 300 (e.g., a local system and / or a cloud system) configured to process the captured images.

[0098] Capsule system 210 may include a swallowable CE imaging device 212 (e.g., a capsule) configured to capture images of the GIT as the CE imaging device 212 passes through it. The images may be stored on the CE imaging device 212 and / or transmitted to a receiving device 214, which typically includes an antenna. In some capsule systems 210, the receiving device 214 may be located on the patient who has swallowed the CE imaging device 212 and may take the form, for example, a strap worn by the patient or a patch attached to the patient.

[0099] Capsule system 210 can be communicatively coupled to computing system 300 and can transmit captured images to computing system 300. Computing system 300 can process the received images using techniques such as image processing, machine learning, and / or signal processing. Computing system 300 may include a local computing device located at the patient's and / or patient treatment facility's premises, a cloud computing platform provided by cloud services, or a combination of a local computing device and a cloud computing platform.

[0100] In cases where the computing system 300 includes a cloud computing platform, images captured by the capsule system 210 can be transmitted online to the cloud computing platform. In various embodiments, images can be transmitted via a receiving device 214 worn or carried by the patient. In various embodiments, images can be transmitted via the patient's smartphone or via any other device connected to the Internet and capable of being coupled to the CE imaging device 212 or the receiving device 214.

[0101] Figure 3 A high-level block diagram of an exemplary computing system 300 that can be used with the image analysis system disclosed herein is shown. The computing system 300 may include a processor or controller 305 (which may be, for example, one or more central processing units (CPUs), one or more graphics processing units (GPUs or GPGPUs), a chip, or any suitable computing device), an operating system 215, a memory 320, a storage device 330, an input device 335, and an output device 340. The CE imaging device 212 is used for collecting or receiving (e.g., a receiver worn by a patient) or displaying or selecting a display (e.g., a workstation). Figure 2 The module or device that collects medical images may be or includes Figure 3 The computing system 300 shown may be executed by the computing system. The communication component 322 of the computing system 300 may allow communication with remote or external devices, for example, via the Internet or another network, via radio, or via a suitable network protocol such as File Transfer Protocol (FTP).

[0102] The computing system 300 includes an operating system 315, which may be or may include any code segment designed and / or configured to perform tasks involving coordinating, scheduling, arbitrating, supervising, controlling, or otherwise managing the operations of the computing system 300 (e.g., the execution of a scheduler). The memory 320 may be or may include, for example, random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffers, short-term memory cells, long-term memory cells, or other suitable memory cells or storage units. The memory 320 may be or may include multiple possibly different memory cells. The memory 320 may store, for example, instructions for performing methods (e.g., executable code 325), and / or data such as user responses, interrupts, etc.

[0103] Executable code 325 can be any executable code, such as an application, program, process, task, or script. Executable code 325 can be executed by controller 305, possibly under the control of operating system 315. For example, execution of executable code 325 can result in the display or selection of medical images as described herein. In some systems, more than one computing system 300 or components of computing system 300 can be used for the various functions described herein. One or more computing systems 300 or components of computing system 300 can be used for the various modules and functions described herein. Devices including components similar to or different from those included in computing system 300 can be used, and these devices can be connected to a network and used as a system. One or more processors can be configured to perform the methods disclosed herein by, for example, executing software or code. Storage device 330 can be or may include, for example, a hard disk drive, floppy disk drive, optical disc (CD) drive, recordable CD (CD-R) drive, universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. For example, data such as instructions, code, medical images, and image streams can be stored in storage device 330 and loaded from storage device 330 into memory 320, where the data can be processed by controller 305. In some embodiments, this can be omitted. Figure 3 Some of the components shown.

[0104] Input device 335 may include, for example, a mouse, keyboard, touchscreen, or touchpad, or any suitable input device. It will be appreciated that any suitable number of input devices may be operatively coupled to computing system 300. Output device 340 may include one or more monitors, screens, displays, speakers, and / or any other suitable output devices. It will be appreciated that any suitable number of output devices may be operatively coupled to computing system 300, as shown in box 340. Any suitable input / output (I / O) device may be operatively coupled to computing system 300; for example, a wired or wireless network interface card (NIC), modem, printer or fax machine, universal serial bus (USB) device, or external hard drive may be included in input device 335 and / or output device 340.

[0105] include Figure 3 Multiple computer systems 300, including some or all of the components shown, can be used with the described systems and methods. For example, the CE imaging apparatus 212, receiver, cloud-based system, and / or workstation or portable computing device for displaying images may include... Figure 3 Some or all of the components of a computer system. Including, for example... Figure 3 The cloud platform (e.g., a remote server) containing components such as the computing system 300 can receive surgical data, such as images and metadata, process and generate research reports, and display the generated research reports for physician review (e.g., on a web browser running on a workstation or portable computer). The "on-premises" option allows the use of workstations or local servers within the healthcare facility to store, process, and display images and / or research reports.

[0106] According to some aspects of this disclosure, users (e.g., clinicians) can build their understanding of cases by reviewing research reports, which include the display of images (e.g., captured by the CE imaging device 212) that are automatically selected, for example, as images of potential interest. Reference Figure 4 The diagram illustrates a view of the colon 400. The colon 400 absorbs water, and any remaining waste is stored as feces before being expelled through defecation. For example, the colon 400 can be divided into five anatomical segments: the cecum 404, the right colon or ascending colon 410, the transverse colon 416, the left colon or descending colon 422 (e.g., the left colon-sigmoid colon 424), and the rectum 428.

[0107] The ileum 408 is the final part of the ileocecal valve (SB), leading to the cecum 404 and separating from it via a muscular valve called the ileocecal valve (ICV) 406. The ICV 406 also connects the ileum 408 to the ascending colon 410. The cecum 404 is the first part of the colon 400. The cecum 404 includes the appendix 402. The next part of the colon 400 is the ascending colon 410. The ascending colon 410 connects to the small intestine via the cecum 404. The ascending colon 410 ascends through the abdominal cavity towards the transverse colon 416.

[0108] The transverse colon 416 is the section of the colon 400 from the hepatic flexure (also known as the right colonic flexure 414) (where the colon 400 bends beside the liver) to the splenic flexure (also known as the left colonic flexure 418) (where the colon 400 bends beside the spleen). The transverse colon 416 hangs outside the stomach and is attached to it by a greater peritoneal fold called the greater omentum. Posteriorly, the transverse colon 416 is attached to the posterior abdominal wall by a mesentery called the transverse mesentery.

[0109] The descending colon 422 is the section of colon 400 from the left colic flexure 418 to the beginning of the sigmoid colon 426. One function of the descending colon 422 in the digestive system is to store feces that will be emptied into the rectum. The descending colon 422 is also called the distal intestine because it is located further away from the proximal intestine along the gastrointestinal tract. The gut microbiota in this region is usually very dense. The sigmoid colon 426 is the section of colon 400 that follows the descending colon 422 and precedes the rectum 428. The name sigmoid means S-shaped. The walls of the sigmoid colon 426 are muscular and contract to increase pressure within colon 400, thus moving feces into the rectum 428. The sigmoid colon 426 is supplied with blood by several branches of the sigmoid colic artery (usually between 2 and 6).

[0110] The rectum 428 is the last part of the colon 400. The rectum 428 holds the formed feces in preparation for expulsion via defecation.

[0111] CE Imaging Device 212 ( Figure 2 The CE imaging device 212 can be used to image the interior of the colon 400. It enters the colon 400 from the SB via the ICV 406. Normally, after entering the colon 400 via the ICV 406, the CE imaging device 212 enters the cecum 404. However, the CE imaging device 212 may occasionally skip the cecum 404 and enter directly into the ascending colon 410. The colon 400 can be wide enough to allow the CE imaging device 212 to move almost unrestricted. The CE imaging device 212 can rotate and roll. The CE imaging device 212 can remain in one place for a long time, can move very quickly through the colon 400, or can move backward through the preceding segment of the colon 400.

[0112] Typically, the segmentation of the GIT into anatomical segments can be performed, for example, based on the identification of the CE imaging device 212 channels between different anatomical segments. This identification can be performed, for example, based on machine learning techniques. Segmentation can also be envisioned based on, for example, diseased and healthy segments of the region of interest, and / or based on specific pathologies and / or combinations thereof. For example, diseases such as Crohn's disease are characterized by diffuse pathology that spreads across parts of the GIT in an almost “carpet-like” manner.

[0113] refer to Figure 5 The diagram illustrates a block diagram of a deep learning neural network 500 for image classification according to some aspects of this disclosure. In some systems, the deep learning neural network 500 may include a convolutional neural network (CNN) and / or a recurrent neural network. Typically, a deep learning neural network includes multiple hidden layers. As explained in more detail below, the deep learning neural network 500 may utilize one or more CNNs to classify images by the CE imaging device 212 (see [link to CE imaging device 212]). Figure 2 One or more images captured by a deep learning neural network are classified as part of a GIT. A deep learning neural network 500 can be used in a computer system 300 ( Figure 3 This will be executed on [the target platform]. Those skilled in the art will understand the Deep Learning Neural Network 500 and how to implement it.

[0114] In machine learning, CNNs (Convolutional Neural Networks) are a class of artificial neural networks most commonly used for analyzing visual images. The convolutional aspect of a CNN involves applying matrix processing operations to local portions of an image, and the results of these operations (which may involve dozens of different parallel and serial computations) are fed into a collection of features for the next layer. CNNs typically consist of convolutional layers, activation function layers, and deconvolutional layers (e.g., in segmented networks) and / or pooling (usually max pooling) layers, which are used to reduce dimensionality without losing too much feature. Additional information may be included in the operations that generate these features. The unique information that provides the features that give rise to the neural network can ultimately be used to provide an aggregation method to distinguish different data input to the neural network.

[0115] Figure 6A topology of a deep learning neural network 500 is shown, comprising at least one input layer 610, multiple hidden layers 606, and at least one output layer 620. The input layer 610, the multiple hidden layers 606, and the output layer 620 all include neurons 602 (e.g., nodes). The neurons 602 between the various layers are interconnected via weights 604. Each neuron 602 in the deep learning neural network 500 computes an output value by applying a specific function to the input value from the previous layer. The function applied to the input value is determined by a vector of weights 604 and biases. In a deep learning neural network, learning is performed by iteratively adjusting these biases and weights. The deep learning neural network 500 can output logits.

[0116] Refer again Figure 5 The deep learning neural network 500 can be trained based on labeled training images and / or objects within those images. For example, the image could be a portion of a GIT (e.g., the rectum or cecum). In some methods according to this disclosure, training may include supervised learning. Training may further include augmenting the training images, including adding noise, changing colors, hiding portions of the training images, scaling the training images, rotating the training images, and / or stretching the training images. Those skilled in the art will understand how to train the deep learning neural network 500 and how to perform such training.

[0117] In some methods according to this disclosure, the deep learning neural network 500 can be used to process images from the CE imaging device 212 (see [link]). Figure 2 The captured image 502 is classified. The classification of image 502 can be used to determine classification scores for various feature measures 506 to determine the adequacy measure of the CE procedure. For example, image classification may include classifying images as cecum, ascending colon, transverse colon, descending colon, or rectum. Each image may include a classification score for each of consecutive segments of the GIT. The classification score includes the output (e.g., logits) of a classical machine learning classifier 700 after applying a function such as SoftMax to make the output represent probabilities. As mentioned above, feature measures are measures of the presence and / or degree of presence or absence of features in multiple images.

[0118] refer to Figure 7A classical machine learning classifier 700 is illustrated according to some aspects of this disclosure. As used herein, the term "classical machine learning classifier" refers to a machine learning-based classifier that requires feature selection and / or feature engineering on the classifier's input. In contrast, deep learning neural networks are examples of machine learning-based classifiers that do not require feature engineering or feature selection. As explained in more detail below, the classical machine learning classifier 700 can be configured to provide scores for various feature metrics, such as motion scores and / or cleanliness scores. The classical machine learning classifier 700 may include linear logistic regression classifiers, decision trees, and / or support vector machines (SVMs). In various embodiments, the classical machine learning classifier 700 does not include CNNs or other deep learning networks. Those skilled in the art will understand how such a classical machine learning system can be implemented.

[0119] A linear logistic regression classifier is a classic machine learning classifier. It estimates the parameters of a logistic model that best describes the probability of each sample belonging to each of the classes. A linear logistic regression classifier is also a supervised learning model. It estimates the parameters of the logistic model. A support vector machine (SVM) is a supervised learning model with an association learning algorithm that analyzes data used for classification. In various embodiments, the output of an SVM can be normalized between "0" and "1".

[0120] In various ways, SoftMax can be configured to map the network's unnormalized output (e.g., the logits of a deep learning neural network and / or a classic machine learning classifier 700) to a probability distribution using one or more of the predicted output classes from the classification scores (e.g., the classification scores of a deep learning neural network). SoftMax is a function that takes a vector of N real numbers as input and normalizes it to a probability distribution consisting of N probabilities proportional to the exponent of the input values. That is, before applying SoftMax, some vector components might be negative or greater than one, and their sum might not equal 1. However, after applying SoftMax, each component will be in the interval (0,1), and the components will sum to 1, making them interpretable as probabilities.

[0121] The classic machine learning classifier 700 can be trained in a supervised manner. Images of parts of the GIT can be labeled and used as training data. Those skilled in the art will understand how to train the classic machine learning classifier 700 and how to implement such training.

[0122] In some methods according to this disclosure, a classical machine learning classifier 700 can be used for imaging devices 212 (see [link to disclosure]). Figure 2The captured images provide classification probabilities for each segment of the GIT. The classification probabilities of the images can include classification probabilities for each image for consecutive segments of the GIT. Segments of the GIT can include, but are not limited to, for example, the SB or a portion thereof (e.g., where the SB can be divided according to length), or the colon or a portion thereof, where, for example, the colon can be divided into multiple segments or regions, such as the cecum, ascending colon, transverse colon, descending colon, and / or rectum. For example, image classification probabilities can be labeled as a portion of the colon (e.g., cecum, ascending colon, transverse colon, descending colon, and / or rectum).

[0123] The following text combines Figure 8 and Figures 9A to 9E Various characteristic measures are described. Such characteristic measures can be obtained through deep learning neural networks (e.g., Figure 5 (500) and / or through classic machine learning systems (e.g., Figure 7 The 700 (or other technologies) may be provided. As explained in more detail later herein, the characteristic metrics can be used to estimate whether the CE procedure is sufficient to capture the event of interest (regardless of whether any event of interest is present). The characteristic metrics disclosed below are exemplary, and other characteristic features are considered to be within the scope of this invention.

[0124] According to various aspects of this disclosure, characteristic metrics may include a cleanliness score that indicates the degree of cleanliness shown in the image. As those skilled in the art will understand, "cleanliness" means the removal of obstructions from the gastrointestinal tract (GIT) so that the GIT can be effectively imaged. Obstructions may include, for example, feces or air bubbles, and other substances. Figure 8 An exemplary image of poor cleaning, captured by a CE device, is shown. The image includes a large amount of fecal residue, obscuring a clear view of the GIT (Geometric Intent). According to various aspects of this disclosure, deep learning neural networks (e.g., Figure 5 (500) and / or classic machine learning systems (e.g., Figure 7 (e.g., 700) or another technique can be used to determine the cleanliness of each image in an image stream captured via CE surgery. Those skilled in the art will recognize various ways of determining a cleanliness score, such as using the techniques described in the following application: Klein A, Gizbar M, Bourke M, Ahlenstiel G. “A Validated Computerized Cleansing Score for Video Capsule Endoscopy”, Dig. Endosc. 2015; 28:564–569, which is hereby incorporated herein by reference in its entirety. These and other techniques used to determine cleanliness scores are considered to be within the scope of this invention.

[0125] According to various aspects of this disclosure, characteristic metrics may include a motion score of an image, which estimates the motion of the CE imaging device (e.g., when the CE imaging device captures the image) while the CE imaging device (e.g., ...) is capturing the image. Figure 2 The degree of motion experienced by 212). Figures 9A to 9E An exemplary graph showing motion scores versus time for images captured in various segments of the GIT is presented. Figure 9A A graph showing the motion score of the cecal portion of the GIT versus time is presented. In the graph, the CE device typically has a low motion score. Figure 9B The graph shows the motor score of the ascending colon portion of the GIT versus time. In this graph, the CE device lasted approximately 2 seconds in the ascending portion and had a relatively high motor score, averaging over 0.5. (Reference) Figure 9C The graph shows the motility score of the transverse portion of the colon against time. The motility score of the CE device is higher at the beginning and end of the graph. (Reference) Figure 9D This chart shows the motor score of the descending portion of the colon against time. This chart covers a range of approximately 3500 seconds. The highest motor scores are found in this chart, averaging approximately 2500 to 3000 seconds. Reference Figure 9E A graph showing the motion score of the rectum versus time is presented. In this graph, the average motion score is almost zero. Those skilled in the art will recognize techniques that can be used to process images to provide motion scores, such as those described herein by reference in U.S. Patent No. 8,792,691, which is incorporated herein by reference in its entirety. These and other techniques are considered to be within the scope of the invention used to determine motion scores. In various embodiments, the motion score may be a characteristic metric. In various embodiments, the characteristic metric may be determined by counting the number of frames with motion scores above a predetermined threshold. In various embodiments, such a characteristic metric may be determined for a segment of a portion of the GIT. For example, the characteristic metric may be determined based on a calculation of 40 frames of motion present in the cecum. These and other embodiments are considered to be within the scope of the invention.

[0126] Figure 8 and Figures 9A to 9EThese are exemplary, and other characteristic measures used to determine the adequacy of CE surgery are considered to be within the scope of the invention. For example, in various embodiments, characteristic measures may include one or more of the following: anatomical colonic segment associated with an image, transport pattern of the capsule endoscope device, CE device communication errors, anatomical landmarks in multiple images, coverage of GIT tissue in multiple images, transport time, per-image indication that an image includes or does not include at least one polyp, a percentage of time indicating the capsule endoscope device capturing an image relative to the duration the capsule endoscope device remains within the GIT portion of interest, and / or a percentage of progress indicating the capsule's displacement up to each image and relative to the overall GIT portion to be imaged, and others. These and other embodiments are considered to be within the scope of the invention.

[0127] Figure 10 The flowchart illustrates a computer-implemented method 1000 for estimating the adequacy of capsule endoscopy. In various aspects, the images may include portions of the GIT detailed above. Those skilled in the art will understand that one or more operations of method 1000 may be performed in a different order, repeated, and / or omitted without departing from the scope of this disclosure. In some methods according to this disclosure, some or all of the operations in the illustrated method 1000 may be performed using a capsule endoscope (e.g., CE imaging device 212 (see...)). Figure 2 ), receiving device 214 (see Figure 2 ) and computing system 300 (see Figure 2 This can be operated as described above. Other variations are considered to be within the scope of this invention. Figure 10 The operation will concern the computing device used to analyze medical images captured in vivo via CE surgery (e.g., system 200). Figure 2 The description refers to a computing system 300 or any other suitable computing system device, including a remotely configured computing device, or its location. It should be understood that the operations shown also apply to other systems and their components.

[0128] As mentioned above, the adequacy measure of CE procedure can provide a metric for whether the imaging coverage provided by the image stream captured during the CE procedure is sufficient to capture the event of interest (regardless of whether any event of interest is present). An advantage is the reduction of false rejections, where a patient is incorrectly removed from the medical condition because the image stream does not visualize any events or indications related to the medical condition. If the CE procedure is determined to be inadequate, the calculation system 300 can recommend repeating the CE procedure or provide information with a warning recommending repeating the procedure.

[0129] Initially, at box 1002, the operation includes accessing images (e.g., time-series images) of at least a portion of the GIT (e.g., colon 400) captured by the CE device during CE surgery. The multiple images may be one of the following: all images captured during CE surgery and uploaded (or received) from the CE imaging device (and / or computing system 300); all images of the GIT of interest (e.g., esophagus, SB, colon, SB, and / or colon) captured and received / uploaded from the computing system 300; and all images of a predefined segment of the region of interest or portion of the GIT captured and received / uploaded from the computing system 300 (e.g., the transverse colon when the region of interest is the colon).

[0130] At box 1004, the operation includes accessing one or more feature measures associated with the image, such as one or more of the feature measures described above. In various respects, the feature measures(s) can be selected in a clinically reasonable manner, which provides the advantage of explaining to clinicians the underlying reasons for excluding certain CE procedures that are insufficient, thus offering a better level of adoption for users of this technology. In various respects, the feature corresponding to the feature measures(s) can be determined based on the level of the feature or the measured correlation between its presence and the adequacy of the procedure.

[0131] In all respects, (multiple) feature measures can be determined based on the accessed image, as explained above, and can be or may be based on motion scores. Figures 9A to 9E ) and / or cleaning score ( Figure 8 As mentioned above, a characteristic metric based on motion score could be the number of images the CE device captures in motion, indicating its motion score. Similarly, a characteristic metric based on cleanliness score could be the average cleanliness score of each segment of the GIT. In all respects, operation can determine the overall characteristic metric of all segments of the GIT by averaging the cleanliness scores of each segment.

[0132] In various aspects, the (multiple) characteristic measures may include the anatomical colonic segment in which images are captured, the transport pattern of the capsule endoscope device, CE device communication errors, anatomical landmarks in multiple images, and / or the coverage of GIT tissue in multiple images. Those skilled in the art will understand how such characteristic measures can be determined based on this disclosure, by reference to documents incorporated herein by reference, and / or knowledge of the art.

[0133] In various aspects, the incomplete surgical characteristic metric can be based on indications of the absence of colon visualization in multiple images, possible visualization of the colon in multiple images, and / or failure to leave the body (e.g., where the CE device has not left the patient's body). The incomplete surgical characteristic metric can have a value of 1 or 0. In various aspects, the score can be zero if the CE device remains in the GIT. For example, the incomplete surgical characteristic metric can have a value of zero in cases where the CE device has not reached the colon or the captured images may only cover a portion of the colon (e.g., due to technical problems, power depletion, etc.). In various aspects, the incomplete surgical characteristic metric can be determined by a machine learning system.

[0134] Some characteristic metrics can be viewed as segmental metrics because such metrics apply to the characteristics of segments / parts of the GIT. Some characteristics can be viewed as global characteristic metrics because such metrics apply to every part of the procedure. Some characteristics can involve incomplete surgical features that indicate whether the procedure is incomplete for any reason, as described above.

[0135] At box 1006, the operation includes determining a adequacy measure of the procedure. In various embodiments, the adequacy measure of the procedure may be based on adequacy measures of various segments of the GIT, a per-procedure global adequacy measure, and / or an incomplete procedure characteristic measure, as described in more detail below. For now, it is only necessary to note that in various embodiments, the adequacy measure of the procedure can be determined by multiplying a weighted segmental adequacy measure, a weighted global adequacy measure, and / or a weighted incomplete procedure characteristic measure of one or more segments of the GIT.

[0136] In each respect, each of the multiple images of the GIT can be associated with one of the multiple consecutive segments of the GIT (e.g., cecum, ascending colon, transverse colon, descending colon, and / or rectum). In each respect, the operation can determine a segmental adequacy measure for each of the multiple consecutive segments of the GIT based on one or more of the following: a motility score, a per-segment cleanliness score, transit time, per-image indication that the image does not contain at least one polyp, a percentage of time indicating the duration the capsule endoscope captured the image relative to the duration the capsule endoscope remained within the GIT segment of interest, and / or a percentage of progress indicating the displacement of the capsule up to each image and relative to the overall GIT segment to be imaged. For example, the operation can analyze each of the multiple images of the GIT to determine a score indicating the average cleanliness level of each segment. Images may include poor cleanliness. For example, an image may include large amounts of feces or feces or dark fluid sufficient to prevent reliable examination. In each respect, a score for each segment of the GIT can be determined, and then a segmental adequacy measure can be determined based on the per-segment score. In each respect, different characteristics can be used for different segments when determining the per-segment score. For example, for the cecum, motility scores can be used to determine cecal segment adequacy measures, and for the ascending colon, cleanliness scores can be used to determine ascending colon segment adequacy measures. In each case, the per-segment adequacy measure of a segment can be multiplied by the per-segment adequacy measure of the previous segment. In each case, segmental adequacy measures can be determined by machine learning systems.

[0137] In various embodiments, a segmental adequacy measure can be a product of at least two of the following: motility score, cleanliness level per segment, and / or transit time. Multiplication is used as an example, and any other function for combining scores is envisioned. In various aspects, a score for each segment of the GIT can be determined, and a segmental adequacy measure can be determined based on the per-segment score of the GIT. In various aspects, a region score can be used. For example, the colon can be divided into two regions (e.g., combining the first three segments of the proximal segment and the last two segments of the distal segment). In various aspects, the segmental probability for each segment can be a non-linear function based on the motility score or the cleanliness level and / or transit time per segment. The segmental adequacy measure can then be based on multiplying all segmental probabilities. In various embodiments, this multiplication can be replaced by other functions, such as a weighted average function. Multiple segmental adequacy measures can be used in various ways to determine the adequacy measure of the procedure.

[0138] As mentioned above, multiple per-surgery global adequacy measures can be calculated for all images and for all GIT segments imaged by the CE device. In various aspects, the global adequacy measure can be based on one or more per-surgery global characteristic measures as well as other measures. In various aspects, the global adequacy measure can be based on the average cleanliness score of all segments, patient demographics, the last segment of the GIT reached by the CE device, and / or the absolute time spent by the CE device in the GIT segment. Patient demographics can include, but are not limited to, age, sex, BMI, weight, height, smoking, family history of colorectal cancer, and / or nutrition. For example, the procedure can utilize the lower adequacy measure threshold for female patients compared to male patients to determine the adequacy of the procedure for certain events of interest. In various aspects, the global adequacy measure can be determined by a machine learning system. The global adequacy measure can be used in various ways to determine the adequacy of the procedure.

[0139] As described above, the adequacy measure of the procedure provides a measure of whether the imaging coverage provided by the image stream captured by the CE procedure is sufficient to capture the event of interest (regardless of whether any event of interest is present). Events of interest can include contraction, fresh bleeding, stenosis, at least one polyp (e.g., a prominent polyp), and / or disease. For example, an event can include one polyp, all polyps, and / or polyps of a specific size (e.g., 6 mm or larger). The term "disease" and its derivatives can also include syndromes (such as IBS), intestinal diseases, etc. Diseases can be diagnosed by indicators of a certain appearance that may be present or seen in the images. These and other events of interest are considered to be within the scope of this invention.

[0140] In various aspects, the adequacy measure of surgery can be determined based on classical machine learning techniques (such as a classical machine learning classifier 700 using feature metrics as input), deep learning techniques (such as a deep learning classifier 500), or heuristics using feature metrics as input. For example, classical machine learning techniques may include, but are not limited to, SVMs and / or decision trees. For example, deep learning techniques may include CNNs. Heuristics may include a set of rules, such as a series of if-then statements. In various aspects, the adequacy measure of surgery may further include a product of at least two of the following: segmental adequacy measure, global adequacy measure, and / or incomplete surgical feature measure.

[0141] At box 1008, the operation includes displaying the determined adequacy measure. According to various aspects of this disclosure, the adequacy measure may be presented as a value, color, and / or category. Values ​​may be, for example, between 0 and 1. Colors may be, but are not limited to, red / yellow / green. Categories may include, but are not limited to, adequate / insufficient and / or good / poor. In various aspects, the adequacy of a CE procedure may be determined based on an adequacy measure exceeding a predetermined threshold level. The operation may further include providing an indication of whether to exclude a CE procedure, wherein the indication is based on the determined adequacy measure. For example, the operation may display an indication to a clinician to exclude a CE procedure. In other aspects, the procedure may be automatically excluded once it is identified as inadequate. Based on the exclusion indication, the clinician may decide to repeat the CE procedure or have the patient undergo a colonoscopy. For example, the operation may display an adequacy measure (e.g., a value) of 0.25 and an indication of inadequacy of the CE procedure based on such a value (e.g., if below a predetermined threshold). The operation may provide the clinician with reasons for the exclusion of the CE procedure, such as “the transit time of the cecum is too short.” Other examples may include, but are not limited to: "overall cleanliness level is too low", "capsule did not pass through the ascending colon", and "capsule did not pass through the descending colon and there are too few motility frames in the cecum and too few motility frames in the ascending colon".

[0142] In all respects, the procedure can provide insufficient guidance for CE surgery and can exclude all images from the study report except for short clips of multiple images that allow clinicians to determine where the capsule should be performed.

[0143] In various respects, the procedure can rule out an insufficient CE (cerebrospinal fluid) procedure. In various respects, the procedure can reject a exclusion when the CE procedure actually reveals the event or part of the event. For example, in various respects, the procedure can reject an exclusion decision if it is certain that at least one significant polyp is present.

[0144] In various respects, the operation can automatically exclude CE procedures based on a determined adequacy metric. If the adequacy metric falls below a predetermined threshold, the operation or other methods or systems can detect predetermined events in multiple images. For example, polyps or polyps of a predefined minimum size can be detected in images provided via the procedure. An event score can then be received based on this detection. The decision to reject the procedure can be made based on the event score or based on the event score and the adequacy metric. As an example, calculating a probability score for the presence of at least one polyp is solved in co-pending U.S. Patent Application No. 63 / 075,795. The entire contents of that co-pending patent application are hereby incorporated by reference. Other techniques for calculating event probability scores will be understood by those skilled in the art.

[0145] In various aspects, relatively low-quality CE surgeries can be excluded from the research report. For example, sometimes a surgery may still be "adequate" according to adequacy measures, but the quality may be very poor (e.g., CE surgeries with significant occlusion in the images or with a large number of images lost due to connectivity issues). Even if adequacy measures indicate that the surgery is adequate, such surgeries can be excluded.

[0146] Accordingly, the adequacy measures described above are used to indicate whether the imaging coverage provided by the image stream captured by the CE procedure is sufficient to capture the event of interest (regardless of whether such an event of interest is actually present in the patient). As mentioned above, when a 3D view of the patient's GIT or a portion of the GIT cannot be constructed, various characteristic measures (such as those described above) provide an indication of whether the CE procedure is adequate. The following section combines... Figures 11 to 20 Another embodiment for determining the adequacy of CE surgery is described.

[0147] Figure 11 A flowchart of another embodiment of a computer-implemented method 1100 for estimating the adequacy of capsule endoscopy is shown. Those skilled in the art will understand that one or more operations of method 1000 may be performed in a different order, repeated, and / or omitted without departing from the scope of this disclosure. Figure 11 The operation can be performed by a computing device used to analyze medical images captured in vivo via CE surgery (e.g., Figure 2 or Figure 3 The computing system 300 or any other suitable computing system device including a remotely configured computing device or its location is implemented. It should be understood that the operation shown also applies to other systems and their components.

[0148] Initially, at box 1110, the operation includes accessing an image (e.g., a time-series image) of at least a portion of the GIT (e.g., colon 400) captured by the CE device during CE surgery. The image accessed at box 1110 may, for example, be one described above. Figure 10 The image accessed at frame 1002.

[0149] At box 1120, the operation involves accessing one or more feature measures. Various feature measures will be described in more detail later in this paper, including, for example, features indicating how many images capture the same tissue region. Figure 12 ), characteristic measures indicating the cleaning ratio ( Figure 16 ), and / or characteristic measures indicating the number of different views ( Figure 18 In various aspects, characteristic measures may further include demographic information of patients undergoing CE surgery. Demographic information may include, for example, age and / or sex.

[0150] At box 1130, the operation includes determining an adequacy measure of the CE procedure based on one or more of the feature measures. As described above, the adequacy measure indicates whether the imaging coverage provided by the image stream captured by the CE procedure is sufficient to capture the event of interest (regardless of whether any event of interest actually exists). In various ways, the adequacy measure can be determined based on heuristics using the feature measure as input, by classical machine learning techniques (such as classical machine learning classifier 700 using the feature measure as input), and / or by deep learning techniques (such as deep learning classifier 500), as well as other techniques. Examples of determining the adequacy measure will be described in more detail later in this paper.

[0151] At box 1170, the operation can access quality metrics indicating the quality of the CE procedure. Quality metrics may include, for example, the average cleanliness score of all segments of the GIT, patient demographics, the last segment of the GIT reached by the CE device, CE device communication errors, suspected CE device retention in the GIT, or the absolute time spent by the CE device in the GIT segment, as well as other indicators of the quality of the CE procedure and / or captured images. For example, a quality metric could compare the time spent by the CE device in the left colon to the time spent by the CE device in the right colon. Various criteria and / or thresholds can be combined with quality metrics to determine whether the CE procedure and / or captured images meet the quality criteria. Other example quality metrics may include if, according to the GIT segmentation algorithm 1720 (… Figure 17 If no more than a predetermined number of segments are reached (e.g., three segments), exclusion and / or warning will be issued; if not all specific segments of the GIT CE device are reached, exclusion and / or warning will be issued; and / or if the entire GIT transit time is less than a predetermined time period (e.g., approximately ten minutes), exclusion or warning will be issued.

[0152] For example, time in the right colon and / or left colon can be a quality metric and can be measured by combining [the time in the right colon and / or left colon]. Figure 17 The GIT segmentation algorithm described herein is used to determine and identify images captured in the right colon and images captured in the left colon. Timestamps associated with such images can be used to determine the amount of time the CE imaging device spent in the right colon and / or the amount of time spent in the left colon. In various embodiments, a quality metric may not be met if the time spent in the left colon and / or the time spent in the right colon does not meet a specific threshold.

[0153] The average cleanliness score of the GIT can be a quality metric and can be determined, for example, by accessing the cleanliness score of each image as described above and averaging the cleanliness scores of all images. In various embodiments, the quality metric may not be satisfied if the average cleanliness score of the entire GIT does not meet a certain threshold.

[0154] Technical failures can be quality metrics and can include communication gaps as a way to determine if too many images are lost. For example, an operation can compare this percentage of lost images to a predetermined threshold. For instance, an operation can calculate the percentage of lost images in the total image set, and if this percentage is greater than approximately 25%, the quality metric may fail. Other percentages can also be used for quality metrics.

[0155] The suspected retention of the CE imaging device in the GIT can be a quality metric. Operation can determine whether the CE device is suspected of being retained in the GIT based on detected segmental transitions, indications of no visualization of the colon in multiple images, possible visualization of the colon in multiple images, and / or not leaving the body (e.g., where the CE device has not left the patient's body). In all respects, if the CE device is suspected of being retained in the GIT, the quality metric may not be met.

[0156] The quality metrics, thresholds, and conditions described above are exemplary, and other quality metrics, thresholds, or conditions are considered to be within the scope of this invention.

[0157] At box 1150, the operation may include applying a set of sufficiency rules, taking into account the sufficiency metric determined at box 1130, the quality metric accessed at box 1170, and the output of polyp detector 1160. Polyp detector 1160 can process the image accessed at box 1110 and can be operated to identify images containing polyps with high confidence. An example of polyp detector 1160 is described in U.S. Patent Application No. 63 / 075,795, which is hereby incorporated herein by reference in its entirety.

[0158] Continuing with reference box 1150, in various embodiments, the sufficiency rule can be based on combining... Figure 19 and Figure 20 The described rules determine the adequacy of CE procedures. In various embodiments, if no quality metric is met, the adequacy rules can provide an indication of insufficient procedures, which will be combined with... Figure 20 The following description is provided. In various embodiments, if an adequacy metric or quality metric indicates that the procedure is insufficient, but the polyp detector identifies an image of at least one polyp with high confidence, then adequacy rule 1150 may determine that the procedure is insufficient if the polyp detector rejects the inadequacy determination. Such adequacy rules are exemplary, and various variations are considered to be within the scope of the invention. For example, in various embodiments, Figure 11 The operation may exclude the polyp detector 1160, and therefore may not reject sufficiency or quality metrics. In various embodiments, Figure 11 The operation may not involve quality measurement. These and other variations are considered to be within the scope of this invention.

[0159] At box 1140, the operation includes displaying an adequacy determination. If the CE procedure is determined to be inadequate, the operation can provide one or more reasons for the inadequacy. For example, reasons for determining inadequacy may include: unvisualized colon, short transit time, poor cleaning, technical malfunctions (such as communication gaps), unvisualized right and / or left colon, and / or only partially visualized right and / or left colon, and other reasons. If the CE procedure is determined to be inadequate, the operation can show clinicians an indication that the CE procedure is inadequate and will be included in the study report. In other respects, once a procedure is identified as inadequate, it can be automatically excluded. Based on the exclusion indication, clinicians can decide to repeat the CE procedure or have the patient undergo a colonoscopy. The operation can provide clinicians with reasons for the exclusion of the CE procedure. For example, “the transit time of the cecum is too short.” Other examples may include, but are not limited to: “the colon is not visualized (retention),” “the right colon is not visualized,” and “the left colon is not visualized and there is a short transit time and a communication error.”

[0160] Specific examples of characteristic metrics, sufficiency metrics, quality metrics, and sufficiency rules will be described below.

[0161] Figure 12 This is a flowchart of a method 1200 for providing characteristic measurements by identifying groups of images that can be captured when the CE device is static or moving slowly and thus can capture the same tissue area. The operation is based on a progress score to efficiently determine the number of different “views” of the GIT, where each group of images corresponds to a different view of the GIT.

[0162] Initially, at box 1202, the operation designates a new group of images. At box 1204, the operation accesses the next image in an image stream (e.g., time-series images) of at least a portion of the GIT captured by the CE device during CE surgery. At box 1206, the operation accesses a progress score of the image, which indicates the movement of the CE device within the GIT during image capture. As mentioned above, those skilled in the art will recognize techniques used to determine the progress score, such as those described in U.S. Patent No. 8,792,691, which is incorporated herein by reference.

[0163] At box 1208, the operation determines whether the progress score of the image is greater than a predetermined threshold. A lower progress score may indicate less motion or no motion, while a higher progress score may indicate more motion. If the progress score of the image is less than or equal to the predetermined threshold, the image can be considered as capturing the same view / organization region of the GIT and can be included in the group, and the operation returns to box 1204 to access the next image. If the progress score of the image is greater than the predetermined threshold, the image can be considered as capturing a different view / organization region of the GIT, and therefore, the operation can return to box 1202 and can specify the image as the start of a new group / view of the GIT. Figure 12 The operation continues until all images in the image stream captured by the CE procedure have been processed. Figure 12 The operation is exemplary, and other techniques for identifying groups of images that can capture the same view are considered to be within the scope of this invention.

[0164] Figure 13 It shows the result of Figure 12 An example of an image group produced by the operation. Figure 13 A series of images 1300 are shown. The first group of images 1310 includes one or more images (not shown) whose progress scores 1314 are all below a predetermined threshold. Thus, each group of images 1310 corresponds to minimal or no motion within the GIT and can be considered as providing a "view" of the same tissue region. In the example shown, the first group 1310 includes six images 1316, all of which are part of a specific group number 1312 (e.g., group "1"). Each of the images 1316 in the first group 1310 has a progress score (e.g., 1314) less than or equal to the predetermined threshold.

[0165] In the example shown, the seventh image 1310b has a progress score greater than a predetermined threshold, and therefore it is designated as the second group. Figure 12 The operation continues to process images 1300 and groups them based on their progress scores. In the example shown, the twelve images 1300 are divided into seven groups. Therefore, the twelve images 1300 can be viewed as providing seven different views of the Git.

[0166] exist Figure 13In the example, the first group comprises six images, while each of the other groups comprises a single image. According to various aspects of this disclosure, a larger number of images of the same "view" of the GIT increases the probability of identifying an event of interest (e.g., a polyp) in a particular view. Accordingly, the number of images in a group can be a characteristic measure representing the probability of imaging an event of interest (e.g., a polyp). In various embodiments, the number of images in a group can be converted into a probability of imaging an event of interest, and such probability can be a characteristic measure. For example, in various embodiments, a group comprising a single image may have a specific probability of imaging an event of interest (e.g., a 15% probability), while a group comprising six images may have a much higher probability of imaging an event of interest (e.g., a 90% probability), and so on for different numbers of images in a group. The probability values ​​are exemplary, and different probability values ​​are considered to be within the scope of the invention. Figure 12 and Figure 13 The provided feature metrics can be used to generate sufficiency metrics, which are described in more detail below.

[0167] Figure 14 This is a flowchart of a method for providing a characteristic metric referred to herein as the average cleanliness ratio. At box 1402, operational access is via... Figure 12 The operation determines the image group (e.g., Figure 13 Group 1310). At box 1404, the operation accesses the cleaning score of each image in the group. As described above, those skilled in the art will recognize how to determine the cleaning score of an image, including, for example, the techniques described by reference to Klein A, Gizbar M, Bourke M, Ahlenstiel G. “A Validated Computerized Cleansing Score for Video Capsule Endoscopy”, Dig. Endosc. 2015; 28:564–569, and others.

[0168] At box 1406, the operation determines the cleanliness ratio of each image in the group. This will be combined... Figure 15A , Figure 15B and Figure 16 Describe the cleaning ratio. For example, in Figure 13 In the first group of images 1310, each of the six images will have an associated cleanliness ratio. At box 1408, the operation involves determining the average cleanliness ratio of the images in the group. The average cleanliness ratio of each group can be a characteristic metric.

[0169] Figure 15AIt is a histogram that includes events of interest (e.g., polyps) and the number of images in each matching image stream for various cleanliness scores, and Figure 15B It is a histogram of the number of images in the entire image stream that match each of the various cleanliness scores. According to various aspects of this disclosure, Figure 15A and Figure 15B The histograms are normalized to have the same Y-axis range. In various embodiments, the Y-axis range can be a probability range of [0,1], such that... Figure 15A and Figure 15B It can be viewed as a probability distribution. For the sake of generalization, Figure 15A The normalized histogram will be called the "histogram of events of interest" and Figure 15B The normalized histogram will be called the "full frame histogram". Figure 15A and Figure 15B The Y-axis value of each part of the normalized histogram will be referred to as the "normalized height". As used in this article, the cleanliness ratio is the following ratio: (normalized height of cleanliness score in the histogram of events of interest) / (normalized height of cleanliness score in the histogram of all frames).

[0170] Figure 16 This is a graph of the cleaning ratio 1602 across the cleaning score, where the cleaning ratio 1602 is represented by circles. In various embodiments, regression analysis can be used to fit the curve 1604 to the plotted cleaning ratio 1602 to map the cleaning score to the cleaning ratio. In the example shown, the fitted curve 1604 is a cubic polynomial. However, the fitted curve can be any polynomial of any degree.

[0171] As per various aspects of this disclosure, the term "cleanliness ratio" may refer to a plotted cleanliness ratio 1602 or a fitted cleanliness ratio curve 1604. See also: Figure 14 Access the cleanliness score of each image in the group, and determine the cleanliness ratio of each image in the group based on that cleanliness score (e.g., Figure 16 (e.g., 1602, 1604). As mentioned above, the average cleanliness ratio of a group can be a characteristic metric.

[0172] Figure 15A , Figure 15B and Figure 16The embodiments shown are exemplary, and various variations are considered to be within the scope of the invention. For example, in various embodiments, separate histograms and cleanliness ratio curves can be created for different portions of the GIT. For example, with respect to the colon, different segments may have different behaviors regarding cleanliness. Typically, most images in research reports come from the cecum because the CE imaging device spends most of its time in the cecum during routine CE surgery. Separate histograms and cleanliness ratio curves / fit curves can be created for different colonic segments, such as separate histograms and cleanliness ratio curves / fit curves for the cecum, right colon or ascending colon, transverse colon, left colon or descending colon, and rectum. These and other variations are considered to be within the scope of the invention.

[0173] Accordingly, involving Figures 12 to 16 The foregoing description provides various characteristic measures, including the probability of imaging an event of interest (e.g., a polyp) for each image group / viewpoint and the average cleanliness ratio for each imaging group / viewpoint. According to various aspects of this disclosure, for each image group / viewpoint, another characteristic measure can be determined as: (probability of imaging an event of interest in the group) × (average cleanliness ratio of the group), and such a measure may be referred to herein as the “group score”.

[0174] In all aspects of this disclosure, the adequacy of CE procedures can be measured through... Figure 12 The sum of all group scores for the image group identified by the operation. A higher sum of group scores may indicate the presence of more multi-frame views with acceptable cleanliness of the GIT, while a lower sum of group scores may indicate the presence of fewer multi-frame views with GIT and / or suboptimal cleanliness. In various embodiments, the sum of group scores may be mapped to, as in Figure 18 The example shows the probability, and this probability can be used as a sufficiency measure. Figure 18 The mapping shown is exemplary. In various embodiments, Figure 18 The mapping can be determined empirically from the training data and / or validation data, can be fitted to the data and / or extrapolated from the data, or can be based on the expectation that the mapping is arbitrary.

[0175] In all aspects, Figure 18 The mapping shown can be provided based on receiver operating characteristic (ROC) curves. As those skilled in the art will recognize, an ROC curve is a graph that illustrates the performance of a classification model at various classification thresholds. To generate... Figure 18For the purpose of mapping, the classification model is configured to classify the sum of scores for each group into one of two categories: a "positive" category, where the imaging coverage provided by the image is sufficient to capture the event of interest (regardless of whether such an event of interest actually exists), and a "negative" category, where the imaging coverage provided by the image is insufficient to capture the event of interest (regardless of whether such an event of interest actually exists). When a specific threshold is used to perform the classification, the classification model will have a specific True Positive Rate (TPR) and a specific False Positive Rate (FPR). Different thresholds will produce different TPRs and FPRs, and in various embodiments, different thresholds can span the entire range of possible values ​​for the sum of scores for each group. As those skilled in the art will understand, ROC curves are generated by plotting these FPR vs. TPR pairs for different thresholds in a biaxial coordinate space, where the x-axis represents the False Positive Rate (FPR) and the y-axis represents the True Positive Rate (TPR); and then performing interpolation between the plotted coordinates or fitting a curve to the plotted coordinates. The ROC curve can be a fitted curve, or the ROC curve can be the plotted coordinates and the interpolation between the plotted coordinates, or some combination of both.

[0176] Based on various aspects of this disclosure, for classification models that categorize group score sums as sufficient or insufficient, ROC curves can be used to generate... Figure 18 Mapping. As mentioned above, the ROC curve is created using different thresholds that can span a range of possible values ​​for the sum of group scores. Therefore, each threshold can be viewed in some way as representative of the sum of scores for a particular group, and the true certainty corresponding to a threshold can be considered as the probability that the imaging coverage provided by the image is sufficient to capture the event of interest (regardless of whether such an event of interest actually exists). Accordingly, Figure 18 The mapping can be used to map the sum of group scores to probabilities that can be used as a sufficiency measure.

[0177] Figure 18 Mapping and Combination Figure 18 The described embodiments are exemplary. Other embodiments are considered to be within the scope of the invention. For example, as described below... Figure 17 As described, the sum of individual group scores can be calculated for different segments of the GIT, and each segment of the GIT can have a similar... Figure 18 The corresponding mapping is shown in the figure.

[0178] Figure 17 This document describes an implementation for determining adequacy measures when image sets span different GIT regions, such as the colon (e.g., cecum, right colon or ascending colon, transverse colon, left colon or descending colon and rectum). For convenience, reference can be made to the colon section in the following paragraphs. However, it is anticipated that the following description will also apply to other GIT regions.

[0179] Figure 17 This is a flowchart of a method for providing sufficiency measures when multiple GIT segments exist. Figure 17 The operation can be performed by, for example Figure 2 and Figure 3 The computing system 300 and other computing systems are executed. At box 1702, operation access is via… Figure 12 The operation determines the image groups. At box 1704, the operation associates each image group with a GIT segment based on input from a GIT segmentation algorithm 1720, which divides the image stream into portions corresponding to the GITs in which the images are captured. Typically, the GIT segmentation algorithm 1720 can be performed, for example, based on the recognition of various markers or transition indicators between different anatomical segments. For example, such recognition can be performed based on machine learning techniques. One method of segmenting an image stream into segments corresponding to anatomical segments is described in U.S. Patent Application No. 17 / 244,988, which is hereby incorporated herein by reference in its entirety.

[0180] At box 1706, the operation determines a segment score for each GIT segment (e.g., cecum, ascending colon, etc.). For example, the segment score for each GIT segment can be the sum of the group scores described above, where image groups that are only a portion of the GIT segment are used for the sum of group scores.

[0181] At box 1708, the operation converts the score of each segment into a mapped probability corresponding to the sum of the group scores, which is combined with the above... Figure 18 Describe it. Each section of Git can have like Figure 18 Individual mappings, such as the one shown, can be generated using ROC curves for each segment. In such embodiments, each GIT segment will have a corresponding probability, which can be interpreted as the probability that the imaging coverage of the GIT segment provided by the image corresponding to the GIT segment is sufficient to capture the event of interest (e.g., at least one polyp or a significant polyp) in the GIT segment, regardless of whether such an event of interest is actually present in the patient. For example, the probability for a colonic segment (cecum, right colon or ascending colon, transverse colon, left colon or descending colon and rectum) could be [P1, ..., P5], and such probabilities could be the result of box 1708.

[0182] At box 1710, the operation involves determining the adequacy measure of CE surgery as a weighted sum of the probabilities of the GIT segments. For example, if the probabilities of the colonic segments (cecum, right colon or ascending colon, transverse colon, left colon or descending colon and rectum) are [P1, ..., P5], then the weighted sum would be... In various embodiments, the weights [w1, ..., w5] can have values ​​determined based on the prior probability that the event of interest is located in each segment. In various embodiments, the prior probability can be determined empirically by compiling known instances of the event of interest (e.g., polyps) in the patient population and considering the presence of these instances in the GIT within the patient population. The percentage of all instances occurring in each segment of the GIT can be determined, and such a percentage can be used as the prior probability of the event of interest occurring in a segment of the GIT. Using a numerical example of the colon, assume the following values ​​are determined:

[0183] appendix Ascending colon transverse colon descending colon rectum a priori 0.08 0.22 0.16 0.38 0.16 sufficiency 0.9 0.8 0.7 1.0 0.0

[0184] The adequacy measure of CE surgery can be calculated as a weighted sum:

[0185] (0.9 * 0.08) + (0.8 * 0.22) + (0.7 * 0.16) + (1.0 * 0.38) + (0.0 * 0.16) = 0.74

[0186] The specific values ​​in the above examples are illustrative, and other values ​​are considered to be within the scope of this invention.

[0187] In various embodiments, prior probabilities can be used to calculate individual adequacy measures for individual portions of the GIT. Continuing with the colon as an example, individual adequacy measures for the left side of the colon (e.g., descending sigmoid colon and rectum) and the right side of the colon (e.g., cecum, ascending colon, and transverse colon) can be calculated. According to various aspects of this disclosure, the prior probability for the left side of the colon can be renormalized to 1, such that 0.38 for the descending sigmoid colon and 0.16 for the rectum become approximately 0.7 for the descending sigmoid colon and 0.3 for the rectum. The adequacy measure for the left side of the colon can be calculated as (1.0 * 0.7) + (0.0 * 0.16) = 0.7. Similarly, the prior probability for the right side of the colon can be renormalized to 1, such that 0.08 for the cecum, 0.22 for the ascending colon, and 0.16 for the transverse colon become approximately 0.17 for the cecum, 0.48 for the ascending colon, and 0.35 for the transverse colon. The adequacy measure of the right side of the colon can be calculated as (0.9 * 0.17) + (0.8 * 0.48) + (0.7 * 0.35) = 0.782. The colon is used merely as an example, and the disclosed technique can be applied to other parts of the GIT to determine the adequacy measure of different parts of the GIT using prior probabilities. In the event that CE procedure is determined to be inadequate, the adequacy measure of different parts of the GIT can be used to explain which part of the GIT might have led to the inadequacy of CE procedure.

[0188] Accordingly, the description above provides examples of various characteristic measures and various ways of calculating sufficiency measures based on such characteristic measures. Figure 19 and Figure 20 The diagram illustrates the sufficiency rules that can be applied based on the outputs of sufficiency measure 1130, quality measure 1170, and polyp detector 1160. Figure 11 Example of 1150). For convenience, Figure 19 and Figure 20 The implementation examples refer to the sufficiency measure as the sufficiency probability, which can be derived from... Figure 17 The probability of the output of box 1710 may be determined by... Figure 18 The probability of mapping and other probabilities. Based on various aspects of this disclosure, it can be applied when all quality metrics are satisfied. Figure 19 The chart can be applied whenever any quality metric is not met. Figure 20 The chart.

[0189] refer to Figure 19 The chart shown depicts the situation when all quality metrics (e.g., ...) are met. Figure 11 When 1170) is used, the sufficiency rule is used to classify CE surgery as sufficient, insufficient, or insufficient but rejected polyp probability values ​​(e.g., Figure 11 (1160) and sufficiency probability values ​​(e.g., Figure 11The combination of 1130). Each "O" 1910 is a graph of the adequacy probability of CE procedure and the polyp probability, wherein at least one polyp is visualized by CE procedure. Each "x" 1912 is a graph of the adequacy probability of CE procedure and the polyp probability, wherein at least one polyp is not visualized by CE procedure. In the example shown, if the adequacy probability is less than or equal to 0.2 (region 1904) or if the adequacy probability is less than or equal to 0.4 and the polyp probability is less than or equal to 0.01 (region 1908), the operation indicates that the CE procedure is inadequate. If the adequacy probability is in the range of 0.4 to 1.0 (region 1902), the operation indicates that the CE procedure is adequate. If the adequacy measure is in the range of 0.2 to 0.4 and the polyp probability is greater than 0.01 (region 1906), the operation indicates that the CE procedure is inadequate but rejected, which indicates that the CE procedure is inadequate based on the adequacy probability, but the inadequacy determination is rejected based on the polyp probability. Accordingly, the procedure can overturn a decision to exclude a CE procedure if the presence of at least one significant polyp is certain based on the polyp probability. As described above, the procedure can further demonstrate to the clinician the underlying reasons for the rejection of an inadequate outcome. As indicated by the “O” and “X” markings in the diagram, some decisions classifying CE procedures may not be consistent with decisions actually made during CE procedures, such as certain markings in regions 1906 and 1902, but most classifications are correct. Since manually reviewing all images of CE procedures to identify events of interest would be impractical, adequacy determination can increase the physician’s confidence in the outcome of the CE procedure.

[0190] Figure 19 The regions 1902 to 1908 and values ​​shown are exemplary, and various variations are considered to be within the scope of the invention. For example, each region may be defined by a lower and upper threshold for sufficiency probability and / or a lower and upper threshold for polyp probability. Such lower and upper thresholds may have the same characteristics as... Figure 19 The values ​​shown are different from the values ​​in the diagram. These and other variations are considered to be within the scope of this invention.

[0191] Figure 20 It demonstrates what happens when no quality metric is met (e.g., Figure 11 When 1170) is used, the sufficiency rule is used to classify CE surgery as sufficient, insufficient, or insufficient but rejected polyp probability values ​​(e.g., Figure 11 (1160) and sufficiency probability values ​​(e.g., Figure 11The chart shows a combination of 1130. In the example shown, if the sufficiency probability is less than or equal to 0.2 (Region 2004) or if the polyp probability is less than or equal to 0.01 (Region 2008), the operation indicates that the CE procedure is insufficient. If the sufficiency measure is in the range of 0.2 to 1 and the polyp probability is greater than 0.01 (Region 2006), the operation indicates that the CE procedure is insufficient but rejected. This indicates that the CE procedure is insufficient based on the sufficiency probability, but the insufficiency determination is rejected based on the polyp probability.

[0192] Figure 20 The regions 2004 to 2008 and values ​​shown are exemplary, and various variations are considered to be within the scope of the invention. For example, each region may be defined by a lower and upper threshold for sufficiency probability and / or a lower and upper threshold for polyp probability. Such lower and upper thresholds may have the same characteristics as... Figure 20 The values ​​shown are different from the values ​​in the diagram. These and other variations are considered to be within the scope of this invention.

[0193] In various embodiments, not all quality metrics (e.g., Figure 19 When there is a set of sufficiency rules, and when no quality metric is satisfied (e.g., Figure 20 Instead of having only one set of sufficiency rules, more than two sets of sufficiency rules can be used. For example, different sufficiency rules can be used if a specific quality metric is not met. These and other variations are considered to be within the scope of this invention.

[0194] Although examples of images captured in vivo by a CE device have been shown and described, the disclosed techniques can be applied to images captured by other devices or institutions.

[0195] The embodiments disclosed herein are examples of this disclosure and may be embodied in various forms. For example, although some embodiments herein are described as separate embodiments, each embodiment herein may be combined with one or more of the other embodiments herein. The specific structural and functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as a representative basis for teaching those skilled in the art to adopt this disclosure in various ways in virtually any properly detailed structure. Throughout the description of the accompanying drawings, similar reference numerals may refer to similar or identical elements.

[0196] The phrases “in one embodiment,” “in an embodiment,” “in various embodiments,” “in some embodiments,” or “in other embodiments” may refer to one or more of the same or different embodiments according to this disclosure, respectively. The phrase “A or B” means “(A), (B), or (A and B).” The phrase “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).”

[0197] Any operation, method, program, algorithm, or code described herein can be transformed into or expressed as a programming language or computer program embodied on a computer or machine-readable medium. As used herein, the terms "programming language" and "computer program" each include any language used to specify instructions to a computer, and include (but are not limited to) the following languages ​​and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, Python, scripting languages, Visual Basic, meta-languages ​​that specify the program itself, and all first-, second-, third-, fourth-, fifth-, or next-generation computer languages. Databases and other data schemas, and any other meta-languages, are also included. No distinction is made between interpreted, compiled, or both interpreted and compiled languages. No distinction is made between compiled and source versions of a program. Therefore, references to programs (where programming languages ​​may exist in multiple states, such as source, compiled, object, or linked) are references to any and all of these states. References to the procedure may encompass the actual instructions and / or the intent of those instructions.

[0198] It should be understood that the foregoing description is for illustrative purposes only. To the extent consistent with this disclosure, any or all aspects detailed herein may be used in conjunction with any or all other aspects detailed herein. Various alternatives and modifications can be devised by those skilled in the art without departing from this disclosure. Therefore, this disclosure is intended to cover all such alternatives, modifications, and variations. The embodiments described with reference to the accompanying drawings are presented only to illustrate certain examples of this disclosure. Other elements, steps, methods, and techniques that are not substantially different from those described in the foregoing and / or appended claims are also intended to fall within the scope of this disclosure.

[0199] While several embodiments of this disclosure are shown in the accompanying drawings, they are not intended to limit the disclosure thereto, as the aim is to make the scope of the disclosure as broad as permitted by the art, and this specification should be read in the same manner. Therefore, the foregoing description should not be construed as restrictive, but rather as merely exemplary of particular embodiments. Other modifications within the scope and spirit of the appended claims will be contemplated by those skilled in the art.

Claims

1. A computer-implemented method for estimating the adequacy of capsule endoscopy (CE) surgery, the method comprising: Access multiple images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging device during CE surgery; Access multiple feature measures associated with the multiple images; The adequacy measure of the CE procedure is determined based on the plurality of characteristic measures, the adequacy measure indicating whether the imaging coverage provided by the plurality of images is sufficient to capture events of interest in at least the portion of the GIT, regardless of whether such events of interest actually exist in at least the portion of the GIT; as well as The adequacy indication of the CE procedure is displayed based on the aforementioned adequacy metric. The adequacy metric for determining the CE procedure is based on at least one of classical machine learning techniques, deep learning techniques, or heuristics. Process the multiple images to identify multiple image groups. In each of the plurality of image groups, each image in the corresponding image group captures the same tissue region. For each of the plurality of image groups, the feature measure among the plurality of feature measures includes the number of images in the corresponding image group. The adequacy measure of the CE procedure is determined based on the number of images in each of the plurality of image groups.

2. The computer-implemented method as described in claim 1, wherein, For each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the average cleanliness ratio of the corresponding image group. The adequacy measure of the CE procedure is determined based on the average cleanliness ratio of each of the plurality of image groups.

3. The computer-implemented method of claim 2, further comprising determining the average cleanliness ratio of each image group by: Access the mapping between cleaning scores and cleaning ratios; and For each of the plurality of image groups: Access the cleanliness score of each image in the corresponding image group. The cleaning ratio of each image in the corresponding image group is determined based on the mapping between cleaning score and cleaning ratio. The average cleanliness ratio of the corresponding image group is determined to be the average cleanliness ratio of the images in the corresponding image group.

4. The computer-implemented method of claim 1, further comprising: The sufficiency metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture events of interest in at least said portions of the GIT, regardless of whether such events of interest actually exist in at least said portions of the GIT. The adequacy indication of the CE procedure includes at least one reason why the CE procedure is determined to be insufficient.

5. The computer-implemented method as described in claim 1, wherein, At least the portion of the GIT includes multiple segments. The adequacy measures for determining the CE procedure include: Determine the sufficiency measure for each of the plurality of segments, and The adequacy measure of the CE procedure is determined based on the adequacy measure of each of the plurality of segments.

6. The computer-implemented method as described in claim 5, wherein, Determining the adequacy measure of the CE procedure based on the adequacy measure of each of the plurality of segments includes: Access the prior probability of the event of interest occurring in each of the plurality of segments, the prior probability being determined empirically based on a patient population; and The adequacy measure of the CE procedure is determined based on the prior probability and the adequacy measure of each of the plurality of segments.

7. The computer-implemented method of claim 1, further comprising: Access at least one quality metric associated with the plurality of images; The adequacy indication is determined based on a first set of adequacy rules when at least one quality metric is met; as well as When any of the at least one quality metric is not satisfied, the sufficiency metric is determined based on a second set of sufficiency rules.

8. A system for estimating the adequacy of capsule endoscopy (CE) surgery, the system comprising: Display device; At least one processor; as well as At least one memory, the at least one memory including instructions stored thereon, the instructions causing the system to perform the following operations when executed by the at least one processor: Access multiple images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging device during CE surgery; Access multiple feature measures associated with the multiple images; The adequacy measure of the CE procedure is determined based on the plurality of characteristic measures, the adequacy measure indicating whether the imaging coverage provided by the plurality of images is sufficient to capture events of interest in at least the portion of the GIT, regardless of whether such events of interest actually exist in at least the portion of the GIT; as well as The adequacy indication of the CE procedure is displayed on the display device based on the adequacy metric. The adequacy metric for determining the CE procedure is based on at least one of classical machine learning techniques, deep learning techniques, or heuristics. When executed by the at least one processor, the instructions further cause the system to process the plurality of images to identify multiple image groups. In each of the plurality of image groups, each image in the corresponding image group captures the same tissue region. For each of the plurality of image groups, the feature measure among the plurality of feature measures includes the number of images in the corresponding image group. The adequacy measure of the CE procedure is determined based on the number of images in each of the plurality of image groups.

9. The system of claim 8, wherein, For each of the plurality of image groups, the characteristic measure among the plurality of characteristic measures includes the average cleanliness ratio of the corresponding image group. The adequacy measure of the CE procedure is determined based on the average cleanliness ratio of each of the plurality of image groups.

10. The system of claim 9, wherein, When executed by the at least one processor, the instructions further cause the system to determine the average cleanliness ratio for each image group by: Access the mapping between cleaning score and cleaning ratio; as well as For each of the plurality of image groups: Access the cleanliness score of each image in the corresponding image group. The cleaning ratio of each image in the corresponding image group is determined based on the mapping between cleaning score and cleaning ratio. The average cleanliness ratio of the corresponding image group is determined to be the average cleanliness ratio of the images in the corresponding image group.

11. The system of claim 8, wherein, At least the portion of the GIT includes multiple segments. The adequacy measures for determining the CE procedure include: Determine the sufficiency measure for each of the plurality of segments, and The adequacy measure of the CE procedure is determined based on the adequacy measure of each of the plurality of segments.

12. The system of claim 11, wherein, When determining the adequacy measure of the CE procedure based on the adequacy measure of each of the plurality of segments, the instructions, when executed by the at least one processor, cause the system to perform the following operations: Access the prior probability of the occurrence of the event of interest in each of the plurality of segments, the prior probability being determined empirically by the patient population; as well as The adequacy measure of the CE procedure is determined based on the prior probability and the adequacy measure of each of the plurality of segments.

13. The system of claim 8, wherein, When executed by the at least one processor, the instruction further causes the system to perform the following operations: Access at least one quality metric associated with the plurality of images; The adequacy indication is determined based on a first set of adequacy rules when at least one quality metric is met; as well as When any of the at least one quality metric is not satisfied, the sufficiency metric is determined based on a second set of sufficiency rules.

14. The system of claim 8, wherein, When executed by the at least one processor, the instruction further causes the system to perform the following operations: The sufficiency metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture events of interest in at least said portions of the GIT, regardless of whether such events of interest actually exist in at least said portions of the GIT. The adequacy indication of the CE procedure includes at least one reason why the CE procedure is determined to be insufficient.

15. The system of claim 8, wherein, The event of interest is a significant polyp, wherein the instruction, when executed by the at least one processor, further causes the system to perform the following operations: Determining that the sufficiency metric indicates that the imaging coverage provided by the plurality of images is insufficient to capture events of interest in at least said portions of the GIT, regardless of whether such events of interest actually exist in at least said portions of the GIT; and A polyp detector, which processes the plurality of images, detects significant polyps in the plurality of images. The adequacy indication of the CE procedure includes an indication that the CE procedure is determined to be inadequate, but the determination is rejected by the polyp detector.

16. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause to perform a method comprising the following operations: Access multiple images of at least a portion of the gastrointestinal tract (GIT) captured by a CE imaging device during CE surgery; Access multiple feature measures associated with the multiple images; The adequacy measure of the CE procedure is determined based on the plurality of characteristic measures, the adequacy measure indicating whether the imaging coverage provided by the plurality of images is sufficient to capture events of interest in at least the portion of the GIT, regardless of whether such events of interest actually exist in at least the portion of the GIT; as well as The adequacy indication of the CE procedure is displayed based on the aforementioned adequacy metric. The adequacy metric for determining the CE procedure is based on at least one of classical machine learning techniques, deep learning techniques, or heuristics. Wherein, when the instruction is executed by the processor, it causes a method to be further performed including the following operations: Access at least one quality metric associated with the plurality of images; The sufficiency indication is determined based on a first set of sufficiency rules when at least one quality metric is met; and When any of the at least one quality metric is not satisfied, the sufficiency metric is determined based on a second set of sufficiency rules.

Images