System and method for generating and displaying studies of in vivo image streams

Abstract

The present disclosure relates to systems and methods for generating and displaying studies of in vivo image streams. Systems and methods are disclosed that may display and / or provide analysis of a plurality of selected images of the gastrointestinal tract of a patient collected in vivo by swallowing a capsule. The image may be displayed for review (e.g., as a study) and / or further analysis by a user. A subset of images representing the image stream and automatically selected according to a first selection method may be displayed. Upon user input, an additional image corresponding to the currently displayed image may be displayed, where the additional image is automatically selected according to a second selection method. The second selection method may be based on a relationship between an image in the in vivo image stream and the currently displayed image.

Description

[0001] This application is a divisional application of an application with a filing date of October 19, 2019, application number 201980075047.5, and invention name “System and method for generating and displaying in vivo image flow research”. Technical Field

[0002] The present disclosure relates to methods, systems and computer program products for displaying and / or analyzing and / or reporting medical images from a series or stream of images captured in vivo. In particular, the present disclosure relates to methods, systems and computer program products for generating and displaying studies of in vivo images captured via a capsule endoscopy (CE) procedure. Background Art

[0003] The capsule system may include a swallowable capsule that captures images of the gastrointestinal tract ("GI" tract or "GIT"). The images may be stored on the capsule and / or transmitted to a receiving device (typically including an antenna). The receiving device receives and may store (e.g., in a storage device in the receiving device) the images. The swallowable capsule may include one or more cameras or imaging devices, a power source, a processor, and a transmitter.

[0004] The capsule, receiving device, or removable storage device of the receiving device may be coupled to a computing device (such as a server). The computing device may process the received images and may provide an interface to display the images, typically as a video or movie (e.g., a series of moving images). This processing may also be performed "locally" on a workstation operated by a medical professional after receiving the images transmitted from a receiver or recorder. In some embodiments, a physical receiving device located on the patient may transfer or download the images to a computer for processing and viewing. The workstation for processing or viewing images may be a local computer, tablet computer, or workstation.

[0005] A healthcare professional (such as a physician) can review the images as they are displayed, for example, on a workstation, terminal, or remotely, for example via a web browser, where the healthcare professional typically views the images as a movie or image stream (which typically includes fewer images than all of the captured images due to, for example, pre-processing).

[0006] It is desirable to present the thousands of images collected by the capsule in a manner that allows a healthcare professional to accurately review the images and discover important images or features in a reasonable amount of time.

[0007] A typical prior art view of a procedure result or procedure study as seen by a physician includes thousands of images viewed as moving images or videos (e.g., each image stream captured by a single camera is displayed one image at a time in sequence). Viewing thousands of images takes valuable time and can be a cumbersome task. Viewing thousands of images as a video can also result in a higher probability of missing an image of interest. It is desirable to reduce the time spent by a medical professional viewing images. It is also desirable to make the viewing task more efficient and more beneficial. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like reference numerals indicate corresponding, similar or analogous elements, and in which:

[0009] Figure 1 is a high-level block diagram of an exemplary computing device that may be used with embodiments of the present invention.

[0010] Figure 2 A first or default screen is shown that is initially presented to a viewer or user according to some embodiments.

[0011] Figure 3 Example screens are shown according to some embodiments.

[0012] Figure 4 An exemplary screen or display showing additional information (eg, studies) is shown according to some embodiments.

[0013] Figure 5 shows the image after the zoom operation according to some embodiments Figure 4 An example screen of .

[0014] Fig. 6A A schematic diagram showing a research view mode according to one embodiment of the present invention.

[0015] Figure 6B According to an implementation scheme Fig. 6A Schematic diagram of a map (e.g., bar) in the default view mode of .

[0016] Figure 7 According to some embodiments, Fig. 6A An exemplary illustration of a display or view of second level information for a selected image is shown.

[0017] Fig. 8A A default research view according to one embodiment is depicted.

[0018] Figure 8B Depicted according to some embodiments Fig. 8A Matrix layout of the image shown.

[0019] Fig. 9A is an exemplary screen shot of a default view mode according to one embodiment of the present disclosure.

[0020] Fig. 9B is Fig. 9A An exemplary screenshot of a display or view of second level information for a selected image in the display.

[0021] Fig.10 A sample display according to one embodiment of the present invention is depicted.

[0022] Fig.11 is a flow chart of a method according to an embodiment of the present invention.

[0023] It should be understood that, for simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn accurately or to scale. For example, the size of some elements may be exaggerated or more prominent relative to other elements for clarity, or several physical components may be included in one functional block or element. In addition, reference numerals may be repeated between the drawings to indicate corresponding or similar elements where deemed appropriate. Summary of the invention

[0024] Embodiments of the present invention may include a capsule device and an optional receiving device. The capsule device may be a swallowable capsule that captures images while traversing the gastrointestinal tract and transmits the images to a receiving device (which typically includes an antenna). The receiving device may receive these images and store them in a storage device, such as in the receiving device. The receiving device may transmit these images to a server or another remote device based on the "cloud" in real time (e.g., during the procedure). The receiving device may be, for example, a belt or patch or any other item worn on the patient. The receiving device may forward the received images to a remote device for display, including real-time display, storage, processing and / or analysis. The receiving device may, for example, transmit these images to a remote device via a patient's cell phone. A local or remote (e.g., "cloud") server or any other suitable computing device may process and analyze these images and other received procedure-related data (e.g., image capture time and patient data) to generate a study of a CE procedure. The computing device may provide an interface locally or remotely to display the study for review by a user (e.g., a medical professional). Other methods of storing, receiving, processing and displaying images may be used.

[0025] Embodiments of the present invention include systems, methods, and computer program products for displaying and / or providing analysis of a plurality of selected images for review (e.g., study) and / or further analysis by a viewer or user. The images are selected from a stream of images captured in vivo by a capsule endoscope traversing a portion of the gastrointestinal tract (such as the small intestine and / or colon). The disclosed methods, systems, tools, and / or computer program products may be particularly useful for diagnosis, treatment decisions, monitoring, and / or screening of various lesions, disorders, abnormalities, and diseases of the GIT.

[0026] The images may be displayed for user review (e.g., as a study) and / or further analysis. A subset of images representing the image stream and automatically selected according to a first selection method may be displayed. Upon user input, additional images corresponding to the currently displayed images may be displayed, wherein the additional images are automatically selected according to a second selection method. The second selection method may be based on a relationship between images in the in vivo image stream and the currently displayed image. DETAILED DESCRIPTION

[0027] In the following specific embodiments, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention can be practiced without these specific details. In other cases, well-known methods, procedures and parts are not described in detail to avoid blurring the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, the discussion of the same or similar features or elements may not be repeated.

[0028] Although embodiments of the present invention are not limited in this respect, discussions using terms such as, for example, "processing," "computing," "determining," "establishing," "analyzing," "examining," and the like may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or convert data represented as physical (e.g., electronic) quantities within a computer's registers and / or memory into other data similarly represented as physical quantities within a computer's registers and / or memory or other information non-transitory storage medium that can store instructions for performing the operations and / or processes. Although embodiments of the present invention are not limited in this respect, the terms "plurality" and "multiple" as used herein may include, for example, "multiple" or "two or more". The terms "plurality" or "multiple" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, and the like. The term set may include one or more when used herein. Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. In addition, some of the method embodiments or elements thereof may occur or be executed simultaneously, at the same point in time, or in parallel.

[0029] The term "position" and its derivatives as referred to herein with respect to an image may refer to the estimated position of a capsule along the GIT when the image was captured or the estimated position of a portion of the GIT shown along the GIT in the image.

[0030] A type of capsule endoscopy procedure may be determined based on, among other things, the portion of the GIT that is of interest and to be imaged (e.g., colon or small bowel ("SB")) or based on a specific use (e.g., for checking the status of a GI disease such as Crohn's disease, or for colon cancer screening).

[0031] The terms screen, view, and display may be used interchangeably herein and should be understood depending on the specific context.

[0032] The terms "default view," "default display," and "default screen" as referred to herein may refer, respectively, to the "view," "display," and "screen" that is first displayed at each initiation of a review of a program study in accordance with the systems and methods of the present invention, and wherein this first view, display, or screen is predefined and not the result of a user selection.

[0033] The terms "default view" and "default display" as referred to herein with respect to a view or display of particular information may refer, respectively, to a first "view" and "display" of particular information according to the systems and methods of the present invention at each initiation of a review of a program study, and wherein this first view or display of the particular information is predefined and not the result of a user selection.

[0034] The terms "by default" or "by default" as referred to herein with respect to the display or view of particular information may relate to information displayed in the frame of a particular view, display or screen without user action or input to upload, initiate or trigger that display.

[0035] Unless explicitly stated otherwise, the terms "around" or "adjacent" as referred to herein with respect to images (e.g., images around or adjacent to another image) may relate to spatial and / or temporal characteristics. For example, images around or adjacent to other images may be images whose estimated locations are adjacent to other images along the GIT and / or images captured near the capture time of another image, within a certain threshold, such as within one or two centimeters, or within one, five, or ten seconds.

[0036] Depending on the context, the terms "GIT" and "a portion of the GIT" may refer to each other or include each other. Thus, the term "a portion of the GIT" may also refer to the entire GIT, and the term "GIT" may also refer to only a portion of the GIT.

[0037] Embodiments of the present invention may differ from prior art CE protocol viewing applications in that prior art applications typically present a movie or image stream (e.g., a series of images presented in time sequence) in a first or default view or display. In embodiments of the present invention, a set of images collected from the GIT via, for example, a swallowable capsule may be processed and then initially presented as a set of static images, for example, in a default or first view, once the images are uploaded or processed after capsule capture and at each initiation of a review of a protocol of an embodiment of the system and method. Each image presented in the default view may have a feature of interest (e.g., a particular instance of a lesion) and may be associated or correspond to other images of the same feature (e.g., the same instance of a lesion) that are not shown in the default view. Indicating or selecting (e.g., clicking with an input device) a displayed single static image to represent an instance of a feature of interest may present a secondary or additional information view, which may be, for example, a moving image or scrollable series of images that also shows the instance of the feature.

[0038] Images of different parts of the GIT (e.g., SB, large intestine or colon, esophagus, stomach, etc.) can be presented, and the user interface can vary depending on the different parts displayed. The type of procedure performed, such as an SB procedure or a colon procedure or both (e.g., a procedure intended to specifically display or examine the SB or a procedure intended to specifically display or examine the colon or both) can determine which part of the GIT is the part of interest. Typically, most of the images displayed have a part of interest. Each part of interest can be divided into multiple segments. Each segment (e.g., an image showing a portion of the GIT estimated to be located in the segment) can be displayed in a different view, for example, via different tabs. For example, if the part of interest is the colon (e.g., the procedure is a colon procedure), it can be segmented into five anatomical segments (cecum, ascending colon, transverse colon, descending colon, and rectum), each with a separate view accessible via a tab. Thus, the user may be presented with tabs labeled Cecum, Ascending Colon, Transverse Colon, Descending Colon, and Rectum, and clicking on Cecum may show a first level group that best represents images in the received image stream estimated to be located in the cecum.

[0039] The second level image may be displayed on the same screen as the first level image (e.g., the first level image may be transformed to be displayed as a moving image or a scrollable image to display its corresponding second level image, and other first level static images remain on the display), or displayed in a different window or pop-up window. Depending on the image capture time, the second level image may precede the corresponding first level image and / or follow the corresponding first level image or be after the corresponding first level image. The second level image may include the corresponding first level image (seed image) or the display of the second level image may include the corresponding first level image. Therefore, the second level image including or in addition to the corresponding first level image may be displayed depending on its capture time. The second level images may then be numbered so that the first level image (e.g., the "best" representative image) is numbered zero, and the second level image or other second level images are numbered according to their distance from image zero in the image.

[0040] The user may click a checkbox or otherwise select an image displayed in the first level to have the image placed in a report or final report (typically displayed or delivered after the user reviews the images); the user may filter the images in the first level so that only "selected" images are displayed. Various tools or image analysis capabilities may be presented, such as automatic lesion size estimation and lesion size estimation tools (e.g., for polyps). Images may be filtered and / or selectively displayed based on user input in other ways.

[0041] Different types of image displays or views may be selected by the user, such as a matrix or grid view, a carousel view (e.g., where the images are displayed as if viewed from the side of a ring of images, with one "current" image being front-facing and largest, and the other images surrounding the "ring" in decreasing size). A map or "heat map" may associate certain images with their spatial orientation along the GIT, with certain features or combinations of features having an "extension" or length of the features indicated, for example, using color along a heat map. The map may represent a portion of the GIT or a portion of the GIT of interest. The map may correspond to first level images in that the map may represent a portion of the GIT from which the images were taken, and / or include markers corresponding to the estimated locations of the images. The map may thus include image indicators or markers that represent the localization of the images according to their location or estimated location in the GIT.

[0042] As used herein, a processor-derived description, such as a location, size, or cleaning level, may be estimated or approximated based on information available to the processor and the computational methods.

[0043] The user can add information to the image, and / or information can be automatically added to the image. For example, the user can label the image with a selectable lesion type. The process can automatically draw a box or border around the location of an automatically identified lesion in an image within the image.

[0044] According to some aspects, creation or generation of a study may include generating additional information or secondary information in addition to the additional one or more images corresponding to the images in the image subset. For example, the additional information may include a division of at least a portion of interest of the GIT into multiple segments; a location of images in the image stream along at least a portion of interest of the GIT; a map representing at least a portion of interest of the GIT, including an indication of images in the image subset along at least a portion of interest of the GIT; a predefined number of images selected from the image subset according to a third selection method; an indication of a feature of interest in the image; an indication of a level of cleaning determined by image, segment, and / or by a protocol; an indication of an estimated size of a feature of interest; an indication of an extent of a feature of interest; a selection of images in the in vivo image stream representing transitions between anatomical segments of at least a portion of interest of the GIT; and combinations thereof.

[0045] In some aspects, a system or method may display a study of a capsule endoscopy procedure, including images captured by a capsule (e.g., swallowed by a patient) during the capsule endoscopy procedure for review by a user. The system may receive a subset of images (e.g., first-level images) selected from a stream or group of in vivo images of at least a portion of the GIT of a patient or subject captured via a capsule endoscopy procedure. According to some aspects, the system may receive a stream or group of in vivo images of at least a portion of the GIT of a patient or subject. The system may then automatically select or select a subset of images from the received stream or group of in vivo images according to a first selection method, the subset of images "representing" the received stream or group of in vivo images. By "representing" is meant a representation made from the perspective of making a medical diagnosis and making a treatment decision with respect to a portion of interest in the patient's GIT, or a representation made for the benefit of making a medical diagnosis and making a treatment decision with respect to a portion of interest in the patient's GIT. For example, the system may identify one or more features of interest (e.g., one or more types of lesions) in the received image. The one or more features of interest may be predetermined based on a particular type of capsule endoscopy procedure. Based on this identification of features of interest, a subset of images may be selected to particularly represent the identified lesions.

[0046] For each image of at least a portion of the image subset, the system may receive, select, or identify one or more images from the in-vivo image stream or group that correspond to an image in the image subset (e.g., a second level image). The one or more corresponding images may be selected according to a second selection method. According to some aspects, for each image in the image subset, the corresponding one or more images are selected and received. According to some aspects, the selected corresponding one or more images may include the corresponding image in the image subset. According to some aspects, the corresponding one or more images may not include the image in the image subset. According to some aspects, the system may select the corresponding one or more images according to the second selection method.

[0047] The second selection method may be based on a filter or relationship between images in the in vivo image stream. Such a filter or relationship between images may be an image identified as including at least a portion from the same feature (e.g., the same specific ulcer) or the same type of feature (e.g., an image identified as including an ulcer), an image captured at a similar time along the GIT, or an image located adjacently along the GIT, or a combination thereof. According to some aspects, a predefined number of images selected from the image subset may be received and displayed. A predefined number of images (e.g., five images, ten images, or any number above ten) may be displayed upon receiving user input. A predefined number of images may be selected according to a third selection method to summarize the image subset. According to some aspects, the third selection method may select an image in the image subset that is determined or estimated to be of most interest. For example, in a colon procedure, an image in the received image stream or group identified with a certain level of certainty as including a polyp may be selected according to the first selection method for the image subset.

[0048] The five images in the image subset identified as including the five largest polyps, or the five images identified as including polyps with the highest level of certainty, may be selected according to the third selection method. Each of the image selection methods described herein may include one or more filters or selection or detection rules, and the selection according to each method may be performed in one or more stages. The selection or detection rules may be applied by utilizing algorithms (e.g., in particular machine learning algorithms and deep learning algorithms). For example, when selecting an image subset from the received image stream, detection of one or more features of interest may be performed at the first stage. Different rules (e.g., algorithms) may be used to detect different types of features, such as passages between different lesions or different anatomical parts of the GIT. Typically, the result of the detection algorithm may be a score assigned to the image, which indicates the level of certainty that the image includes the detected feature. The relationship or filter based on which the second selection method is based may produce different corresponding additional images for different first-level images or different images in the image subset. At least the first selection method may be based on deep learning techniques, such as neural network techniques. The first selection method and the second selection method may each include one or more selection rules.

[0049] At the second stage, images may be selected from the image stream based on the detection results of the first stage. The selection may include, for example, different selection rules for different types of features (such as different types of lesions and passages between anatomical parts). The selection rules at the second stage may be based, in particular, on thresholds determined for the detection score. For example, for the SB procedure, different rules for selecting images of the image subset at the second stage may be determined and based on: the type of lesions or features identified in the image, the type of extension of the lesions identified along the GI tract (e.g., focal, diffuse, or "carpet-like" distribution), and the severity or location of the identified lesions. These rules may also be based on the maximum number of images to be displayed by a particular feature, by the type of feature, and / or by the entire set of images to be selected. For example, for focal lesions, the system may select only one image to represent the lesion that will appear in the image subset (e.g., one rule). For diffuse lesions, the system may select multiple images to represent the lesions in the image subset (e.g., another rule).

[0050] As another example, in a colon protocol used as a cancer screening tool, the features of interest may include only one lesion: a polyp. The system may then include a rule according to which an image is selected for each identified polyp subset of images. The identified polyps may be determined according to a threshold value of the polyp detection score of the image. Alternatively or in addition, another rule may be determined according to which the first subset will include a maximum number of images (e.g., 50, 100, 160, 200, or a maximum number of about tens, a maximum number of about 100, or a maximum number of about hundreds). If both rules are applied, and the number of images selected according to the first rule exceeds the maximum number determined according to the second rule, a third rule may be applied. Such a third rule may determine that the maximum number of images according to the second rule of the images with the highest detection scores selected according to the first rule are selected for the image subset. Images of identified polyps with higher detection scores may be considered more important, for example because such images have a higher probability of showing polyps.

[0051] According to some aspects, rules based on tracking features (such as ulcers of polyps) may be included. Such rules may be applied in additional stages. Such rules may allow for the identification of images that image the same specific feature (e.g., the same polyp). Such rules may be used in a second selection method to determine one or more images that show at least a portion of the same feature shown in an image in a subset of images. Such feature tracking may be performed, for example, based on methods such as disclosed in U.S. Pat. No. 9,430,706 (incorporated herein by reference in its entirety) or optical flow techniques.

[0052] The system creates or generates a study of a capsule endoscopy procedure, for example, by utilizing the methods described herein. The study may include at least a selected subset of images (e.g., a level one view) and optional additional information. The additional information may include, for example, one or more additional images corresponding to each image of at least a portion of the "level one" image subset selected (e.g., by the system). The additional information may also include information such as: localization of the displayed image along the GIT; indication of the location and / or type of the identified or suspected features in the displayed image; indication of the range of the identified or suspected features or the type of features along the GIT (e.g., the range of "carpet-like" lesions or diffuse disease); indication of the segmentation of the portion of the GIT of interest; indication of the anatomical portion of the GIT; and display of images representing anatomical passages between different anatomical portions of the GIT. According to some aspects, a portion or all of the additional information may be displayed only upon user input, such as upon a user request via an action (e.g., by activating a control or hovering over a particular display area). According to some aspects, the level one image and / or the level two image may be displayed in a static manner. Typically, level two images are not displayed only if a user selects a level one image for additional review. Typically, a set of second level images includes one or more images that are not included or displayed as first level images. Study images can be displayed in the same view as the first level images (e.g., when displayed in the same window) or in a different view (e.g., when displayed in a pop-up window).

[0053] A study may be displayed in one or more views for user review. Each study view may include display of at least a subset of images. Different study views may be characterized by, for example, image display layout (e.g., matrix or carousel), image display browsing (e.g., static mode or video), what additional information (if any) is displayed for the study, and the manner in which the additional information is displayed. For example, according to some aspects, two study views may be displayed. One view includes display of a subset of images (e.g., "first level" or "level one" images) and additional information (e.g., GIT segmentation and cleaning level of each segment) in a matrix layout in a static manner, but does not include display of the one or more images corresponding to the images of the image subset (e.g., "level two" or "second level" images). The second view may include display of the subset of images in a carousel layout and display of all additional information included in the study (including second level images). With respect to multiple study views, one view may be defined as a default study view. In contrast to other study views, the default study view is not selected by the user for display. Once a specific procedure is selected for study review, the default study view will be displayed. Therefore, the default study view is the first study view to be displayed. In the case of single study views, the single study view is the default study view.

[0054] The study view may include one or more views. The image subset is displayed in a main or default view of the study view. The main or default view of the study view may also include display of additional information, such as a map localizing the images in the image subset along the GIT; a wash level indication for each image in the image subset; a segmentation of the portion of interest of the GIT; the capture time of the images in the image subset; and / or an estimated transit time of the portion of interest of the GIT (e.g., an estimate of the time required for the capsule device to traverse the portion of interest of the GIT). Other additional information, such as display of additional one or more images corresponding to the images in the displayed image subset, may be displayed only upon receiving user input. Such "optional" display of information (e.g., for the user to decide whether to display the additional information) may be in the main view or in a separate view. For example, the additional one or more images corresponding to the images in the image subset may be displayed in the main view, such as in the same image window in which the corresponding images in the image subset are displayed or in a different dedicated area of ​​the main view of the study view. In another example, the additional one or more images corresponding to the images in the image subset may be displayed in a separate view, such as in a pop-up window.

[0055] A study display or view may include a number of characteristics. Study display characteristics may include the manner in which a subset of images is displayed; the layout of the display of images (e.g., a carousel or matrix), including the number of images displayed simultaneously; the manner in which images are navigated between (if not all images are displayed simultaneously); and the display of additional information. For example, characteristics may include what additional information may be displayed in a particular display or view and what additional information is not displayed in a particular display or view. Characteristics relating to the display of additional information may include the determination of what information is displayed in a particular view by default and what information is displayed in a particular view only upon user request (e.g., the display of indications of features in an image). Study display characteristics may specifically include the layout of the display of secondary images, such as in a carousel or as a video clip, in the same window as the primary images or in a pop-up window or another window or screen, one image at a time or several images simultaneously. According to some aspects, browsing between images in a displayed set of images can be performed per image (e.g., each browsing action replaces only one image currently displayed) or per page or screen (e.g., in case multiple images are displayed simultaneously, each browsing action replaces all images currently displayed), in a static manner (e.g., where browsing between images or pages of images is performed only in accordance with user input), or in an automatic manner (e.g., when a user activates a browsing control and browsing is automatically performed between images or pages of the displayed set of images). According to some aspects, the display of images (e.g., a selected subset of images) is performed per segment of a portion of interest of the GIT and based on a segmentation of the portion. For example, in a research view, the set of images can be displayed per segment, where each segment is displayed by activating a tab. According to some aspects, some display or view characteristics can be predefined by the user or can be changed upon user input (e.g., the number of images displayed simultaneously in the view). In some embodiments, a user may browse or cause the display to move between images or "pages" of the set of first level images (e.g., if the display includes x images, browsing to the next page will present the next x images). Automated browsing (as opposed to video display) may automatically move to the next image or next "page."

[0056] In one embodiment, in a view or display of secondary images, only one image is displayed at a time (as opposed to primary images, where multiple primary images are displayed at a time). The secondary images may be displayed as a video clip, for example using an autoplay mode.

[0057] A study may include all images and information provided by the system to the user, such as all data related to a case or procedure presented to a physician, including image capture time, images (e.g., primary and secondary), and additional data such as analysis results of data related to the case and / or procedure that the system may perform. A study may also include patient information (e.g., details, background, history).

[0058] According to some aspects, embodiments of the system may display a subset of images on a display device (e.g., a monitor) for user review: typically the images displayed are one or more representative images of each group, so each of the different groups may have one or more representative images displayed in a "first level" display. The first level display may display these images by "default," e.g., without user input or selection, and the first level display may display the images in a static manner rather than a moving image or movie.

[0059] Primary images or "representative" images captured by a single imaging device of a capsule device used in the procedure may be displayed simultaneously. The number of images or a maximum number of images that may be displayed simultaneously in a view or display (eg, of primary images) may be predefined.

[0060] According to some aspects, the study includes first level information displayed in a default view of the study. Therefore, the display of the first level information is mandatory and it is not for the user to decide whether to display this type of information. According to some aspects, the first level information consists only of a selected subset of images representing an image stream or group received from a capsule device or from a receiving device. The study may also include second level information. The second level information may be displayed only upon user input (e.g., at the user's request), so its display may be optional. The second level information may provide additional information to the first level information. The second level information or at least a portion thereof is associated with or corresponds to an image in the image subset.

[0061] This selection and display of procedure-related information can allow the user to conduct the review process more efficiently, reduce review time and arrive at a better diagnosis. According to the method and system of the present invention, a user can review the study in a manner that suits his or her skills and preferences and / or the needs of a specific patient, procedure and / or study. In this way, the disclosed system and method can allow for a personalized review of procedure studies. For example, a skilled user may require less information than a novice user to make a diagnosis. Therefore, a novice or less skilled user may choose to display and review additional information that is not necessary for a skilled user. As another example, a review of a study of a procedure performed in subjects who are believed and / or known to be ill may involve the display of more additional information than a review of a study of a procedure performed in subjects who are believed and / or known to be healthy.

[0062] In addition, additional information that is not usually or hitherto displayed or provided to the user in the framework of a study of a CE protocol may be provided according to embodiments of the system and method. Such information may assist and support the user in making diagnostic and / or therapeutic decisions, thereby enabling faster and better diagnosis and care. The additional information includes information generated or implemented, in particular, by employing prior art computer-aided decision-making techniques, such as deep learning techniques.

[0063] According to some aspects, to display the second level information, a user may select an image in the first level display (e.g., by providing user input to a graphical user interface (GUI)) that corresponds to a group of images and therefore may correspond to other images in the group that have not yet been displayed. According to some aspects, all images in a group may include or image the same automatically found feature of interest, and therefore images in the group other than the single image displayed in the first level (the images corresponding to the first level images) may show the same features shown in the images displayed for the group in the first level. Upon receiving user input related to the first level images, the system may display the one or more images corresponding to the selected first level images in the currently displayed subset of images. This may cause a second level display of additional information. For example, the selected group of images may be displayed in a static manner, and then the user may be able to scroll through the group of images, thereby causing the images in the group to be displayed in sequence, or another display method may be used, such as displaying the group of images as a video clip. During the display of the images, the user may add or annotate information relative to the displayed images. After the user has finished reviewing the images, the system may create or generate a report that includes images from the displayed images selected by the user and does not include other images. In some embodiments, the report may include only the images viewed by the user and possibly user input information, and does not include other images in the captured in vivo image stream. Typically, only the selected subset of images and images corresponding to the images in the subset are available for display for user review in the report.

[0064] According to some aspects, an interim study may be displayed and optionally generated by the disclosed system and method. An interim study may be generated based on a partial image stream or group received from a capsule device or from a receiving device in communication with the capsule device. The partial image stream may include only images captured until a certain time point relative to the beginning of the procedure (e.g., the subject swallows the capsule) or until the capsule reaches a certain position within the portion of interest in the GIT and before the end of the procedure. The generation of the interim study is typically performed by a medical professional during the procedure to receive the partial image stream and to review the interim study. According to some aspects, the interim study is generated and displayed before the capsule device reaches the end of the portion of interest in the GIT. The generation and display of the interim study may be performed by the disclosed system according to the disclosed method with desired changes for user review. The interim study may include a subset of images selected from the received partial image stream. The interim study may typically include less additional information than a complete study (e.g., the study). For example, in some embodiments, the interim study does not include secondary images. The user can then generate an interim report based on the interim study using the systems and methods disclosed herein. The interim study can be used to provide needed medical diagnosis and treatment, including referring the patient for a medical procedure to be performed on the same day or shortly after the CE procedure is completed.

[0065] For example, if a medical professional identifies a polyp that should be removed when reviewing an interim study, the patient can be referred for a same-day colonoscopy. A same-day colonoscopy can have advantages, including sparing the patient from having to undergo another colon preparation. According to some aspects, a user can determine parameters, such as determining a certain point in time of the portion of the image stream from which the interim study is generated or a certain location within a portion of interest of the GIT. According to some aspects, a user can select the time of day at which he will receive the interim study.

[0066] The last image may be displayed. This last image may be the last image captured by the capsule or an image identified as being at the last location captured by the capsule (note that the capsule may go back and forth). This image may be displayed, for example, to let the physician know where the capsule has been capturing images and whether it has reached a portion of interest. This last image may be part of a subset of images or may be part of additional information.

[0067] According to some embodiments, images captured in vivo can be received and can be (e.g., by Figure 1 A processor (such as a processor shown) automatically selects a plurality of these images for display. In some embodiments, a subset of these selected images can be identified automatically and / or by a user, and a case report or report can be generated that includes only images from the identified subset of images (e.g., one or more images or all images in the subset).

[0068] The images selected for display may be, for example, images suspected of including a particular lesion, abnormality, or any abnormal condition and / or images selected to represent or illustrate a certain portion of the GIT or a certain anatomical region of the GIT.

[0069] The disclosed systems, methods, and computer program products may be implemented or executed via one or more software applications that may receive or have access to the selected image and / or the entire image stream captured in vivo and / or a portion thereof. Such applications may, for example, display the selected image to the user, interact with the user (e.g., via a GUI), facilitate the user's analysis of the displayed image, provide automatic analysis of the displayed image, and / or generate a report. According to some embodiments of the present disclosure, such applications may perform the selection of the image to be displayed. The system and computer program product according to the present disclosure may include one or more non-transitory computer-readable storage media (which may also store data, such as images, image streams, etc.). The system according to the present disclosure may include one or more hardware processors, one or more displays, one or more I / O devices, and / or one or more communication devices utilizing one or more communication technologies. The one or more software applications may include program code stored in the one or more non-transitory computer-readable storage media and executable by the one or more hardware processors. These images and / or any additional data may be displayed to the user via the one or more displays, and the user may interact with the one or more applications via the one or more I / O devices. The image stream, a portion thereof, or a selected subset of the image stream may be received by the one or more communication devices or accessed remotely (eg, via the Internet).

[0070] Figure 1 A high-level block diagram of an exemplary computing device that can be used with embodiments of the present invention is shown. The computing device 100 may include a controller or processor 105, which may be or include, for example, one or more central processing unit processors (CPUs), one or more graphics processing units (GPUs or GPGPUs), chips or any suitable computing or operating device, an operating system 115, a memory 120, a storage device 130, an input device 135, and an output device 140. The module or equipment that collects or receives the medical images collected by the swallowable endoscopic capsule (e.g., a receiver worn on the patient) or displays or selects to display the medical images (e.g., a workstation) may be or include such as Figure 1 The communication element 122 may allow communication with a remote or external device, for example, via the Internet or another network, via radio, etc.

[0071] The operating system 115 may be or may include any code segment designed and / or configured to perform tasks related to coordinating, scheduling, arbitrating, supervising, controlling, or otherwise managing the operation of the computing device 100 (e.g., scheduling the execution of programs). The memory 120 may be or may include, for example, a random access memory (RAM), a read-only memory (ROM), a dynamic RAM (DRAM), a synchronous DRAM (SD RAM), a double data rate (DDR) memory chip, a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short-term memory unit, a long-term memory unit, or other suitable memory unit or storage unit. The memory 120 may be or may include a plurality of possibly different memory units. The memory 120 may store, for example, instructions (e.g., code 125) for executing a method and / or data such as user responses, interrupts, etc.

[0072] Executable code 125 may be any executable code, such as an application, program, process, task or script. Executable code 125 may be executed by controller 105 under the control of operating system 115. For example, executable code 125 may cause the display or selection of medical images as described herein to be displayed when executed. In some embodiments, more than one computing device 100 or components of device 100 may be used for multiple functions described herein. For various modules and functions described herein, one or more computing devices 100 or components of computing devices 100 may be used. Devices including components similar or different to those included in computing device 100 may be used, and these devices may be connected to a network and used as a system. One or more processors 105 may be configured to perform embodiments of the present invention by, for example, executing software or code. Storage device 130 may be or may include, for example, a hard disk drive, a floppy disk drive, a compact disk (CD) drive, a CD recordable (CD-R) drive, a universal serial bus (USB) device or other suitable removable and / or fixed storage unit. Data such as instructions, codes, medical images, image streams, etc. may be stored in storage device 130 and may be loaded from storage device 130 into memory 120 where it may be processed by controller 105. In some embodiments, the memory may be omitted. Figure 1 Some of the components shown.

[0073] Input device 135 may be or may include, for example, a mouse, a keyboard, a touch screen or a touch pad, or any suitable input device. It should be appreciated that any suitable number of input devices may be operably connected to computing device 100, as shown in box 135. Output device 140 may include one or more monitors, screens, displays, speakers, and / or any other suitable output devices. It should be appreciated that any suitable number of output devices may be operably connected to computing device 100, as shown in box 140. Any applicable input / output (I / O) device may be connected to computing device 100, such as a wired or wireless network interface card (NIC), a modem, a printer or fax machine, a universal serial bus (USB) device, or an external hard drive may be included in input device 135 and / or output device 140.

[0074] Embodiments of the present invention may include one or more articles of manufacture (e.g., memory 120 or storage device 130) such as a computer or processor non-transitory readable medium or a computer or processor non-transitory storage medium, such as, for example, a memory, a disk drive, or a USB flash drive, which encode, include, or store instructions, such as computer-executable instructions, which when executed by a processor or controller perform the methods disclosed herein.

[0075] You can use methods including Figure 1 Multiple computer systems including processors in the capsule, receiver, cloud-based system, and / or a workstation or portable computing device for displaying images may include Figure 1 Components. Including such as Figure 1 A cloud platform (e.g., a remote server) of components in the medical device may receive procedure data such as images and metadata, process and generate a study, and may also display the study for review by a physician (e.g., on a web browser executing on a workstation or laptop). An "on premise" option may use a medical facility's workstation or local server to store, process, and display images.

[0076] According to a certain embodiment of the present disclosure, a user (e.g., a physician) can build his or her understanding of a case by reviewing a display of static images (e.g., captured by a swallowable capsule) that are, for example, automatically selected as images of possible interest. The case can be, for example, images captured during a capsule endoscope's passage through a patient's gastrointestinal tract (GIT) or a portion thereof (e.g., the small intestine and / or colon). The static images can be displayed in one or more screens and can be arranged in various layouts (e.g., arranged in a matrix, arranged in rows, arranged in columns, and / or arranged in or on a graphical representation of the GIT or a portion thereof). According to some embodiments, the display of the static images can be a default study display or study mode or view. According to some embodiments, these images are displayed only in a static manner (as opposed to an image stream or video). The user can browse between static images (e.g., images displayed in a static manner), for example, by scrolling between one or more images (e.g., between groups of images). In some embodiments, the static images can be displayed horizontally to simulate the elongated structure of the GIT. Such a display can ease the physician's review task. The number of static images displayed simultaneously can be predetermined or preselected by the user. In some embodiments, the number of images displayed simultaneously can be changed by the user.In some embodiments, the number of images displayed simultaneously can be between three and seven images.

[0077] According to some embodiments of the present disclosure, a relatively small number of images from the captured images are displayed for user review on a case-by-case basis. By "relatively small number" is meant about a few hundred at most or at least on average, as opposed to current methods, which display image video streams that typically include thousands of images (e.g., about 6,000 images) on a case-by-case basis. In some embodiments, only a few hundred images at most are displayed for user review. In some embodiments, the number of images displayed for user review is about 1,000 at most (i.e., less than 2,000 images on a case-by-case basis).

[0078] Reviewing static images (e.g., images displayed in a static manner) can allow a user to have more control over when and how each image and / or each group (or array) of images is displayed, unlike viewing or reviewing a video stream (e.g., a movie) of images. Unlike viewing or reviewing a video stream of thousands of images, reviewing a relatively small number of static images can significantly ease the review process for the user, reducing case-by-case reading time and leading to better diagnoses.

[0079] The selection of only a relatively small number of images to be displayed and reviewed per case or study can be achieved, in particular, by utilizing a selection or decision method that provides high sensitivity per case (e.g., by providing a high probability of identifying an image of interest) as well as high specificity (e.g., by providing a high probability of identifying an image that is not of interest). According to some embodiments, image selection can be performed by utilizing prior art methods (such as deep learning methods).

[0080] In some embodiments according to the present disclosure, additional information related to the capsule endoscopy procedure may be displayed (e.g., in addition to the selected image). The additional information may be displayed by default (e.g., displayed in a default screen) or may be optional and displayed upon request of a user or following an action or input by a user.

[0081] In some embodiments, the additional information may relate to a currently displayed image or group of images. For example, a user may indicate a currently displayed static image or multiple static images to receive additional information related to the indicated image. The additional information may include, for example, one or more of the following: one or more images surrounding or adjacent to the indicated image; an indication of a region of interest within the indicated image; and / or the type and / or size of a lesion and / or any other structure or feature of interest that the indicated image is suspected or identified as including (e.g., with a certain level of certainty).

[0082] In some embodiments, a display of a case (e.g., a set of images taken from a patient during a single passage of a capsule through the patient or during a procedure) may include two levels or categories of information that may be displayed separately or together based on user input and settings: a first level (e.g., "level one"), which includes a default display that displays selected static images for user review by default (e.g., a main view of a study default view); and a second, optional level (e.g., "level two"), which includes display of additional information and may be displayed upon request of the user or following an action by the user.

[0083] A default or first or primary screen or display (e.g., of a default study view) may include "first level" images of a case that may have been automatically selected as images of interest, such as images showing a lesion (with a certain level of certainty). In some embodiments, if multiple images are identified as capturing a particular lesion (e.g., an ulcer), only one of those images may be selected to be presented in the default or primary screen (e.g., the first level image). Other features or structures may be identified as structures of interest separate from the lesion (e.g., a certain anatomical structure or foreign body, anomaly, combination thereof, etc.). A user may select an image from the first level and "activate" that image to view images of a second level that are related to the image selected in the first level. For example, if a user selects (e.g., clicks or hovers or otherwise indicates) an image of an ulcer from the first level display, then in the second level, for example, the user may scroll between other images of the ulcer (which may precede and / or follow the selected image).

[0084] In some embodiments, the display of options (e.g., via a control) or the display of second level or category information relative to a first level focused image (e.g., an image selected by a user) may be performed via a separate window or screen (e.g., a pop-up window). Typically, such a separate window or screen will appear upon a user request or action, such as upon receiving user input related to an additional display level or category. In some embodiments, a first level focused image is displayed in a separate view (e.g., a separate window or screen) with its associated second level image. A separate window or screen may provide more screen area to display the focused image and its associated second level information.

[0085] In some embodiments, the second level information and / or controls for displaying the second level information may be presented in the same initial or default screen: for example, a GUI item that allows a user to input a request for a second level display or to control the second level after the second level is displayed, wherein the second level display may appear with the first level display, for example, in the same screen or display area or in a dedicated screen or display area. In such embodiments, a separate window or screen is not required or displayed to present the second level information. By providing the second level information in the same screen or window as the first level or default information, this allows the user to remain in context, thus facilitating and improving the review process.

[0086] In some embodiments, the second level information may include additional images. Additional images may also be selected from the received series or group of captured images. In some embodiments, the second level information may include multiple images around one or more static images displayed on the first level and indicated by the user. The surrounding images may be selected based on their capture time relative to the capture time of the indicated image and / or based on their position along the imaged GIT portion relative to the position of the indicated image. In some embodiments, a predetermined number of images that are most adjacent to the image indicated in the first level may be displayed in the second level. In some embodiments, the second level display may include the indicated first level image and a display of images captured around or approximately at the capture time of the indicated first level image. In some embodiments, for example, in the case where two images in the first level are estimated to be adjacently positioned in the imaged GIT portion, the second level display may be combined for two or more images of the first level. In some embodiments, the additional images and the indicated images may be displayed as static images and / or as moving images (e.g., displayed as a video clip, a moving image sequence, or a movie) in the second level.

[0087] In some embodiments according to the present disclosure, a static image may be displayed in a localized manner, for example, along with location information providing the location of a capture site of the image within the GI tract. In some embodiments, all images selected for display may be displayed in a localized manner. In some embodiments, localization of an image may be provided only at the request of a user (e.g., as part of the second level information). Typically, a user request or user selection or user mode change or other input is received as an input at a computing device (e.g., such as Figure 1 In some embodiments, only images selected by the user as images of interest (e.g., images identified by the physician as depicting polyps in a colon procedure) are displayed in a localized manner. Localization of a finding or lesion may provide important information that may influence, for example, a treatment decision or the treatment provided. Thus, for example, the location of an identified polyp may determine whether a sigmoidoscopy or colonoscopy should be performed to remove the polyp and may guide the physician in doing so.

[0088] The imaged portion of the GIT can be divided into multiple segments. For example, the small intestine can be divided into three segments of equal length, for example, according to the methods disclosed in U.S. Patent Publication No. 2019 / 0244351 (which is incorporated herein by reference in its entirety) or other known methods. http: / / www.cs.technion.ac.il / ~cs234313 / projects sites / S18 / 12 / site / (which is incorporated herein by reference in its entirety) divides the colon into, for example, five anatomical segments: the cecum, the ascending or right colon, the transverse colon, the descending or left colon, or the left colon-sigmoid colon and the rectum, or into two anatomical segments separated by the splenic flexure. The display of the images (e.g., the first-level images, the second-level images, or both) may include an indication of which segment each image is positioned in. In some embodiments, the location of each image within a segment may be indicated. In some embodiments, the identified region of interest (e.g., the identified feature of interest in the image) or its location (e.g., the location within the image) may be indicated in the image. For example, an outline may define the identified region, or an arrow may indicate the location of the identified region in the image. The location of the image along the imaged GIT portion may be determined by known position estimation methods. For example, the location of the image along the imaged GIT portion may be determined based on a computerized assessment of the progress of the capsule device through the subject's GIT. Such computerized assessment may be based on, for example, the methods and systems disclosed in U.S. Patent 8,792,691, entitled “System and method for detecting motion patterns of in vivo imaging devices,” to Krupnik et al. and assigned to a common assignee of the present application, or other known methods.

[0089] Generally speaking, the division of the GIT into multiple anatomical segments may be performed, for example, based on the identification of capsule endoscope traversal between different anatomical segments.Such identification may be performed, for example, based on machine learning techniques.

[0090] In some embodiments, a map representing at least the entire imaged GIT portion or a segment thereof may be displayed. The map may include an indication of the selected image (e.g., a first level image and / or a second level image). Thus, the map may provide the user with an overview of the imaged GIT portion or segment thereof regarding suspected areas or areas of possible interest, their extent (e.g., their occurrence along the length of the GIT), and / or their distribution. The extent may be estimated or generated automatically by the system and presented to the user visually (e.g., an indication of the length on the map) or numerically (e.g., the size of the extent, such as in percentage). The user may use the map to navigate between the selected images. In some embodiments, the map may be in the form of a bar or in a form similar to that of the GIT. In some embodiments, the indication of the selected image may be in the form of a heat map, i.e., an abstract representation of all or a portion of the GIT, such as a bar showing the selected image according to its location along the GIT. "Heat" may refer to a color used to convey information on the map, such as a color assigned to a line representing certain images along a bar. Different colors may represent, for example, different types of images, such as first-level or second-level images. Such a heat map or bar may include lines (e.g., colored with different colors) indicating segmentation of GI portions, etc. Colored segments of the map may provide certain information about those segments. The system or GUI may receive a user selection of an image indication or location in the map and, in response, display the image represented by the user-selected image or location indication or additional information related to the image indicated by the map.

[0091] The indication of the selected image in the map can be based on its estimated location along the imaged portion of the GIT. In some embodiments, segments of the imaged portion of the GIT can be indicated in the map. In some embodiments, the currently displayed segment (i.e., the segment where the currently displayed image is located) can be indicated in the map (e.g., by coloring or highlighting the segment). The selected images can be indicated in the map based on their time sequence. In some embodiments, the focused image can be indicated in the map. The user can select an image to be displayed or an image to be focused on from the map.

[0092] In some embodiments, the map can include indications of the type of features or structures of interest identified along the portion of the GIT represented by the map. These indications can be, for example, icons representing the type of feature (eg, hemorrhage, ulcer, or polyp).

[0093] In some embodiments, the map and / or graphical representation may be displayed above the selected image and / or on a top section of the screen. Such a top display may facilitate navigation through the selected image via the map (e.g., the user would not be required to move their gaze (e.g., lower their gaze) to view the map).

[0094] The cleaning or cleanliness level that may be provided or displayed by embodiments of the present invention may describe the level of cleanliness of the intestine relative to contents, feces, and food debris, typically after "preparation" prior to an examination procedure (such as capsule endoscopy) that is intended to clean the GI tract from such visual obstructions. Known standards for colon cleanliness levels include 4 levels (other numbers of levels or standards may be used). There are also several methods for assessing the cleanliness of the small intestine; in one embodiment, the cleanliness standard for the colon can be used to describe the cleanliness level of the small intestine.

[0095] In some embodiments, the additional information may include an indication of the level of cleanliness of the image or a graphical representation that provides an indication of the level of cleanliness of the imaged portion of the GIT. This may be done for each image of a captured sequence of images or only for selected images and / or for each segment of the imaged portion of the GIT and / or for the entire imaged portion of the GIT (e.g., by a computer such as Figure 1 The cleaning level can be estimated or determined by the system in the image automatically). Generally speaking, the imaged GIT portion may include substances such as contents that may blur the tissue of the GIT (e.g., substances naturally present in the GIT that may blur the imaging) and / or bubbles. These images can then be analyzed to detect such obstructive elements. A score can be assigned to each image, which indicates the cleaning level detected in the image. In some embodiments, the cleaning level can be presented in the form of a chart or a bar (e.g., a color bar). In some embodiments, a graphical representation of the cleaning level can be displayed below the mapping map or above the mapping map and displayed in a corresponding manner. In some embodiments, some categories such as low cleaning level, medium cleaning level, and high cleaning level can be determined, and the cleaning level of the image can be determined based on these categories.

[0096] In some embodiments, it can be (for example, by Figure 1 The system shown in the figure automatically determines and assigns a score to each image or each selected image and / or segment and / or procedure, which score can reflect the mucosal coverage of the imaged segment of the GIT (e.g., the colon). The procedure for this matter can be a set of images captured by a specific capsule after being swallowed by a specific patient at a specific time instance. For example, such a score can reflect how much (e.g., in percentage) tissue (mucosa) of the GIT segment is captured by these images. Such a score can be based on, for example, estimated or determined wash levels, contents, bubbles, capsule transit time, capsule speed, and / or capsule dynamics. The score can provide information about the appropriateness of the procedure to the physician or viewer. The score can be provided as a supplement or alternative to the wash level, and the score can be displayed in a similar manner as described above with respect to the display of the wash level. The score can be provided to the user by default or upon request before, during, and / or after the review of the selected images.

[0097] In some embodiments, the additional information may include a sequence of images in the captured image series that are identified as including an anatomical segment of the imaged GIT portion or a passage between anatomical segments. Such a passage may be, for example, a passage from the stomach to the small intestine, a passage from the small intestine to the colon, or a passage from the colon to the outside of the patient's body. In some embodiments, the additional information may include only a representation of such an image sequence (e.g., a summary of these images). In some embodiments, the representation of the image passage sequence may include a color bar. Such a passage representation may allow a user to more closely review the passage between different anatomical segments of the imaged GIT portion. In some embodiments, the image passage sequence or a representation of such a sequence may be displayed, for example, in a map representing the imaged GIT portion upon user request or after a user action (e.g., hovering over an indication of such a passage). In some embodiments, such an anatomical passage may be a feature of interest to be identified and represented in the first-level image or in a selected subset of images. For example, a single image may be selected to represent a walkthrough in the image subset (eg, a first level image), and the one or more images associated with the selected image in the image subset (eg, a second level image) may be additional images selected to represent the walkthrough.

[0098] In some embodiments, an automatic size estimate of a feature, lesion, or any other finding (such as a polyp) automatically identified in an image and / or indicated in an image by a user may be provided. The automatic size estimate may be provided as additional information (e.g., secondary information) and / or upon user request and not by default. In some embodiments, an indication of a region of interest and its size estimate may be provided by default. According to some embodiments, the automatic size estimate may be based on a known size estimation method that provides a size estimate, for example, based on user input. The user input may indicate at least one dimension of the feature to be measured as shown in the image, such as a length or diameter represented by two or more image pixels indicated by the user. Such methods may be found in U.S. Patents 9,412,054 and 9,911,203, which are incorporated herein by reference in their entirety; other known methods may be used. To generate a complete automatic size estimate, an automatic estimate of the user input may be generated. For example, by segmenting the identified feature of interest in the image and selecting two edge points or two perimeter points of the segmented feature that are farthest from each other. In some embodiments, the user can correct or replace the two points automatically selected and receive a size estimate based on the input provided by the user.

[0099] In some embodiments, an automatic interim report may be generated during a capsule endoscopy procedure. Such an automatic interim report may be based on, for example, images captured up to a certain time and / or images based on certain identified anatomical regions of the imaged portion of the GIT and / or images based on specific findings (e.g., significant findings such as heavy bleeding or polyps that need to be removed). An interim report may be generated, for example, after a predetermined period of time has passed since the patient swallowed the capsule endoscopy device or since it has been identified that the capsule endoscopy device has reached an anatomical landmark (e.g., the beginning of the small intestine or the beginning of the colon). As another example, an interim report may be generated once it has been identified that the capsule endoscopy device has reached a certain anatomical landmark or once a certain lesion or event (such as bleeding or polyps) has been identified. The interim report may then be provided to a designated physician.

[0100] The automatic interim report may include images selected to represent the received set of image streams and / or specific findings. In certain embodiments, the automatic interim report may also include additional information as disclosed herein. In some embodiments, the automatic interim report may include an indication of a specific finding, a suggestion for a diagnosis, and / or a suggestion or recommendation for a therapy or course of treatment action.

[0101] In some embodiments, the generation of an automatic interim report or interim study for review by a medical professional according to various embodiments (which indicates in real time the identified lesions or any other findings that may require medical intervention (such as surgery or minimally invasive procedures)) can allow such procedures to be performed without delay, for example immediately or just after or shortly after the completion of a capsule endoscopy procedure. In some cases, such as in the case of heavy bleeding, immediate medical intervention may be required. However, even when the required medical procedure is not urgent (for example, in the case of identifying a polyp that requires removal), the interim report can allow the fasting and / or bowel preparation performed by the patient for the capsule endoscopy procedure to be used for the required medical procedure, such as by performing a colonoscopy on the same day. A colonoscopy on the same day can be more convenient and less difficult for the patient, for example, who will not have to undergo exactly twice.

[0102] The generation of automatic interim reports or interim studies in real time and during active procedures can be facilitated, for example, by utilizing technologies such as distributed computing, decentralized computing, and / or cloud computing. In some embodiments, the automatic interim reports or interim studies may include scores related to the estimated or determined level of cleansing or to the appropriateness of the procedures up to that stage, as described in the present disclosure. In the event that a decision is made to perform a subsequent medical procedure (e.g., a same-day colonoscopy), such information can be used by a physician to provide instructions to the patient regarding preparation and / or diet and / or to modify the instructions provided to the patient.

[0103] Embodiments may apply, mutatis mutandis, to the display of static images and / or moving images (e.g., videos). In some embodiments, additional information included in a study according to the systems and methods disclosed herein may include a suggestion for a diagnosis and / or a suggestion or recommendation for a therapy or course of treatment action. Such a suggestion may be based on processing and analysis performed by and according to the systems and methods disclosed herein, and may be based on relevant standards of care.

[0104] In one embodiment, the selection of images of interest to be displayed for user review is performed in two stages. In the first stage, (e.g., by a user such as Figure 1 A computing device in the GIT) automatically selects a plurality of images of interest from the captured series of images and displays the plurality of images of interest to a user, for example, in a matrix, grid, or other layout. The selected images may be displayed in a localized manner, for example, together with location information providing the location of the capture site of the image within the GIT. In a second stage, a user may (for example, by providing user input) select an image from the automatically selected images to receive a display of additional information about the selected image (for example, the focused image). The additional information may include the automatically selected additional images. The user selection of images may then be used to generate or populate a map or report for the entire case.

[0105] Figure 2 A first screen, such as a default screen, is shown that is initially presented to a viewer or user (e.g., as a default screen for a study's default display). The viewer can navigate between various regions or segments of the imaged GIT portion that are displayed in Figure 2 In the example of the example of the small intestine, there are three thirds. Each third can be represented by one of the three tabs 210 on the exemplary screen. When, for example, the user clicks Figure 2 When a segment or small intestine trisection is selected by selecting a corresponding tab 210 in the GUI displayed in the GUI, an automatically selected image 206 related to the trisection is displayed to the user, for example, by a matrix layout 220 at the center area of ​​the screen. In this example, each trisection represents one-third of the travel time of the capsule through the small intestine. In some embodiments, each trisection may represent an estimate of one-third of the length of the small intestine. In some embodiments, these trisections may be parsed based on the detection of landmarks along the small intestine. The user may click or otherwise indicate a button 202 to switch between the report screen and the first view. Estimated entry and exit times 204 for the small intestine or other GIT segments may be displayed.

[0106] In some embodiments, a user can switch between a level one display and a level two display. For example, switching to the level two display can be done by clicking on an image shown in level one. When a particular image is selected, for example by the user clicking on an image while displayed on a GUI, a larger version of that image is displayed in an "additional information" (e.g., second level) area 230 of the screen. Each image displayed with additional information can correspond to a first level image in the default view, for example by showing the same features or lesions shown in the first level image. The user can view additional information related to the selected image in images 206 in the "additional information" or second level area, such as additional still images captured in chronological order before or after the selected image or short clips (e.g., a moving image sequence or movie) that include images surrounding the selected image (e.g., images captured a few seconds before or after the selected image or images captured a few seconds before or after the selected image). In the "additional information" or second level area, the user can view additional information related to the selected image in images 206, such as additional still images captured in chronological order before or after the selected image or short clips (e.g., a moving image sequence or movie) that include images surrounding the selected image (e.g., images captured a few seconds before or after the selected image or images captured a few seconds before or after the selected image). Figure 2 , the secondary images are displayed in the same display view of the primary images. The user may also identify the selected image as an image of interest or otherwise mark the selected image for inclusion in the case report, such as by clicking a dedicated check box 208 that may be located on the selected image in the primary display area. Additional functionality on this screen may be available to the user, such as the option to add a mark (e.g., via the user entering information into the GUI) or annotation to the selected image.

[0107] Such as Figure 2 The screen or display in the example of may include a heat map or other information presentation 240, which may include, for example, a graphical representation of information showing each segment or trisection in graphical form. When the user selects an image of interest, an indication (such as a vertical line 242) may be added to the graphical representation or map of the corresponding segment at a position corresponding to the position of the selected image within the segment. Indications such as 242 and similar indications shown in other figures may be image indicators or markers that represent the localization of the image based on its position or estimated position in the GIT. This second stage selection performed by the user may be separate from or combined with the user selection of images for the report as described above. Therefore, in some embodiments, after the user selects an image to populate the map, he may subsequently perform another selection of images to include in the report. In other embodiments, the image selected by the user to populate the map may also be the image to be included in the report.

[0108] exist Figure 2 In the exemplary display of FIG. 1 , if the number of automatically selected images for a segment or trisection is higher than the number of images that fit in the center display area, scrolling up and down is available to the user. Figure 2 In the example of , an image selected by a user for display in the "Additional Information" area may be highlighted (eg, by a blue frame) and a corresponding marker on the heat map area may also be highlighted simultaneously.

[0109] Figure 3 In some embodiments, the map or case map 310 generated based on the user selection of the image can be displayed in a separate screen, such as Figure 3 As shown in the example. Figure 3 In the example of FIG. 310 , a case map 310 is an elongated bar with various indicia, time markers, and other symbols 312 that graphically illustrates the GIT of a subject of the case (e.g., a person whose GIT is imaged by a capsule). The map may include the entire case or substantially the entire case (including the entire imaged portion of the GIT (which is in FIG. 310 )). Figure 3 This is only a sample representation of a portion of the entire GIT. Figure 3 In the example of , a graphical representation of the entire unfolded small intestine is shown, but other exemplary graphical representations of the entire case are possible. In some embodiments, the exemplary graphical representation may include elements 314 that are similar to or suggestive of anatomical features of the relevant segment. For example, Figure 3 The representation in includes indications of preceding and subsequent organs along the gastrointestinal tract (eg, images of the stomach and colon shown at respective opposite ends of a representation of the small intestine).

[0110] Alternatively, the screen may display a map that more schematically represents the segments.

[0111] Individual markers can be used to delineate individual segments, such as the three thirds of the small intestine. Figure 3 In the example of , these markers are vertical dashed lines 316 that are at the beginning and end of the graphical representation of the small intestine and are at the same distance along the representation.

[0112] The image selected by the user can be represented by a mark on the map. Here, for example, these marks are completely vertical lines 312 and the user can access the corresponding image by, for example, clicking on these marks or hovering on these marks.

[0113] Figure 3 The mapping screen or display of the may also include a graphical representation or indication of the estimated or determined cleaning level, such as a score of the automatically measured cleaning level of the corresponding segment, and / or an indication of the image quality during the capsule's travel in the segment. Similar indications of the cleaning level are shown in other figures herein. The cleaning score may be calculated, for example, based on an automatic estimate of the portion or percentage of the image frame that is covered with contents or otherwise does not display the tissue wall of the corresponding organ. In this screenshot, the cleaning level is represented as a bar 320 including a chart. The cleaning level bar also includes a horizontal dashed line 322, which may represent a threshold level that may help the user assess whether the cleaning level at any given point is above or below the threshold. The threshold may be, for example, an average value or a limit of an appropriate value.

[0114] Figure 3 The map shown may be displayed on a screen or display (eg, Figure 2 ) or together with the selected image.

[0115] In one embodiment, an image from the captured series of images may be automatically selected. A map including an indication of the automatically selected image is initially displayed to the user or displayed by default (e.g., a pre-populated map). The map includes labels for automatically identified regions or segments of the imaged organ. The user may then access a more detailed presentation of the regions of the map that he or she may wish to explore.

[0116] In one embodiment, a graphical representation (e.g., a map) of the entire case, such as the entire portion of interest of the GIT (e.g., the entire SB or the entire colon) is presented to the user. For example, a graphical representation of the entire extension of the small intestine can be presented to the user, which is divided into three tertiles, for example, based on the automatic identification of the capsule endoscope entering the small intestine, its exit from the small intestine, and the measurement of the transit time therebetween. In some embodiments, the division of the GIT portion can be based on an estimate of the length traversed by the capsule endoscopy device. Markings on such a graphical representation (e.g., in the form of completely vertical lines) can indicate automatically selected images that can be or are likely to be images of interest, such as first-level images or images in a selected subset of images. Diffuse findings (e.g., regions of an organ that can include adjacent groups of images of interest or multiple images of interest) can be identified on the map. In such an embodiment, diffuse findings can be displayed as bars that have a lighter shade than the image of interest markings and extend from a corresponding position at the start of the image grouping to a corresponding position at its end.

[0117] Figure 4 An exemplary screen or display showing additional information is shown. In the same screen, a user may provide input, for example, by clicking on a corresponding mark 510 on the map 502 or hovering over the corresponding mark, thereby viewing the selected image. When a user selects an image, for example, by clicking on a corresponding mark 510 or thumbnail 512 displayed in the display area of ​​the map, a larger version 530 of the same image may be displayed in an "additional information" area 540 (e.g., here, in the lower left corner of the screen). The user may then view additional images from the corresponding segment of the organ or a short clip including images captured before or after the selected image (e.g., images around the selected image). Other functions may be available to the user, such as the addition of annotations or marks to the displayed image or the selection of images to be included in the report. Marks 504 (e.g., thirds division marks) that divide the organ into multiple segments may be displayed in their corresponding positions on the graphical representation of the segment.

[0118] Figure 5 shows the zoom operation after Figure 5 502 to view the selected set of images from the corresponding segment. The selected images (e.g., corresponding to) the corresponding segment can be displayed as, for example, a time-ordered series, grid, list 520, etc. of thumbnails 522 on the graphical representation of the corresponding area. The user can also use a reduced representation 540 of the map (e.g., on the Figure 5 ) to navigate between segments or sub-regions of the map (eg, by providing input to the GUI).

[0119] In some embodiments, a user can edit the map, such as by providing input via a GUI to cause the process to remove the marking of a selected image, such as if the user determines that the corresponding image is not an image of interest.

[0120] After completing the review, the user may cause the process to continue to display a report generation screen where the user may review the images selected for the report, such as images selected by the user as images of interest.

[0121] In one embodiment, in the default view mode, the automatically selected images can be displayed in a column layout on one side of the screen in a static manner, such as a vertical column on the right side of the screen. The user can select images to be displayed in the image display area, such as on the left side of the screen. Each selected image can be displayed together with an indication of the identified region of interest within the image (e.g., by coloring the region of interest with a specific color). The display may include check boxes related to the identification of the region of interest. The user can then identify the region of interest and check the appropriate check boxes accordingly (e.g., bleeding, ulcers, or other). A mapping or heat map of the entire portion of interest of the GIT or a section thereof may also be displayed, for example, below the image display area. The mapping may include an indication of the selected image and may also be used by the user to navigate between these images.

[0122] In one exemplary embodiment, the review of an image may be performed based on two levels of information. A first level of information may be presented in a default view mode, and a second level of information may be presented upon a user request (e.g., upon receipt of a request from a user such as Figure 1 The second level information is displayed when the user inputs to the GUI or after a user action. The default view mode may include (for example, by Figure 1The system shown in the figure displays an image automatically selected from a series of images captured of the GIT or a portion thereof (e.g., the small intestine). The imaged portion of the GIT is divided into multiple segments (e.g., the SB is divided into three segments / tertiles, or the colon is divided into five anatomical parts or segments). The selected images are displayed in a static manner according to the segment in which they are located. The user can browse these static images. The default view mode may also include the display of a map of the entire imaged portion of the GIT, which indicates the segments and indicates the selected images according to their estimated positions within each segment. In some embodiments, the default display of the map may include only the segment in which the displayed image is located. In some embodiments, the user can provide input to the process to cause a zoom-in (e.g., an enlargement of the size of the displayed image) of a segment of the map representing a segment of the imaged portion of the GIT.

[0123] The second level information may include a display of additional images surrounding the selected image and may be displayed separately from the first level image (e.g., in a pop-up window or screen). The default view or display (e.g., the first level information) may also include an indication of the section of the GIT portion imaged displayed in the second level information. The default view may also include an indication of the wash level of the series of images captured in the GIT portion. The surrounding images displayed in the second level information may be displayed in a static manner and / or as a video clip, short movie, or moving image.

[0124] Fig. 6A and Figure 6B An example of a case of the small intestine displayed and analyzed is provided.

[0125] Fig. 6A A schematic diagram illustrating a default view mode according to the disclosed embodiment is shown. Figure 6B According to an implementation scheme Fig. 6A Schematic diagram of a map (e.g., bar) in the default view mode of .

[0126] according to Fig. 6A and Figure 6B In the illustrated embodiment, the default view 700 may include a simultaneous display of a group of static images 712 that includes only some images (e.g., four images), thereby allowing the user to carefully and clearly review the displayed images. The group of images may be displayed in a row 710 layout, thereby simulating the layout of the GIT, or in another layout. The images may be displayed substantially in the center of the view. The user may, for example, move the focus (e.g., primary or maximum display) between the images in the displayed image group or scroll between the image groups via user input. The selected images in each segment (e.g., each third position) may be divided into such image groups and displayed according to their chronological order (e.g., according to their capture time). The third position to which the currently displayed image group belongs may be indicated in the view.

[0127] As Fig. 6A and Figure 6B In the example shown, the default view may include a map 740 located, for example, in the bottom portion of the screen. The map 740 may be in a bar or row or line format and may include information indicating the type of information (in one example, Figure 6B An exemplary map is a map of the entire small intestine. The map may be divided into multiple sections, such as three thirds 745, by vertical dashed lines 747. The map may also include a portion of the colon. Fig. 6A and Figure 6B A vertical line 760 and / or a cursor 762 of a different color as shown (e.g., a blue line 760 and a cursor 762) indicate an in-focus image in the map. The frame of the in-focus image may also be a different color (e.g., blue, as in FIG. Fig. 6A and Figure 6B ). The user may also receive an indication of whether a frame has been reviewed in the second level information. Thus, for example, a frame of an image that has been reviewed in the second level may also change its color, for example from black to gray.

[0128] Figure 6B The first sub-row 741 in includes a heat map indicating the selected image positioned along the imaged portion of the GIT (here including the small intestine). The selected image is indicated as a vertical line 770 of a particular color (e.g., gray) and is positioned along the bar according to its estimated position along the small intestine. A user can provide input to the line representing an image (e.g., hovering the mouse over the line) to receive a display of a thumbnail (e.g., a zoomed-out image) of the image and an indication of its capture time. Portions of the bar can be colored with different colors to indicate anatomical regions of interest, such as, for example Fig. 6A and Figure 6B , (e.g., provided as light blue areas indicating estimates of the proximal and distal regions of the small intestine). At the end of the heat map, at least one image of the next anatomical portion of the GIT (here, the colon) may be displayed, among other things, to demonstrate that the entire small intestine or the portion of the GIT of interest has been imaged and that the procedure has been completed. The line indications of these images may change their color (e.g., to orange) after the user reviews these images.

[0129] The second sub-row 742 may include indications of the segments of the imaged GIT portion (here, the small intestine) whose images are to be displayed in the second level information. These segments may be indicated by indications 780 (such as gray rectangles). Each rectangle (e.g., second level segment) may include and / or relate to one or more selected images of the first level, such as may be shown in FIG. Fig. 6A and Figure 6BThis is the case where two or more first-stage selected images are relatively adjacent to each other.

[0130] The third sub-row 743 may include a graph indicating the level of cleaning of the captured images along, for example, the small intestine. The graph may be colored according to the level value, for example, when the value (eg, cleaning) is relatively low, the graph may be colored red.

[0131] like Fig. 6A and Figure 6B As shown, the transit between different anatomical segments of the imaged GIT portion, such as gastro to SB (e.g., stomach to small intestine) and / or SB to colon, may also be indicated. The transit may be indicated above the indication (e.g., dashed line) of the relevant segment in the map and may include the transit time. Optionally, a small color bar may be displayed above the indication of the relevant segment, thereby representing a series of images captured in the identified transit area. The user may provide input to the color bar (e.g., hover) and review the transit images. The color bar may only represent a sample of the images of the identified transit area.

[0132] like Fig. 6A The images shown can be different; for example, selected images can be displayed in a matrix, so that more images can be displayed simultaneously. The user can request, for example, that another view display mode be used instead of Fig. 6A According to this alternative view, the images of each third position can be displayed in a separate matrix, such as a separate tab. The user can scroll between the images in each third position.

[0133] Figure 7 Shown according to Fig. 6A An exemplary illustration of a display or view of second level (eg, study) information for a selected image is shown.

[0134] The second level of information may be implemented in accordance with an embodiment of the present invention by a pop-up window 910 that displays the selected image 912 by default. The window / display / view may display a subseries of images around the selected image (e.g., as a number of images before or after the selected image, or captured a number of seconds before or after the selected image). The user may cause the process to scroll the images in a static manner (e.g., move forward and backward in sequence or time) using an input device (such as a mouse wheel) or he may cause the process to display the subseries of images as a video clip. A map 920 may be displayed below the image display area.

[0135] according to Figure 7In the example of FIG. 1 , the map may include a portion of a first sub-row of the map of the default view in the first level, the portion representing the relevant segment of the imaged GIT portion. The segment representing the image sub-series of the second level may be indicated with a color (e.g., orange). An additional second level map may be presented above the portion of the first level map, which includes an indication of a selected image of the first level display included in the second level image sub-series. When such an image (e.g., a first level selected image) is displayed in the image display area of ​​the second level data pop-up window, it is indicated in the second level map (e.g., indicated by a square; in the example shown, the relevant square is colored blue and a blue cursor is located over the marker).

[0136] A portion of the first level map representing the region including the second level image may be displayed below the image display area. The selected image is indicated in the map, and the first level selected image under review is indicated, for example, by coloring the image line indication with a different color.

[0137] In one embodiment, one image can be selected at a time from a selected group or "stack" of images by browsing through them horizontally (e.g., in a carousel). A heat map representing the imaged portion of the GIT and indicating the selected image can be displayed, for example, above the image display area. Each image displayed can be referenced to its indication in the heat map, for example, by a line connecting the image and its point or representation on the heat map or map.

[0138] One embodiment can display one image at a time from a selected group or stack of images. A user can view a single image by providing input to the GUI using an input control (e.g., a mouse, a scroll wheel, etc.) to cause vertical scrolling through the images, thereby selecting one image at a time. A heat map or other mapping map representing the imaged GIT portion and indicating the selected image can be displayed, for example, next to the image display area. Each image displayed can be referenced to its indication in the heat map, for example, by a connecting line connecting the image to the associated heat map or mapping map location or image indication.

[0139] The default view mode (e.g., the primary view of the default view of the study) may include display of a heat map representing the imaged GIT portion and indicating the selected images. The images may be positioned in the heat map according to their estimated locations in the imaged GIT portion. The selected images may be displayed in the image display area. The selected images may be displayed in a static manner or as part of a video clip including additional related images at the request of the user or after the user's action. The user may hover or click on an image indication, such as on a heat map, to receive, for example, a thumbnail and / or static display and / or clip of the image.

[0140] Fig. 8ADepicts a screenshot or display of an alternative viewing mode according to one embodiment. Fig. 8A In the illustrated embodiment, the images in the first level or selected subset of images may be displayed in a slider or "carousel" 1300 with one image 1302 more prominently displayed in the center, while other images 1304 are displayed to the side in decreasing size with distance from the main center image 1302, whereby user input (e.g., to arrows 1310 at the edge of the screen, or using a mouse scroll bar) causes a "rotation" such that another of the other images 1304 in the view moves to the center and becomes the center image 1302. A map 1320 with washout 1326 and other additional information may be displayed in the first level image carousel display 1300 (typically above).

[0141] The second level image can be displayed in the same window as the first level, so there may be no need to activate another window. The user can, for example, hover over or click on the image, and the image will be "activated" or become "in focus", such as Fig. 8A For example, a user may provide input to the GUI to switch to level two, and the center image 1302 may be viewed as a video or image stream of all images (e.g., second level images) related or associated with a particular instance represented by a center image in level one, or a "scrollable" sequence of images displayed in a static manner.

[0142] In this embodiment, static images (e.g., automatically pre-selected images) are displayed horizontally in a carousel format 1300. Fig. 8A In the specific example in FIG. 1300 , five images are displayed simultaneously. The center image 1302 is the image in focus. After that, the number of images typically displayed simultaneously is typically an odd number of images. In some embodiments, various odd numbers of images may be displayed simultaneously. These images may be displayed in a carousel manner, allowing the user to browse horizontally between these images and allowing no image to hide another image. The size of these images changes according to their position in the display carousel 1300. The center image 1302 (which is the image in focus) is the largest. The size of the other images 1304 changes relative to their proximity to the center image. The closer the image is to the center image 1302, the larger it is. This layout may allow the user to review one image carefully and more clearly, while at the same time reviewing and receiving information about adjacent or surrounding images. Such a view may allow comparison between adjacent images while still displaying the image in focus in detail.

[0143] A bar or map 1320 (eg, a heat map) may depict a representation of the entire imaged GIT portion or a segment thereof. Fig. 8AThe exemplary display shown in is a display of the small intestine. A map 1320 is displayed above the display of these images and on the top area of ​​the screen. Thus, the map represents the small intestine and is divided into three trisections (e.g., by vertical dashed lines 1322). The map may include information about the position of the displayed image along the imaged GIT portion and / or the level of cleansing of the displayed image and / or the captured image and / or the imaged GIT portion. The map may include two or more sub-rows presenting different information. The in-focus image may be indicated in the map, for example, by a connecting line 1324, such as Fig. 8A In some embodiments, all currently displayed images may be indicated in the map (eg, by symbols or lines or connecting lines).

[0144] In an embodiment of the present invention, map 1320 includes two sub-rows 1326 and 1328. The first sub-row 1326 may include information about the cleaning level of the captured image. In an embodiment of the present invention, the cleaning level is presented in the form of a chart and may be presented in the form of a color bar (e.g., Fig.10 Different colors indicate different cleaning level categories, such as: low, medium and high. The second sub-row 1328 may include a bar on which the selected image is indicated according to its estimated position along the imaged portion of the GIT (e.g., indicated by a vertical line).

[0145] In addition, according to an embodiment of the present invention, the secondary information is presented in the same initial or default screen (e.g., the main or default screen of the study view). The focused image (e.g., the center image 1302) is shown with additional controls 1308 that allow the user to scroll, for example, between images surrounding the selected image displayed as the secondary level. The scroll bar may indicate the numbering of the surrounding images relative to the focused image. For example, the focused image may be indicated as "0", the next right image as "1", the next right image as "2", and so on. The image before the focused image (i.e., the next left image) may be indicated as "-1", the image before that image as "-2", and so on. Thus, the user can be oriented relative to the distance and direction (in image and time) related to the initially selected image 1302. In addition, a text box may be displayed below the focused image, allowing the user to enter information, such as the identified lesion type and annotations. In some embodiments, once an image is selected as the focus image and / or once a user arranges an image as the center image 1302 (e.g., by scrolling) or once a user hovers over a focus image, clicks on a focus image, or upon any other user action, a second level of controls and / or data may appear (e.g., below the image).

[0146] Figure 8B Depicted Fig. 8AA matrix layout of image 1300 is shown.

[0147] Fig. 9A is an exemplary screen of a default view mode according to one embodiment of the present disclosure. Fig. 9B is Fig. 9A An exemplary screenshot of a display or view of second level information for a selected image in the display.

[0148] In this embodiment, the case displayed for review is the case of the colon. Fig. 9A In the default screen shown (e.g., the default screen of the study view), images selected (e.g., automatically by an image selection process) from a stream of images captured during a procedure are displayed in a matrix 1400. The colon can be divided into five anatomical segments: cecum, ascending colon, transverse colon, descending colon-sigmoid colon, and rectum. The selected images displayed can be localized (e.g., displayed with localization information) according to these five segments and displayed segment by segment - for example, the images may not be localized at a specific location, but rather by being indicated as being located within one of the five colon segments or in another GIT segment or category. The images of each segment can be displayed in a dedicated display area, and the user can switch between the display of the segments via tabs. Fig. 9A The particular screen shown shows images identified as being captured in the descending colon-sigmoid colon segment. Images selected by the user as images to be included in the report may be indicated by, for example, a blue frame 1410. Additional information 1420 may be displayed to the user. Such additional information may be ( Fig. 9B ): a graphic representation indicating which segment of the colon is shown; patient-related details; the date of the procedure; the time of the procedure in the relevant region of the GIT (here, the precolonic region and the colon); and whether the study is complete (in the example shown, whether the capsule has reached the end of the colon). Such indications of procedure time and completeness may be determined based on the identification of relevant landmarks, such as: passage or entry into the colon and exit from the body.

[0149] Additional information, such as secondary information, may be displayed upon user request. For example, a user may click on a displayed image to open a pop-up window 1450 that displays the image by default and allows review of additional information, such as Fig. 9B As shown. Additional information displayed or potentially displayed (e.g., upon user request) according to embodiments of the present invention may include automatic indication of regions of interest in an image (e.g., the region of a polyp or which head of the capsule captured the image) in the case where the capsule endoscopy includes more than one camera. The indication of the region of interest may be, for example, by segmenting the region and / or by pointing to such region with an arrow. Automatic identification and indication of regions of interest (e.g., suspected polyps) may provide guidance to physicians and improve diagnosis, such as detection of polyps.

[0150] Fig.10 A sample display according to icon 1010 is depicted (in one embodiment, a depiction of a stack of images, with the number "3" near it corresponding to the three images 1032 selected by the user). The selected images 1032 may be included in a window or dialog box 1030. The user may click or select (e.g., input to the GUI using a mouse) icon 1010, and the system may cause window 1030 to appear, including the images checked or selected by the user. The user may then review the checked images 1032 all at once in an adjacent or row manner.

[0151] In another embodiment, the system may display a certain number of the most relevant or suspicious images in another view or display (e.g., "top X most suspicious images," such as the 5 or 10 most relevant images). Such a view may provide a first quick indication to the user and may be useful, for example, for colon screening. In the context of colon screening, the user may wish to determine if polyps are present (polyps may indicate the possibility of cancer and the need for a colonoscopy). This view may be added in another tab or window.

[0152] The automatic polyp (or other feature) size estimation function may calculate the diameter, length, or other size estimate of the polyp in response to a user's request and provide an indication thereof. The size of the polyp or feature may be automatically calculated by, for example, measuring the maximum distance between two points in the perimeter of the identified polyp. The user may select other points to measure the size of the polyp and the process may calculate the size according to the user's selection. For example, the display of the feature or polyp may include a bounding box of the polyp such that the polyp is located within the box. A line in the bounding box (which represents the measured length) may be displayed, and the size may be displayed or presented, for example, near the line. In the event that the user believes that the automatically selected point is inaccurate, the user may change the position of each of the two end points of the line. One advantage of automatic polyp size estimation or any other feature of interest is standardization. For example, current methods determine polyp size based on manual user input (e.g., points defining the length of the polyp) or according to manual estimation, and the input of one or more different users may provide or cause different size estimates for the same polyp.

[0153] In some embodiments, the cleaning levels or other data may be displayed per image, per segment, and / or per procedure.

[0154] In some embodiments, an option may be provided to present the most suspicious images first in each segment (eg, in a matrix view) or by another method (eg, by asterisks).

[0155] In some embodiments, the representation of the GIT portion of interest in the map may also indicate secondary images, not just primary images. Thus, for example, the extent of disease and lesions may also be indicated.

[0156] In some embodiments, in order to provide personalization of the study, the parameters of the study can be adjusted or determined according to the patient (e.g., how many images will be included in the first level / second level; how many images of the study will be selected to represent a certain segment of the GIT portion being studied; thresholds for selecting images to be included in the study, etc.). In some embodiments, this adjustment can be performed manually, such as by the user executing the procedure or the referring physician entering data. In some embodiments, such parameter selection can be automatic, or at least a portion thereof can be automatic (e.g., automatic adjustment of some parameters to be determined), such as based on information or details of the patient (including the patient's medical history). In automatic adjustment, natural language processing (NLP) algorithms or machine learning-based tools such as the ComprehendMedical system provided by Amazon can be utilized. Such tools can make free text interoperable and searchable. The system can then use the patient's medical background to adjust the study. For example, if the system identifies in the patient's (eg, case) free text that the patient has a family history of polyps, the system may modify the study or presentation to have more images (eg, to increase the sensitivity of the study) compared to a healthy patient without a history of polyps.

[0157] In some embodiments, speech-to-text or automatic speech recognition systems (e.g., Amazon's "Transcribe") may be integrated to significantly reduce free text reporting time, for example by allowing the user to begin "writing" a report using speech-to-text while reviewing an image. The use of speech-to-text may reduce the need for clicks or other manual user input. For example, a user may speak and take an action based on "show me a photo of an ulcer in a collection"; thereby allowing the system to take that action without clicks or manual user input (e.g., to display a collection of typical disease images that can be searched by key descriptors and used as a viewing aid to compare study images to reference images). Speech-to-text may allow a "chatbot" to assist the system GUI, such that a user can ask the system a question and have the system look in a "FAQ" or other repository to see if there are sufficiently similar questions and provide answers.

[0158] In some embodiments, the system may display transition segments in the GIT portion of interest. For example, as an alternative to a color bar, an image representing the transition segment from each side (e.g., stomach and colon to SB) may be presented or provided (and indicated in the map), for example, at the first level or default view of a study having a second level display including transition (between segments) images. In some embodiments, transition images (e.g., images indicating transitions between GI segments) may be displayed at each endpoint of the displayed map, respectively, after user input, such as after the user clicks on an icon indicating stomach or colon. Such anatomical transition images may include multiple images identified as being located near an anatomical transition (e.g., from SB to colon). One option for display of transition images may be to include one image for each transition in an image subset (e.g., first level images), which then represents the transition as the one or more corresponding images (e.g., second level images). The images in the image subset may then appear in the map outside the portion of interest (e.g., in both endpoints). Another option is to represent such images in a small color bar above the map at the two end points of the section of interest, or to receive a window or screen including a display of these images by having the user click on icons at the end points of the section (e.g., the SB icon and toilet icon in a colon study).

[0159] In some embodiments, the default view of a study may show a matrix display of a subset of images to allow the user to quickly review the primary images first. The user may then switch to another view of the study, such as a "carousel display" as described elsewhere. For example, clicking on an image in the matrix display will initiate display of other study views, including a carousel display of a subset of images, and wherein the clicked image is the center image of the carousel (e.g., a zoomed-in view). This may be most advantageous when the study includes relatively many images (e.g., several hundred images). In some embodiments, the matrix view may also include a display of secondary images and / or other additional information.

[0160] In some embodiments, machine learning, artificial intelligence, or AI may be used to automatically identify, localize, and size precancerous colon polyps and small bowel lesions or other features. Embodiments may use or execute modules that act as "detectors" for certain features or lesions such as NLH (nodular lymphoid hyperplasia), ulcers, vascular dilation, erosions, blood, lesions, or inflammatory strictures.

[0161] A report may be generated after the user review based on the user input during the review.

[0162] Fig.11 A flow chart showing a method according to an embodiment of the invention is shown. Fig.11 The operations in can be used as follows Figure 1The system shown is illustrated, but other systems may be used.

[0163] In operation 1100 , a subject or patient may swallow or ingest a swallowable capsule, for example, including one or more imagers or imaging devices (eg, digital imagers or cameras), a transmitter, and other units (such as a power source and antenna).

[0164] In operation 1110 , the capsule may traverse the GIT or a portion of the GIT of the subject (eg, by the subject's natural peristaltic motion) to transmit images to an external receiver (eg, via a radio transmitter).

[0165] The images transmitted by the capsule may be received by an external receiver (although other methods of transmitting images may be used), recorded, and processed in operation 1120. Various combinations of equipment may be used to receive and store these images.

[0166] In operation 1130, in one embodiment, a group of images of interest may be automatically identified or selected, for example, by one or more processors using, for example, a first selection method. Each image of interest may show a different automatically identified feature or lesion. Groups of images may be created, each group including one first level image of interest and a corresponding group of images that are not in the first level of interest but show the same features as one of the corresponding first level images of the group. Each first level image may be identified as the "most representative" image in the group.

[0167] In operation 1140, the first level images may be displayed as static (e.g., non-video) images in a primary or default view of a study default view on a monitor or display device (e.g., without any user input or command to initiate such display). The first level images may be a subset of images in the in vivo image stream. The user may review the first level images. The first level image subset may represent the received image stream.

[0168] In operation 1150, the user may provide and the system or GUI may receive input to display a second level image, wherein one or more additional images corresponding to the currently displayed image in the subset of images may be displayed on a display device or monitor. For example, an image in a group corresponding to the selected or clicked first level image may be displayed. The one or more additional images in the group may have been automatically selected from the received in-vivo image stream according to a second selection method. The second selection method may be based on a relationship between an image in the in-vivo image stream and the currently displayed image.

[0169] In operation 1160, in one embodiment, the user can change the display of the secondary images; for example, causing the images to be displayed as moving images, or to move forward and backward through the image set in a static manner, such as using a scroll wheel. The study can be displayed in, for example, a carousel mode, a matrix mode, or another mode.

[0170] A report may be generated or displayed to the user in operation 1170. The report may include the images selected by the user from the displayed images and possibly other information, such as annotations inserted by the user.

[0171] Other or different operations may be used.

[0172] Embodiments can improve the technology of capsule endoscopy procedures, particularly the technology of study generation, medical image display, and medical diagnosis, and thus can also improve the ability of medical professionals to diagnose information and provide better and more effective treatments. For example, medical image processing and display technology can be improved by providing a system that allows doctors to find, view, and analyze regions of interest in the GIT more quickly, and provides automatic analysis that can support and improve doctors' decision-making compared to existing medical technology. For example, in existing GIT imaging systems, doctors may be forced to view lengthy videos. In addition, automatic identification, localization, and / or sizing of features of interest are not provided.

[0173] Although several embodiments of the present disclosure have been shown in the above drawings, the present disclosure is not intended to be limited thereto, as the present disclosure is intended to be as broad as the art allows, and the specification is intended to be read equally broadly. Therefore, the above description should not be understood as limiting, but only as an illustration of a specific embodiment.

[0174] It should be clear that most of the principles, elements and features described above and / or shown in the accompanying drawings for the embodiments according to the present disclosure are not limited to the embodiments in combination with which these principles, elements and features are described / shown, and may, mutatis mutandis, be independently applied to each other or to any other embodiments according to the present disclosure in any combination deemed appropriate by a person skilled in the art.

Claims

1. A system for image processing, comprising: one or more processors; A non-transitory storage device having stored thereon instructions that, when executed by a processor, cause the processor to: receiving a subset of images automatically selected from a stream of in vivo images of at least a portion of interest of a GIT of a subject captured via a capsule endoscopy procedure, wherein the images in the subset of images are selected based on a first selection method; Displaying the automatically selected subset of images in a static manner on a display device; upon receiving user input, displaying additional information about a currently displayed image from the automatically selected subset of images; and A report is generated that includes images selected by the user from among the displayed images. 2 . The system of claim 1 , wherein the additional information comprises one or more additional images automatically selected from the in-vivo image stream according to a second selection method.

3. The system of claim 1, wherein the displaying of the subset of images comprises simultaneously displaying a plurality of images in the subset, wherein the plurality of images are captured by a single imaging device of a capsule device used in the capsule endoscopy procedure. 4 . The system of claim 2 , wherein the one or more additional images include at least one image in the image stream that is not included in the selected subset of images.

5. The system of claim 1, wherein the one or more additional images are displayed in the same window as the corresponding currently displayed image.

6. A system according to claim 1, wherein when the instructions are executed by the processor, the processor displays a map, the map representing at least the portion of interest of the subject's GIT and corresponding to at least the selected image subset, wherein the map includes localized image indications, and the localized image indications represent the images in the image subset according to their estimated positions along at least the portion of interest of the subject's GIT.

7. The system of claim 1, wherein: dividing the at least the portion of interest of the subject's GIT into a plurality of segments; and localizing the images in the subset of images along at least the portion of interest of the GIT of the subject according to the segment, and The method also includes displaying, using the one or more hardware processors, an indication of the segment at which the currently displayed image in the subset of images is located.

8. The system of claim 1, wherein the instructions, when executed by the processor, cause the processor to display an indication of the level of scrubbing of an image in the captured in-vivo image stream.

9. The system of claim 1, wherein according to the first selection method, images identified as including features of interest are selected.

10. A system for generating a study of a capsule endoscopy procedure for review by a user, wherein the capsule endoscopy procedure includes capturing an in vivo image stream of at least a portion of interest of a subject's GIT, the system comprising: one or more processors; as well as A non-transitory storage device having stored thereon instructions that, when executed by a processor, cause the processor to: selecting a subset of images from the captured in-vivo image stream according to a first selection method, wherein the selected subset of images represents the captured in-vivo image stream; for each image in at least a portion of the subset of images, selecting one or more corresponding additional images from the in-vivo image stream according to a second selection method; as well as The study of the capsule endoscopy procedure is generated, wherein the study includes the selected subset of images and additional information, and wherein the additional information includes the selected one or more additional images corresponding to each image in at least a portion of the subset of images.

11. The system of claim 10, wherein the instructions, when executed by the processor, cause the processor to display the study on a display device for review by a user.

12. The system of claim 11, wherein the display of the study comprises: displaying the selected subset of images on a display device; Upon user input, the one or more additional images corresponding to a currently displayed image in the subset of images are displayed on the display device.

13. A system according to claim 10, wherein in addition to the additional one or more images corresponding to the images in the image subset, the generation of the study also includes generating additional information, the additional information being selected from: a division of at least the portion of interest of the GIT into multiple segments; the position of images in the image stream along at least the portion of interest of the GIT; a mapping map representing at least the portion of interest of the GIT, including an indication of the images in the image subset along at least the portion of interest of the GIT; a plurality of images selected from the image subset according to a third selection method; an indication of a feature of interest in an image; an indication of a cleaning level; an indication of an estimated size of a feature of interest; an indication of a range of a feature of interest; selection of images in the in vivo image stream representing transitions between anatomical segments of at least the portion of interest of the GIT; and combinations thereof.

14. A system according to claim 10, wherein the relationship on which the second selection method is based is selected from: images identified as including at least a portion of the same feature, images identified as including at least a portion of the same type of feature, images captured at a similar time, images localized adjacently along the at least portion of the subject's GIT, and combinations thereof.

15. The system of claim 10, wherein the one or more additional images include at least one image in the image stream that is not included in the selected subset of images.

16. The system of claim 10, wherein the one or more additional images include only images in the image stream that are not included in the selected subset of images.

17. The system of claim 10, wherein the relationship based on which the second selection method is performed produces different corresponding one or more additional images for different images in the subset of images.

18. The system of claim 10, wherein at least the first selection method is based on deep learning techniques.

19. The system of claim 10, wherein according to the first selection method, images identified as including features of interest are selected.

20. The system of claim 10, wherein the first selection method and the second selection method each include one or more selection rules.

Images