A technology for dynamically prioritizing, systematizing, and displaying cells and tissues in order of their diagnostic significance.

Abstract

A system and method are provided for rendering interactive displays for use in reviewing specimen slides by a pathologist. The system renders a series of slides containing all the information from a digitized slide and presents tiles in a linear manner in order of diagnostic significance. Additionally, tiles containing cells with high degrees of abnormality are provided with images of adjacent cells / tissues to provide the reviewing pathologist with context that is beneficial to the determination of abnormality.

Description

Technical Field

[0001] Cross-Reference to Related Applications The present invention claims the benefit of U.S. Provisional Patent Application No. 63 / 146,956, filed on February 8, 2021, the content of which is incorporated herein by reference.

[0002] The present invention generally relates to the field of identifying, classifying and presenting potentially harmful tissues and cells. More specifically, the present invention relates to an improved computer system and method that analyzes digital microscopic images of tissues and cells on a slide, dynamically identifies, prioritizes and highlights specific regions on the slide that are of most interest (e.g., potentially cancerous), and enables medical professionals to make diagnosis, prognostic evaluation, treatment decisions and perform treatment with higher efficiency and accuracy than prior art.

Background Art

[0003] Reviewing microscope slides containing cells is a time-consuming task that is prone to human error. For example, cytological specimens of tissues and cells are routinely obtained in various diagnostic scenarios and analyzed manually by pathologists. In other situations, a non-laceration brush biopsy sample of tissue is obtained using a brush stiff enough to penetrate the tissue to obtain an adequate sample of the tissue and cells. In the process of obtaining these full-thickness tissue specimens, single cells, cell clusters and tissue fragments are collected and transferred to a microscope slide, and the pathologist will examine the microscope slide for abnormal cells (e.g., dysplasia). This is time-consuming, as explained below, and can lead to missed diagnoses and inappropriate, non-ideal or inadequate treatment.

[0004] Other types of cell specimens examined by pathologists include cell block slide preparations. Cell blocks are prepared by suspending cells and tissues in a liquid culture medium in a tube and then centrifuging the preparation to obtain a precipitate. After centrifugation, the solution is discarded, and the precipitate is embedded in paraffin or a similar medium to form a pellet. Once formed, the pellet is cut into continuous slices, so that each slice represents a cross-section of the cells / tissues in a given plane within the pellet. The slices are then mounted on microscope slides for examination under a microscope by the pathologist.

[0005] Regardless of the biopsy technique used to collect tissue and cell samples, biopsy specimens typically contain hundreds of thousands of cells randomly spread across the surface of the slide. Cell preparations are usually heterogeneous, with areas on the slide containing high cell density, other areas containing low cell density, and other areas containing no cells at all. To screen and detect disease, pathologists must examine every cell, cell cluster, and tissue fragment on the slide. This is extremely difficult, as it requires manually traversing the entire surface of the slide and examining cells and tissues using a microscope.

[0006] The problems associated with this manual approach are diverse.

[0007] Firstly, pathologists don't know where to begin looking on a slide for cells that suggest disease. Often, pathologists need to select an area of ​​the slide that shows no signs of disease and then scan through the slide to find cells that suggest disease. As a result, even if the pathologist initially knows which areas of the slide are problematic and which do not show signs of disease, the manual approach is more time-consuming than other methods.

[0008] Secondly, pathologists need to move sector by sector across the slide to examine each area of ​​the cells. This requires pathologists to slowly and carefully traverse a large surface area on the slide to search for abnormal cells and manually record which cells are being examined, where they are located on the slide, and which cells are abnormal. This manual process is cumbersome and often leads to human error, such as pathologists overlooking target cells or tissues (e.g., malformations, metaplasia). During slide screening, pathologists first review the slides at low resolution to identify areas of interest. Once areas requiring further analysis are identified, pathologists switch to high magnification microscopes to examine cells and tissues in detail. However, pathologists may miss small or isolated malignant cells during this process, in which case evidence of disease or abnormality may be missed.

[0009] Thirdly, when a pathologist identifies an isolated cell or tissue (such as a malignant cell) on a slide, it is extremely helpful for the pathologist to view information about cells and tissues in the area adjacent to the isolated cell or tissue in question. In this regard, in many cases, the isolated cell or tissue in question is not necessarily an indicator of disease. On the other hand, if the adjacent cells and tissues are also the subject of diagnosis (for example, if they are also abnormal or malignant), the likelihood of disease increases even further. Conversely, if the adjacent cells and tissues are healthy, the likelihood of disease is low.

[0010] Computer-based systems exist, such as those provided by Indica Labs, that present target tiles associated with the determination or tagging of specific cells or tissues as being subject to diagnosis. However, as the inventors of this application recognize, such computer-based systems do not utilize computer technology for dynamic resizing of tiles, for example, resizing based on the identification of multiple individual diagnostic targets in adjacent areas of a slide. Rather, as the inventors of this invention recognize, they simply present individual tiles associated with the determination or tagging of each diagnostic target, or simply present the entire slide.

[0011] Unfortunately, as the inventors of this invention recognize, when examining a target tile generated by a well-known computer system, the pathologist re-examines the entire given tile or slide, making it difficult for the pathologist to access information specifically located in adjacent areas of the cell and tissue. As a result, in order to determine the presence or absence of disease, the pathologist must spend considerable time examining the areas adjacent to the cells and tissue of interest on the physical (or digital) specimen slide.

[0012] The aforementioned process is extremely time-consuming, and in many cases, pathologists spend a long time examining adjacent areas only to discover that the target cells or tissue are not present in those areas (e.g., there are no abnormalities).

[0013] As the inventors of this invention recognize, if pathologists were given immediate access to information about cells and tissues in areas adjacent to the target cell or tissue area, it would be a significant improvement in diagnostic advantages and effectiveness, enabling them to rule out or diagnose diseases with greater efficiency. Unfortunately, no such well-known system or method exists.

[0014] Fourth, it is extremely difficult for pathologists to accurately record areas of slides that have already been reviewed, areas that have not yet been reviewed, and areas where abnormal cells are located. Due to these difficulties, pathologists often review the same area of ​​a slide multiple times, which can lead to increased processing time, wasted resources, and further human error.

[0015] Finally, because the target cells and tissue fragments (e.g., malformed tissue) are randomly distributed throughout various areas of the slide, reviewing such slides is generally done in an unsystematic and unstructured manner. As a result, pathologists may spend valuable time pursuing abnormal cells and reviewing normal or non-target cells, sometimes even at the expense of missing important disease cells.

[0016] For these reasons, the manual methods used by pathologists to re-examine and analyze cells and tissues on slides are unsystematic, time-consuming, and often lead to missed diagnoses of diseases (such as cancer).

[0017] Conventional systems have been developed for the purpose of performing computer-aided analysis of specimen slides, and they address some, but not all, of the problems associated with manual methods used by pathologists. Such conventional systems include computer and software systems (e.g., U.S. Patent No. 6,327,377) developed by the applicant and its predecessors. These systems include classifiers and specially trained neural network computers configured to identify the most abnormal-looking cells on specimen slides.

[0018] However, such systems only identify each individual isolated cell and do not show the tissue regions adjacent to the identified individual cells. In diagnosing abnormalities related to tissues of the gastrointestinal tract (and other body parts), it is extremely difficult to make a diagnosis based on a single abnormal cell. Therefore, it is very important for pathologists to re-examine the tissue adjacent to the abnormal cell and assess whether other abnormal cells are present in the same region. If a complete cluster or grouping of adjacent cells is presented to the pathologist, the pathologist is provided with the context necessary to make a diagnosis. Therefore, having information about adjacent cells increases the credibility and accuracy of the pathologist when making a diagnosis. Unfortunately, the inventors of this invention recognized that the prior art cannot selectively provide useful information about adjacent cells along with the cell under consideration, based on the level of the adjacent cells being diagnosed. Furthermore, the inventors of this invention recognized that the prior art cannot employ computer techniques such as dynamic resizing for any such purpose.

[0019] Furthermore, cells that test positive for abnormality tend to group into clusters. Conventional systems, on the other hand, typically show only one section or part of such clusters, rather than all of them. In particular, the inventors of this invention recognized that current computer-based systems cannot accommodate the dynamic resizing of presented images related to diagnostically significant cells or tissues based on features (such as clustering) of neighboring cells or tissues. As a result, pathologists may not always have the advantage of the complete context of clusters, thereby compromising diagnostic accuracy. Alternatively, pathologists may determine that the presented partial information is not conclusive and be forced to manually search for the complete cluster under a microscope to find the missing information. This is a time-wasting process, and if the pathologist is unable to find the remaining cluster, such failure can lead to a misdiagnosis.

[0020] While conventional systems are useful, they do not provide pathologists with reasonable confidence that all abnormal cells are displayed, nor that complete clusters or groupings of cells are displayed. This is because such conventional systems are not configured to display all abnormal cells within a dynamically resizable area of ​​the slide based on a reliable computer assessment, but rather merely present tiles based on individual and non-aggregated examinations of subsets of such cells. Furthermore, conventional systems do not provide pathologists with a way to ensure they have reviewed all areas and cells of interest on the slide. Conventional systems do not identify and systematize areas of specimens of higher-order diagnostic interest, nor do they prioritize systematizing those areas in a smart gallery so that pathologists can focus their analysis on such areas of higher-order interest. These shortcomings can also lead to missed diagnoses or incomplete diagnoses and misdiagnoses.

[0021] As can be seen from the above explanation, the methods used by pathologists (both with and without conventional computer systems) involve extensive manual review of specimen slides. Manual review is very time-consuming, prone to errors, and can lead to missed diagnoses and misdiagnoses, so reducing the degree of manual review required to make a diagnosis is advantageous. When dealing with the detection of diseases (such as cancer), the goal is always to minimize errors and maximize accuracy. Another goal is to maximize the efficiency of reviewing slides without sacrificing accuracy. Conventional methods have been somewhat successful in meeting these objectives, but there has been a long-standing need for systems that improve the accuracy and efficiency of conventional methods and for systems that replace unordered manual review of slides.

[0022] The spatial and temporal inefficiencies in the process of identifying malignant tumors, pre-malignant tumors, and disease states from sample slides, as well as the challenges of inefficient work redundancy, misdiagnosis, and accuracy, are resolved by this disclosure. [Overview of the Initiative]

[0023] Embodiments of the present invention provide a computer system that (1) automatically scans and analyzes cells and tissues on a specimen slide, (2) classifies such cells and tissues according to their diagnostic importance, (3) employs computer-based dynamic resizing of a digital slide that incorporates multiple combined target regions, such as in embodiments where each target region has a size specified by a computer-implemented scoring methodology, (4) creates a gallery of image tiles corresponding to the entire specimen slide, and (5) presents the image tiles to a pathologist in a systematic order of diagnostic importance.

[0024] The diagnostic importance of abnormal cells directly relates to the presence of individual abnormal cells and whether such abnormal cells are grouped together physically or spatially. In this regard, cells that test positive for abnormality usually do not appear in isolation, but rather, they are more commonly grouped with other abnormal cells (e.g., in adjacent areas). Therefore, the presence of groups of abnormal cells is diagnostically important and, if identified, provides pathologists with a certain degree of confidence that the “positive” diagnosis is accurate.

[0025] In embodiments of the present invention, the system dynamically identifies diagnostically important cell groups (e.g., abnormal cell groups) and systematizes such diagnostically important groups on a display (e.g., together as part of dynamically resized tiles) so that when a pathologist begins analyzing a slide, they can immediately revisit such groups. In this regard, according to embodiments of the present invention, the system presents such diagnostically important groups to the pathologist in a manner that enables such groups to be revisited for the first time. For example, in embodiments, the system generates image tiles (e.g., large image tiles) of the diagnostically important cells that are more prominent than non-diagnostically significant cells, such as isolated cells (which would be displayed as smaller tiles).

[0026] In embodiments of the present invention, larger tiles (those of the greatest diagnostic significance) may be displayed first in a linear display because they contain the most important information needed to make a diagnosis (e.g., the cell population and tissue of interest). Isolated abnormal cells and / or ungrouped abnormal cells are displayed in smaller tiles and shown after the larger tiles because they contain information of less diagnostic significance. The size of the image correlates with the number of positive events in the same vicinity. The more events in the same vicinity, the larger the image, and the larger the image size, the higher the reliability. What constitutes the same vicinity can be determined by the type or cluster of cells being examined. Generally, in a sample, the distance between cells is often much smaller than the distance between cell clusters. In embodiments, the proximity distance is determined, where the distance is greater than the distance between cells but smaller than the distance between clusters. In embodiments, a distance of 5.5 to 6.5 times the distance between cells, for example 6 times, may be appropriate for GI or esophageal specimens. For example, if the distance between positive events is less than 6 times the distance between cells, those two events can be assumed to be from the same cluster. Other body parts may have different distances from each other. An example of proximity is shown in Figure 18.

[0027] In an embodiment of the present invention, the system analyzes digitized specimen slides stored in a digital memory, and assigns a score to each region of the digitized specimen based on, for example, the degree of abnormality. That is, in an exemplary embodiment, along a continuum of scores, specimen regions considered to be highly abnormal are assigned a high score, and normal specimen regions are assigned a low score. In an exemplary embodiment, the system uses the score to specify a region of interest, whereby a higher score results in a larger region of interest, and a lower score results in a smaller (e.g., smaller) region of interest. A region of interest is an image segment or image region adjacent to (e.g., surrounding) a cell or tissue to be diagnosed. In an embodiment of the present invention, these regions of interest determined by the score are used to determine how much image area surrounding the target cell or tissue is included in an image tile and provided to a pathologist as a unit. In embodiments, a threshold may be applied; for example, a region of interest may be specified only when the score exceeds a predetermined threshold, such as 0.5 or 0.8.

[0028] In embodiments of the present invention, once a target area (e.g., a geometrically bounded shape corresponding to a group of diagnostically important cells or an area of ​​a digital slide) is identified using the aforementioned scoring system, the system dynamically generates a series of tiles (e.g., corresponding to at least initially such a geometrically bounded shape), each of which may contain images of cells, cell clusters, and / or tissue fragments in any combination. In embodiments, the tiles are presented on the display in a linear manner according to their diagnostic importance (e.g., from the tile with the most diagnostically important cells to the tile with the least diagnostic importance), and are systematized and ordered (e.g., through presentation or emphasis from left to right or top to bottom, or in chronological order). Ranking may be based on a neural network score, which is trained to recognize abnormalities and typically assigns scores from 0 to 1, with 1 being the most confident that it is abnormal. According to embodiments of the present invention, the entire slide is presented to the pathologist in an imaged, condensed, and ordered manner. In the embodiment, all specimen information, i.e., all specimen information determined to be diagnostically relevant through computer processing, is presented, for example, in a series of tiles on an electronic display, so that the system allows the pathologist to review the tiles without having to review specimen slides individually. Even if a high score is assigned, there is a finite and small possibility that it may be a (probabilistic) false positive event. To address this, identifying adjacent positive events improves the problem. When multiple positive events are present in the same vicinity, the probability that their combined event is a false positive is low. This reflects how positive cells inherently arise; they are rare, but when they do appear, they generally appear in clusters. This allows the pathologist to review specimen slides more quickly and with greater accuracy while reducing the overall possibility of overlooking diagnostically important potentially harmful cells and tissues.

[0029] According to one embodiment, in order to create a series of tiles, the microscope slides are first digitized and then morphologically classified by a computer. As part of such automated classification, the cells present on the slides are scored by the computer according to a scale of their abnormality. The system uses the computer-generated scores to determine how much of the image area surrounding the target cells is presented to the pathologist. For example, in an embodiment of the present invention, the higher the rank of a cell for abnormality, the larger the size of the image area adjacent to the high-ranking cells included in the tile. An r-tree data structuring method may be employed. In this regard, when a pathologist detects abnormal cells, it is important to examine the tissue and cells adjacent to the abnormal cells in order to provide an accurate diagnosis. This is because, when diagnosing tissue (e.g., of the gastrointestinal tract or esophagus), one abnormal cell does not always provide enough information to make an accurate diagnosis. Rather, it is often necessary to examine the surrounding tissue as well. If the adjacent cells are also abnormal, or very close to being abnormal, the pathologist can make a more efficient (e.g., faster) and more accurate diagnosis than if the adjacent cells were not included in the tile.

[0030] Therefore, in an embodiment of the present invention, the tile that is the object of most diagnosis (and, for example, tiles that are physically grouped together because the computer-implemented approach groups a plurality of contact areas together, as further described herein) will be larger than tiles that are not diagnostically relevant and / or have isolated features, since the larger tile will contain the target cells and surrounding areas that have been determined for diagnostic relevance of the target cells and / or the associated one or more adjacent positive regions. When displayed chronologically or positionally on a pathologist's electronic display, larger tiles are preferentially positioned first in a sequence of tiles because they may be of the highest importance, while the smallest tiles are positioned last in the sequence of tiles because they may be isolated and of the least importance. As a result, areas of the slide that are most likely to be significant can be reviewed first by the pathologist, or otherwise presented to the pathologist in a manner that suggests that the pathologist should devote their maximum attention to these areas.

[0031] Embodiments of the present invention also remove "dead areas" on a slide (e.g., areas that have no cells or areas that have cells with less than a predetermined degree of apparent diagnostic relevance), thereby greatly reducing the two-dimensional surface area on an electronic display that needs to be reviewed by a pathologist, which in turn reduces the time that would otherwise be required to review the slide.

[0032] Accordingly, it is an object of the present invention to render a specimen preparation as a presentation of a sequence of tiles or other ordered tiles such that a pathologist does not need to traverse an entire conventional slide, thereby reducing both the time required for traversal by the pathologist and / or the two-dimensional slide surface area to be traversed.

[0033] Another object of the present invention is to present tiles to a pathologist in a priority order according to their diagnostic significance.

[0034] Another object of the present invention is to determine the extent to which supplementary image context should be provided to the pathologist.

[0035] Another object of the present invention is to create a linear display or other ordered display from a whole slide image.

[0036] Another object of the present invention is to provide a diagnostic display platform that tracks a user's revisit activity.

[0037] Another object of the present invention is to provide a diagnostic display that allows the user to select specimen regions that are positive and / or negative for abnormalities and render one of a variety of display formats that compare and display them.

[0038] Furthermore, the present invention addresses the problems of spatial and temporal inefficiencies in proceeding from sample slides to the identification of malignant tumors, pre-malignant tumors, and disease conditions, as well as the challenges of redundant work, misdiagnosis, and inaccuracy, with practical and actionable solutions. The processes and systems described herein modify the display in a manner that brings about dynamic interaction with image data provided from the initial sample, thereby reducing the spatial and temporal inefficiencies in proceeding from sample slides to the identification of malignant tumors, pre-malignant tumors, and disease conditions, as well as the challenges of the spread of redundant work, misdiagnosis, and inaccuracy. Without the present invention, the quantification and grading of samples would be slower, less quantitative, and qualitatively different.

[0039] The present invention satisfies the aforementioned objectives in an improved computer system and method that, in particular, (1) analyzes digital microscopic images of cells and tissues taken from a biopsy and arranged on a slide, (2) based on a score that rates the potential malignancy of the cells and tissues, (a) identifies and highlights areas on the slide that contain multiple cells and / or tissues that are most likely to be the most dangerous (e.g., cancerous) and therefore require initial and very careful re-examination by a pathologist, (b) identifies and dehighlights areas on the slide where cells are isolated (and therefore likely to be benign), thereby dynamically outputting digitized image tiles that indicate to the pathologist that such areas are less important and can be quickly re-examined, (3) tracks areas on the slide that have been re-examined by the pathologist (which would be extremely difficult to do without this software), and (4) enables the pathologist to annotate the slides.

[0040] This system and method increases the accuracy of disease detection and diagnosis while reducing the potential for errors associated with the conventional methods and systems described above. The system of the present invention also increases the efficiency with which slides can be reviewed and diagnosed by pathologists without sacrificing the accuracy of diagnostic findings, compared to conventional methods and systems.

[0041] A method for treating cancer in a subject comprises the steps of: receiving a diagnosis or identification of cancer in a specimen from the subject, wherein the cancer is diagnosed or identified in the specimen using the method described herein; and administering a predetermined amount of therapy to the subject for the diagnosed or identified cancer.

[0042] A method for reducing the likelihood of cancer or treating a malformation in a subject comprises the steps of: diagnosing or identifying a malformation in a specimen from the subject, wherein the malformation is diagnosed or identified in the specimen using the method described herein; and administering a predetermined amount of therapy to the subject to reduce the likelihood of cancer or treat the malformation.

[0043] In this embodiment, the therapy is anti-cancer small molecule therapy, anti-cancer radiotherapy, anti-cancer chemotherapy, anti-cancer surgery, or anti-cancer immunotherapy.

[0044] In the embodiment, the therapy is small molecule therapy, radiotherapy, chemotherapy, surgery, or immunotherapy for malformations.

[0045] A method for treating cancer in a subject comprises the steps of: receiving a diagnosis or identification of cancer in a specimen from the subject, wherein the cancer is diagnosed or identified using at least to some extent the system or process described herein; and administering a predetermined amount of therapy to the subject for the diagnosed or identified cancer. In one embodiment, the administering healthcare provider is provided with a complete or partial diagnosis or identification of cancer in a specimen from the subject by identification or diagnosis using the processing or method described herein.

[0046] A method for reducing the likelihood of cancer or treating a malformation in a subject comprises the steps of: receiving a diagnosis or identification of a malformation in a specimen from the subject, wherein the malformation is diagnosed or identified in the specimen using the system described herein as part of the diagnosis or identification; and administering a predetermined amount of therapy to the subject to reduce the likelihood of cancer or treat the malformation.

[0047] In this embodiment, the therapy is anti-cancer small molecule therapy, anti-cancer radiotherapy, anti-cancer chemotherapy, anti-cancer surgery, or anti-cancer immunotherapy.

[0048] In the embodiments, the therapy is small molecule therapy, radiotherapy, chemotherapy, proton therapy, surgery, or immunotherapy for malformations. In the embodiments, surgery involves inserting a stent. In the embodiments, surgery involves removing malignant tissue. In the embodiments, the cancer is esophageal cancer or oral cancer. In the embodiments, the esophageal cancer is adenocarcinoma or squamous cell carcinoma. In the embodiments, the cancer is ESCC (esophageal squamous cell carcinoma). In the embodiments, the treatment includes pembrolizumab. In the embodiments, the treatment includes nivolumab.

[0049] The features and advantages of this disclosure will be better understood by referring to the following detailed description in conjunction with the attached drawings. [Brief explanation of the drawing]

[0050] [Figure 1] Figure 1 is a schematic diagram of a system configuration according to an exemplary embodiment of the present invention. [Figure 2] Figure 2 is a flowchart showing exemplary steps for dynamically creating tiles according to an exemplary embodiment of the present invention. [Figure 3A] Figure 3A is a schematic diagram of a digitized slide region showing three exemplary overlapping envelopes generated by the system, according to an exemplary embodiment of the present invention. [Figure 3B] Figure 3B is a schematic diagram of the frame that is dynamically generated by the system and captures the overlapping envelopes in Figure 3A. [Figure 3C] Figure 3C is a schematic diagram of the resulting tile formed according to Figures 3A and 3B. [Figure 4] Figure 4 is a schematic diagram of a display screen showing a series of tiles on one side of a split screen and a reduced view of the entire digitized slide on the second side of the split screen, according to an exemplary embodiment of the present invention. [Figure 5] Figure 5 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen and an enlarged view of the area on the entire digitized slide from which the tiles are derived, according to an exemplary embodiment of the present invention. [Figure 6] Figure 6 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen and two tagged areas in a thumbnail of the entire digitized slide shown on the second side of the split screen, according to an exemplary embodiment of the present invention. [Figure 7] Figure 7 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen according to an embodiment of the present invention, where the tiles are generally smaller than the tiles shown in Figure 6. [Figure 8]Figure 8 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen according to an embodiment of the present invention, where the tiles are generally smaller than the tiles shown in Figure 7. [Figure 9] Figure 9 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen according to an embodiment of the present invention, where the tiles are generally smaller than the tiles shown in Figure 8. [Figure 10] Figure 10 is a schematic diagram of a display screen showing a series of tiles on the first side of a split screen according to an embodiment of the present invention, where the tiles are generally smaller than the tiles shown in Figure 9. [Figure 11] Figure 11 is a schematic diagram of a display screen showing a browsing pane in which tissue regions positively stained by immunohistochemical staining are tagged by the system, according to an embodiment of the present invention. [Figure 12] Figure 12 is a schematic diagram of a display screen showing a browsing pane in which immunohistochemically negative tissue regions are tagged by the system, according to an embodiment of the present invention. [Figure 13] Figure 13 is a schematic diagram of a display screen showing a browsing pane in which tissue regions that are immunohistochemically positive and negative, respectively, are tagged by the system, according to an embodiment of the present invention. [Figure 14] Figure 14 is a schematic diagram of a display screen showing an upper viewing pane that displays the positive sample area and a lower viewing pane that displays the negative sample area, according to an embodiment of the present invention. [Figure 15] Figure 15 is a schematic diagram of a display screen showing a browsing pane that provides meandering search features according to an embodiment of the present invention. [Figure 16] Figure 16 is a schematic diagram of a display screen showing a browsing pane that provides a visual tracking system for tracking slide areas viewed by a user, according to an embodiment of the present invention. [Figure 17] Figure 17 is a schematic diagram of each of the two sides of a split screen configured to move in tandem according to an embodiment of the present invention, which allows for simultaneous viewing of corresponding juxtaposed views of different staining or other treatments of similarly configured samples, for example. [Figure 18] Figure 18 shows a non-restrictive example of proximity determination and subsequent selection of displayed tiles. The shaded rectangular area represents tiles that are displayed as having high confidence. [Modes for carrying out the invention]

[0051] Embodiments of the present invention will now be described with reference to the drawings shown above. However, the drawings and description herein are not intended to limit the scope of the present invention. It will be understood that various modifications of this description of the present invention are possible without departing from the spirit of the invention. Furthermore, features described herein may be omitted, additional features may be included, and / or features described herein may be combined in a manner different from the specific combination described herein, all without departing from the spirit of the invention.

[0052] As described above, embodiments of the present invention relate to digitizing cell preparations, analyzing the digitized images for diagnostically significant information, and packaging the entire digitized slide into a series of tiles organized according to their diagnostic significance. The analysis includes detecting morphological and other biological features of cells and tissues using either an algorithmic classifier, a neural network computer, or both. For example, such classifiers and computers may be used to identify cells with abnormal nucleus-to-cytoplasmic ratios that tend to indicate a lack of cell maturation and associated malignancy. In embodiments, the size of the images provided as input and / or as images in the training set of the neural network classifier may be 21×21 or 50×50 pixels or similar, and in embodiments, may be sized to reflect the width of two, three, or four cells in each direction. The system then identifies a diagnostically significant group of cells or tissues. Tiles of various sizes are then dynamically created, containing the most diagnostically significant (e.g., suspicious) cell groups.

[0053] Typically, in the preparation of cytological specimens for pathology, clinicians transfer and mount cells and / or tissues onto glass microscope slides. The slides are then sent to the laboratory for further processing and medical diagnosis. Further processing may include staining the slides to enhance the contrast of the sample (or specific features of the sample) when viewed under a microscope, or highlighting these proteins by using antibodies that bind to specific proteins. Such stains may include, for example, Feulgen, Papanicolaou, hematoxylin eosin (H&E), Alcian blue, and immunohistochemical stains such as cdx2, muc2, and p53, to name a few. Laboratory technicians may also apply coverslips and labels to the slides.

[0054] Some cell / tissue preparations are relatively flat and occupy a two-dimensional plane. On the other hand, it is understood that other samples may involve three-dimensionality. For example, transepithelial brush biopsy instruments, which use a stiff brush to sample epithelial tissue, often extract cells, cell clusters, and tissue fragments. The resulting specimens have a unique thickness, and the biological material spread across the slide can be essentially three-dimensional in at least some areas. Such thick, three-dimensional specimens are produced, for example, by using the applicant's WATS3D sampling instrument, which is used to sample epithelial tissue from a patient's esophagus. In embodiments, the sample is acquired by broad-area transepithelial sampling. In embodiments, the sample is acquired by a brush or spatula and placed in a container such as a bottle containing a liquid fixative. In embodiments, the bristles containing the portion of the brush instrument used to acquire the biological sample are cut from the rest of the brush instrument, and the bristles containing the portion with the biological sample are placed in place in the bottle containing the liquid fixative. The liquid containing the sample is sent to a laboratory where the bottle is opened and placed on an instrument for preparing sample slides.

[0055] Embodiments of the present invention relate to scanning either a conventional flat preparation or a preparation that has thickness and is three-dimensional in its features.

[0056] In embodiments of the present invention, as shown in Figure 1, for example, the system configuration 20 according to an embodiment of the present invention includes an optical system 21 for acquiring a set of images from slides. The optical system 21 may include a microscope (e.g., a high-magnification microscope), a slide positioning stage, and a camera. For example, a computer processor 24 controls the movement of the stage, for example, in the z direction and / or other directions, to acquire a sufficient number of images to constitute a complete slide image of the specimen slides. The system 20 further includes a storage device 26 for storing the acquired images. The storage device 26 may include a hard drive or an SSD (solid-state drive), or other types of memory devices (e.g., high-speed memory devices), and / or distributed storage such as cloud-based storage. The processor 24 (or a group of cooperating processors) processes the set of slide images to generate a digitized complete slide image. The processor 24 may be special image processing hardware (such as a graphics processing unit, i.e., a "GPU"), or may be used in combination with such hardware, to increase processing speed.

[0057] Those skilled in the art will understand that the optical system 21 may be configured to capture and store images after each movement of the stage, or alternatively, to continuously capture images at regular time or distance intervals while the stage moves at a constant or variable speed along the X or Y direction, for example. In embodiments, the interval is determined according to the average size of the target cell type, and may be, for example, 0.25, 0.5, 0.75, or 1 times the cell width. In embodiments, the interval is determined according to the image size captured at each position, and may be, for example, 0.125, 0.25, or 0.5 times the image width. A meandering path or other path may be employed to provide the slide image as a whole. For example, in such a path, movement may occur in both the X and Y directions, and results at regular, small intervals are provided to the neural network. In embodiments, one or a few higher scores (e.g., exceeding a threshold such as 0.5) with low neighboring scores along a pathway may be selectively discarded (e.g., as potentially false positives), while longer sequences of higher scores may be selectively used for envelope formation, for example, with the envelope formed around the central position of a series of higher scores or the central position of the highest score along a high-scoring pathway segment. For example, three, five, or seven consecutive higher scores may be required for envelope formation, with the second, third, fourth, or median score being used as the center of the envelope. In embodiments, the number of higher scores forming a pathway segment may be considered in determining the initial envelope size. For example, in embodiments, the envelope size may be proportional to the number of higher scores considered, or multiplied by a coefficient proportional to this number. Advantageously, this may result in the envelope size more closely matching or proportional to the size or other characteristics of the potential target cell or cell cluster or tissue.In embodiments, the discovery of locations with high scores may result in a change or addition to the path, for example, high scores (e.g., 0.9 or higher) may result in additional routing perpendicular to the preceding path at the high-scoring locations within a predetermined segment length (e.g., for intervals or scan counts of 3, 5, 7, 9, or 11). For example, focusing in the z-direction may be performed manually or automatically. In embodiments of the present invention, a continuous and ongoing approach may be faster when generating the entire digitized slide image. In embodiments, commercially available scanners such as those from APERIO or 3DHISTECH may be employed.

[0058] In embodiments of the present invention, the substantially three-dimensional specimen slide undergoes additional processing before being examined by a pathologist and / or a computer system. Specifically, the digital microscope image of the captured cell specimen may be further processed by an enhanced EDF system that produces an enhanced focused digital image that preserves diagnostically important objects and their spatial relationships to each other. This reduces artifacts and false images and increases the accuracy of the computer analysis system because the diagnostically important objects are displayed in focus on the computer. For example, in embodiments of the present invention, the specimen slide is digitized by an image processing technique described in U.S. Patent No. 8,199,997, the contents of which are fully incorporated herein by reference.

[0059] In embodiments, images of cells or tissues, such as esophageal cells or tissues, may be acquired using brush sampling techniques (e.g., using WATS or WATS-3D brushes or wide-area transepithelial sampling techniques and instruments). In embodiments, such approaches may be used to supply input to systemic analysis as described herein, to provide images to a library that may be used in a training set for a neural network as described herein, or both.

[0060] After the specimen slides are digitized, the digital images of the slides are analyzed by, for example, a computer processor (e.g., 24) to detect target cells, cell clusters, and tissue fragments. In embodiments of the present invention, the entire digitized slide image is first processed by an algorithmic classifier and then secondarily processed by a neural network computer. The algorithmic classifier locates a first group of candidate objects that may be cell nuclei within the digitized image. The digitized image may then be secondarily analyzed by a neural network computer. In embodiments, the algorithmic classifier or neural network may also calculate a score for the cells. For example, the numerical value of the score may correlate with the presence or apparent or probabilistic presence of morphological features, such as those associated with cancerous cells or malformed cells. Algorithmic classifications are described in general terms in U.S. Patents 5,939,278, 5,287,272, and 6,327,377, all of which are incorporated herein by reference. Suitable neural networks include convolutional neural networks.

[0061] In embodiments of the present invention, the improved algorithmic classifier identifies individual cells having morphological attributes consistent with the features of the subject to diagnosis, e.g., malformed or cancerous cells. In embodiments of the present invention, such classification may be performed by a neural network, implemented, for example, in combination with storage 26 on a processor 24, and trained to recognize the cells or cell clusters most likely to represent malformed or cancerous cells or cell clusters. In embodiments of the present invention, the neural network may be used to provide a secondary classification after the algorithmic classifier. In embodiments of the present invention, the neural network is provided as input one or more images identified by the algorithmic classifier in a first pass. The neural network may be trained with a training set comprising multiple images of cells or tissues tagged with scores corresponding to their morphological features. In exemplary embodiments, images may be obtained from or stored in a database having at least one set of library images. In exemplary embodiments, images may be tagged with an index indicating whether the cells or tissues have an abnormal condition, e.g., previously assigned by a pathologist (e.g., 1, associated with a positive diagnosis or evaluation, and 0, associated with a negative diagnosis or evaluation). In other embodiments, library images with a lower degree of diagnostic accuracy may be assigned a probability score, for example, 0.5. In other embodiments, the score may be assigned according to a range of scales, for example, 0 to 100 (representing probability as a percentage).

[0062] In embodiments, the training set data may include filtered image data filtered according to whether multiple pathologists each consent to the evaluation or whether they consent to the evaluation beyond a set percentage (e.g., 80%). Such evaluations may be provided by one or more pathologists, for example, through display and evaluation methodologies described elsewhere in this disclosure. In embodiments, these results may be electronically transmitted, for example, via a secure connection over the Internet, to a processor associated with one or more neural networks, such that the training data is consequently filterable.

[0063] In embodiments of the present invention, cells or tissues under examination may be selectively fed into multiple neural networks trained on image data. For example, in embodiments of the present invention, cells may be provided as input to a neural network trained (e.g., depending on the library) on sharp or relatively sharp images of normal and abnormal cells, thus labeled (e.g., to 0 or 1). In cases where the result determined by such a neural network is not sufficiently clear (e.g., result 0.3–0.7), the input may be selectively provided to a second neural network, which is trained on a training set including images of less sharp or edge-case cases of cells or tissues, tagged by a pathologist as to whether they still represent a normal or abnormal state (e.g., from a second library, or with respect to images whose evaluation varies among pathologists). This allows the evaluation to be performed through an image set that is more suitable for the input images of cells or tissues. In exemplary embodiments, further passes may be performed in an iterative manner, for example, depending on a further library or a portion thereof used as the training set for the neural network associated with scoring. In the embodiment, different neural networks or different training sets may be used, each trained in response to images relating to a specific cell type, a specific abnormal condition, or both.

[0064] In embodiments, for example, the neural network may be optimized as a result of evaluating whether the scoring output by the neural network matches, with respect to the relevant tile, a later evaluation by a pathologist or a later evaluation by a vote or average evaluation of several pathologists (for example, while being re-examined using such display functionality as described elsewhere in this application). For example, weights associated with the neural network that can determine, for example, the importance that a particular image in the training set may have, may be adjusted in the output score. In embodiments, this may occur according to a backpropagation process or may be based on a gradient function with respect to an error curve associated with the neural network or training data. In embodiments, such optimization may be performed on a convolutional neural network.

[0065] In embodiments, the neural network and / or training set may be specifically selected depending on the initial classification, e.g., the classification of an improved algorithmic classifier. For example, if the algorithmic classifier identifies cells suspected of being malformed, a neural network trained on cells with varying degrees of apparent diagnostic likelihood of malformation may be selectively employed; on the other hand, if the algorithmic classifier identifies cells suspected of being cancerous, a neural network trained on cells with varying degrees of apparent cancer may be selectively employed.

[0066] According to embodiments of the present invention, a neural network computer is trained to assign a score to each image in a digitized image depending on the likelihood that a given image is likely to represent a malformation or a morphological abnormality associated with cancer. Cells and objects that are likely to be abnormal are given high scores by the neural network computer, while objects that are less likely to be abnormal are given low scores. For example, in embodiments, scores may be assigned on a scale of 0 to 1 based on the apparent likelihood of having abnormal features. For example, in embodiments, a cell exhibiting the morphological features of a benign cell may receive a neural network score of 0.1 or near there, while a cell that closely resembles a known malignant image may be given a score of 0.9 or near there. Cells appearing anywhere between these limitations will be given a score of, for example, 0.5 or near there, or somewhere in the range of 0.3 to 0.7. Thus, in exemplary embodiments, the range of output of the trained neural network represents a continuum corresponding to the degree of certainty that a cell is abnormal. In exemplary embodiments, the processor 24 and / or storage 26 may track classifications for, for example, presentation to a pathologist via display, or for use in the computer-based dynamic resizing and presentation processing described herein.

[0067] In embodiments of the present invention, the system uses the output score of a neural network (either alone or in combination with additional information, such as classification) to determine the amount of information associated with a given target cell. For example, referring to the flowchart shown in Figure 2, first, in step 51, the microscope slide passes through an optical system that systematically images the slide. In events where the features of the slide to be digitized are three-dimensional, an extended depth of field (EDF) routine is started (step 51.1) to capture diagnostic information and preserve the spatial relationships between cells. In step 52, the digital image of the microscope slide is rendered by, for example, a processor 24 and stored in a digital storage medium such as a computer or storage 26. Next, in step 53, the computer performs image analysis to detect the target object for diagnosis. In step 54, the neural network computer assigns a score to each cell image based on the degree of cellular abnormality. In step 55, the system assigns an image region extending outward from each target cell, which is hereby called an "envelope".

[0068] In exemplary embodiments, the neural network score may be based on, in addition to, or substitute for the aforementioned features relating to the probability of abnormal conditions with respect to the size, shape, and / or optical density of the cell nucleus, or other such cell features that may be determined to be diagnostically relevant. In this regard, for example, according to the embodiment, cells exhibiting highly atypical nuclei may be given a higher score (and therefore a greater scope of influence based on their larger envelope). Thus, for example, in exemplary embodiments, cells with large nuclei (which may exhibit aneuploidy indicating malformations or cancer) may benefit from additional context for the pathologist making a diagnosis.

[0069] Figures 3A–3C illustrate an exemplary method for producing tiles from a slide according to an embodiment of the present invention, where the tile size is derived from a neural network score. Referring to Figure 3A, a digitized slide region 28 is shown. The slide region 28 contains an image of a tissue fragment (e.g., 32) formed from individual cells (e.g., 34, 38, 42). In the neural network scoring step, each cell in region 28 may be assigned a neural network score. For example, cell 34 may be ranked relatively high in terms of abnormality (e.g., 0.75). As a result, the system creates a corresponding large envelope 36 (e.g., calculated according to a fixed constant multiple of the score, e.g., 15 × 15 microns, e.g., 20 microns with respect to the edge of the boundary rectangle) with cell 34 at its exact center or approximately center. In embodiments, the size of the envelope may be determined by referring to the average size of a cell of the type being examined. For example, in one embodiment, if the cells being examined have an average diameter of a given number of microns (e.g., 5 to 10 microns for esophageal cells), the envelope dimensions may be determined based on a score that is a multiple of the average size, for example, 2, 3, 4, 5, or 6 times the average size. In one embodiment, the envelope dimensions may change exponentially depending on the score, for example, by multiplying the dimensional size by a constant base (e.g., 2, 3, 4) obtained by raising the score to a power, or by a constant base less than 1 obtained by raising the reciprocal of the score to a power (e.g., 0.5, 0.75), so that the dimensional size at a low score is significantly reduced compared to the dimensional size at a high score.

[0070] In other embodiments, if the score exceeds a predetermined threshold (e.g., 0.5 or 0.8), the envelope dimensions may be fixed (e.g., 2, 3, 4, 5, or 6 times the average size). The threshold may be dynamically modified based on whether the resulting false positive or false negative rate exceeds a predetermined ratio, for example, by later evaluation by a pathologist using a representation methodology such as the one described herein.

[0071] In contrast, cell 38, being morphologically closer to normal, receives a lower neural network score, and as a result, the system generates an envelope 40 around cell 38 that is smaller in size than envelope 36.

[0072] Cell 42 also exhibits a moderate level of abnormality, and thus the system generates an envelope 44 surrounding cell 42. Envelope 44 is larger than envelope 40 because it contains cells with higher credibility or specificity (i.e., higher neural network scores) than the cells in envelope 40. On the other hand, as explained, envelope 36 is larger than envelope 44 because its cells are more abnormal than the cells in envelope 44.

[0073] Each of the envelopes may be a quadrilateral or rectangular shape with left and right borders (e.g., 36a, 36b) and top and bottom borders (e.g., 36c, 36d). For example, in the case of a rectangular shape, in embodiments, the determined height may be multiplied by a fixed value so as to reach the width, constituting a known aspect ratio of the screen used by the pathologist (e.g., 16:9 following such a common aspect ratio). In exemplary embodiments, other shapes may be employed, such as a circle surrounding the cell from which the score was derived (e.g., a circle defined by a center point and a radius). As further shown, envelope 44 has a region that overlaps with the upper border 36c of envelope 36, and envelope 36 has a lower segment that overlaps with the upper segment of envelope 40. It is understood that envelopes 44, 36, and 40 become relatively larger due to the presence of high-rank cells within each envelope. The presence of high-ranking cells within each of the envelopes 44, 36, and 40 causes these three envelopes to become relatively large, and therefore they have overlapping segments. Furthermore, as described herein, it is understood that in various embodiments, when the respective envelopes do not overlap in a region but share a common boundary or part thereof (i.e., the respective envelopes are tangent or adjacent), similar treatment may or may not be provided.

[0074] In embodiments of the present invention, the envelope size may be determined by a combination of factors that, in combination, contribute to ensuring that the system substantially captures all cells, including clusters or groups of cells. In slides with millions of cells, it is a difficult problem to identify which are neighbors of each cell. In embodiments, it is preferable to use an r-tree data structure for speed purposes to combine adjacent rectangles to obtain larger rectangles. Grouped cells (e.g., within a cell cluster) tend to be positioned closer to each other than cells unrelated to that group. The system takes these spatial tendencies into account to ensure that the envelope is appropriately sized to encompass groupings of cells with a high degree of confidence that the grouping is presented within the envelope. In exemplary embodiments, the envelope size is not derived directly or alone from the neural network score, but rather the neural network score may be further modified. For example, in embodiments of the present invention, the neural network score may be multiplied by a dynamically generated multiplier, e.g., one based on the distance between cells. For example, in embodiments of the present invention, the multiplier can be derived by calculating the average distance between each cell nucleus in a given sample or sample region (e.g., a region defined according to region determination performed according to the neural network score before multiplication), and then multiplying the average distance by a numerical value (e.g., 2, 3 or greater). Thus, if the average distance between cell nuclei is multiplied by 2, the multiplier will present twice the distance between cells. In other embodiments, the multiplier or additional multiplier may be based on classification. For example, a first type of estimated cellular abnormality generally associated with closely related cell groupings may have a related multiplier less than 1, such as 0.7, while a second type of cellular abnormality generally associated with heterogeneous cell groupings may have a related multiplier greater than 1, such as 1.5. In embodiments of the present invention, the neural network score is multiplied by a multiplier of 1 or greater to obtain the final result, which is then converted into units of distance that form an envelope.A specific multiplier can generally or typically be set or reset to encompass an influence region that encloses a given point, for example, a relevant feature that tends to occur around a cell of interest. For a sample, if the average distance between cell nuclei is determined to be, for example, 0.5 times the average cell width of the cell type of interest, then cells that fall within this region are considered to be within the influence region. The influence region is set to, for example, 0.5 to 1.5 times such cell width. This can be repeated for each cell to determine the entire influence region. For example, a cell whose cell width falls within a value of 2.0 may be considered outside the influence region.

[0075] The multipliers described in embodiments of the present invention help ensure that the tiles are of an appropriate size and that there is a high degree of confidence that each cell in the cell grouping is included. In this regard, for example, a small multiplier is used when the distance between each nucleus is small, as the cells are closely clustered together. On the other hand, if the cells are more loosely related and located far apart from each other, a larger multiplier will be generated to produce a larger envelope size in order to ensure that sufficient representative cells are obtained. A feedback mechanism may be introduced, for example, feedback provided by one or more pathologists using the system of the present invention regarding whether the indicated envelope is too large or too small, and such feedback may be used to adjust or dynamically adjust the tile size or multiplier adopted. A neural network may also provide such feedback by being trained on images of optimally sized and suboptimally sized cell clusters tagged with an index indicating whether the multiplier should be increased (as in the case of an image that does not include the entire cluster), decreased (as in the case of an image that includes far more than the entire cluster), or maintained (as in the case of an image that adequately shows the entire cluster).

[0076] In embodiments of the present invention, once an envelope is generated, the system detects envelopes that include overlapping regions or, in some embodiments, adjacent regions (step 56, Figure 2), and generates a tile of a size and shape that captures all closely overlapping envelopes, for example, as shown in Figure 3B. Specifically, the system dynamically generates a frame 46 of a size and shape that captures envelopes 44, 36, and 40. As shown in Figure 3B, in order to capture envelopes 44, 36, and 40 in a single tile, in one embodiment, a frame 46 is formed that is large enough to capture the highest point defined by the upper boundary line 44a of envelope 44, the lowest point defined by the lower boundary line 40d of envelope 40, the leftmost point defined by the left boundary line 44b of envelope 44, and the rightmost point defined by the right boundary line 40a of envelope 40 in the manner of a minimum boundary rectangle. In other embodiments, a rectangle may be formed to include a box while maintaining a specific aspect ratio, such as a ratio related to the pathologist's electronic display (e.g., 16:9). In other embodiments, the frame 46 may consist solely of its combined shape, or may be formed as potentially non-rectangular, based on its combined shape. For example, two rectangles sharing a region around their corners may be combined into a concave octagonal frame. An R-tree may be used to merge rectangles. Alternatively, two overlapping circles may be combined into a larger circle, ellipse, or other shape that borders such constituent circles. The existence of overlap between two circles may be determined, for example, by whether the distance between their centers is less than the sum of their radii (or less, in some embodiments where adjacency is sufficient to overcome the overlap). On the other hand, the use of quadrilaterals and rectangles may be advantageous in saving computational resources, for example, with respect to reforming such envelopes or tiles into larger tiles. Such more complex or larger frames may be stored along with their corresponding border rectangles or other frames, so that the figures for each are selectively selected or presented, for example, according to specific display software or according to a specific choice of the pathologist using the program.

[0077] Figure 3C shows a resulting tile 41 that, according to various embodiments of the present invention, will be formed and displayed, for example, as part of a series of tiles representing an overall slide, or as part of a side determined to be appropriately diagnostically relevant (step S7, Figure 2).

[0078] In embodiments of the present invention, the system is configured to apply threshold criteria to determine whether or not to merge overlapping envelopes. For example, in embodiments of the present invention, the system merges only envelopes containing cells with neural network scores exceeding a certain threshold. In embodiments of the present invention, overlapping envelopes are merged if the neural network score is at least 0.6, or at least 0.7, or at least 0.8. Thus, in embodiments of the present invention, the system does not blindly merge overlapping envelopes, but rather applies these criteria to ensure that the envelopes to be merged contain diagnostically important information.

[0079] Therefore, in embodiments, clusters may be constructed using influence region processing to determine overlap in order to identify “bad” cells, e.g., malignant cells, but the results of the neural network score and / or classifier may be used to assign a credibility score to cells determined to overlap. For example, a credibility score may be applied, where cells exhibiting a pathological archetype as determined by a neural network (e.g., trained on images of pathological and non-pathological cell clusters, each tagged with a probability score based on whether they are pathological or not, or the probability that they are pathological) may be given a credibility score of up to 1.0, and normal cells may be given a score of 0 or less. In embodiments, the credibility score may be used as a distance multiplier to determine whether a cell or determined region would be considered to overlap with a neighboring cell or determined region, and healthy, normal cells or regions based thereon have a multiplier of zero and are therefore excluded. In embodiments, this results in merging into a displayed field or region, which occurs as a function of both the proximity of cells and regions and the apparent pathologicalness of such cells / regions. Therefore, the relative positions and proximity of “bad” cells or potentially “bad” cells (e.g., malignant cells, malformed cells, etc.) are taken into consideration in the final displayed tiles, including merging. In exemplary embodiments, the type of classification (e.g., potentially cancerous, potentially malformed, etc.) may also be considered as a multiplier in the merging determination. Ultimately, in embodiments, bin packing or other presentations on an electronic display shown to a pathologist may place as much information as possible on a given display, subject to constraints such as readability and available screen space. For example, the most important areas may be displayed in a readable size on the display along with the next most important areas, and so on, until it is not possible to add additional areas to the screen. The displayed results may be weighted, highlighted, or sorted, for example, by size, by a combination score or average score of the constituent envelopes for the constituent cells, or by credibility.One embodiment is a linear display that has the most important and largest tile information on the left side and progresses linearly to smaller tile sizes / less important information towards the right. Advantageously, in this embodiment, this can result in a time reduction of more than five times compared to the prior art in the evaluation time required for identification or diagnosis.

[0080] In particular, in some embodiments, if an envelope generated by the system does not overlap with or touch an adjacent envelope, that envelope may be rendered as a tile by itself. For example, referring to Figure 3A, if, for example, envelope 44 does not overlap with or touch envelope 36 because its constituent cells happen to be located significantly above and / or to the left, envelope 44 may be rendered as a tile by itself (and not containing cells in envelope 36), and may be smaller than a tile made from overlapping envelopes (for example, a tile made from a combination of envelopes 36 and 40).

[0081] Figure 4 shows a display screen 48 according to an embodiment of the present invention, displaying tiles generated according to the software of the present invention. As shown, the display 48 is divided into a left browsing pane 50 and a right browsing pane 52. In the embodiment, panes 50 and 52 may be separated by a dividing line 54. As shown, the browsing pane 50 has an upper border 56, a lower border 58, a left border 60 and a right border (which may be, for example, a dividing line 54 as shown). In the embodiment of the present invention, the left browsing pane 50 displays tiles according to one embodiment of the present invention, and the right pane 52 displays a digitized image of the entire slide. In the embodiment, the images displayed in each of panes 50 and 52 may be panned and zoomed in by clicking on or within the image, using a mouse scroll wheel, or by one or more other tools built into or integrated with the browsing software or the display screen.

[0082] In embodiments of the present invention, the tiles displayed in the left pane 50 are confined in the y-direction between the upper boundary line 56 and the lower boundary line 58. In embodiments, the tiles can be scrolled only in the y-direction (or only in the x-direction in other embodiments). In embodiments, scrolling or movement may occur in different directions (e.g., also in the x-direction, or in other embodiments, also in the y-direction), but once the pathologist has completed scrolling in the y-direction, the images may automatically snap to the figure containing the area between the boundaries, or otherwise return, to ensure that all relevant images eventually appear on the pathologist's screen (e.g., when the mouse button used for movement is released). In embodiments, snapping or returning requires input from the pathologist (e.g., a mouse click), but is required before any additional scrolling in the primary direction may occur. Thus, the customizability of the figure and the completeness of re-viewing can be advantageously achieved. Furthermore, and advantageously, in the embodiment, the tiles may be presented as a linear continuum from the region of highest diagnostic significance (e.g., large tiles, or tiles associated with high confidence scores or combined scores or important types of classifications) to the tile of lowest diagnostic significance (e.g., smaller tiles, or tiles associated with lower confidence scores or less important types of classifications). In the embodiment, since larger tiles may contain the most diagnostically significant information (cell populations of interest), for example, in an event where the first tile of highest diagnostic significance immediately confirms a “bad” (e.g., malignant) outcome for which pathological re-examination has been requested, the pathologist may be able to re-examine the large tile first, enabling a rapid and accurate diagnosis. In the embodiment, smaller tiles may be re-examined later in the process because such tiles are less likely to contain (or do not contain) cells of interest.

[0083] In this regard, as mentioned above, tiles containing overlapping envelopes tend to be larger in many cases than tiles formed by a single envelope. In embodiments of the present invention, a bin packing algorithm is employed to preferentially pack the generated tiles in descending order of size from largest (most diagnostically significant) to smallest (least diagnostically significant). This places the largest tiles first along the linear presentation, with less important tiles presented towards the end of the linear presentation.

[0084] In the embodiment, the bin packing algorithm employed may be a one-phase algorithm, a two-phase algorithm, a hybrid first-fit, a hybrid-next-fit, a hybrid first-fit, a floor-ceiling, a finite next-fit, a finite first-fit, a finite bottom-left, a next bottom-left, an alternative direction two-dimensional bin packing algorithm, and a two-dimensional bin packing algorithm such as rectpack. In the embodiment, if space allows, larger tiles may be placed first, and smaller tiles may be placed between larger tiles.

[0085] In embodiments, other or additional systems may be employed, for example, the slides may be rearranged into separate sets (e.g., each relating to a separate classification) so that re-examination of various conditions may be performed sequentially by a pathologist or in a programmed order of choice by the pathologist or the software. In embodiments, the tiles may be presented linearly for re-examination (e.g., in a line without tiles being presented on top of other tiles) rather than being packed into a two-dimensional space.

[0086] For example, referring to Figures 6–10, a series of screens 48 are shown, each displaying a figure in the viewing pane 50 as the user scrolls in the y-direction (e.g., right to left). In other embodiments, scrolling may occur from top to bottom (e.g., x-direction), right to left, or bottom to top. As illustrated, the pane 50 in Figure 6 contains very few large tiles. These tiles may present the most diagnostically significant information in the specimen. In Figure 7, the tiles are, schematically, smaller in size than those in Figure 6. Referring to Figures 8 and 9, the size of the tiles gradually decreases as the tiles continue in the y-direction. Figure 10 represents the tiles presented at the end of the series. In particular, while the tiles in Figures 6–9 contain tissue fragments and cell clusters, the tiles shown in Figure 10 mainly contain single cells, which may provide a minimal amount of clustered diagnostic information, as described.

[0087] In the embodiment, the pathologist may first review the largest tiles (here in Figure 6), and since these may contain the most diagnostically significant information, they may spend the most time reviewing these tiles. Without the present invention, the pathologist would not know where to begin reviewing cells on the slide and would therefore have to choose a random area to start. Consequently, in many cases, the pathologist may, through guesswork using conventional computer-implemented and non-computer-implemented review tools, start with the least diagnostically significant areas on the slide and spend a great deal of time searching for more important areas.

[0088] Once the pathologists have completed their review of the tiles in Figure 6, they will move on to the next series of tiles in Figure 7, as these are the next most diagnostically significant tiles. The pathologists then proceed through the tiles in Figures 8–10, spending less time on each series of tiles as they get smaller. This is because smaller tiles may contain less diagnostically significant information.

[0089] By including the most diagnostically significant information in the largest tile, and by structuring the tiles from largest to smallest, pathologists are guided to begin with (and spend the most time reviewing) the most diagnostically significant information. This solves a significant problem faced by pathologists with known manual and computer-based review methods. In fact, with conventional techniques, pathologists often have to guess where on the slide to begin, frequently spending excessive time on areas of the slide that are not significant for diagnostic purposes.

[0090] Furthermore, because the largest tiles group together the most diagnostically significant cells, there is a greater likelihood that pathologists can make more accurate diagnoses compared to conventional diagnostic methods, which require pathologists to first locate the cells of interest and then map their positions relative to adjacent cells. Conventional methods are not only cumbersome but also prone to significant human error.

[0091] In embodiments of the present invention, tiles displayed in the left viewing pane 50 are mapped to the right viewing pane 52. In this regard, the user may select a tile in the viewing pane 50, which will then be displayed in the context of the fully digitized slide in the right viewing pane 52. For example, Figure 4 shows a group of tiles in the left viewing pane 50. The right viewing pane 52 shows a reduced view of the entire digitized slide. In embodiments, for example, if the user wants to see where the information of tile 41 can be found within the context of the digitized slide, the user can select tile 41, and the right viewing pane 52 will zoom in from where tile 41 is derived into the area of ​​the slide, as shown in Figure 5. In embodiments, the user can also zoom out to see an adjacent area, or zoom in to see the target cell in more detail. This may make it easier for a pathologist to record the location of the cell being examined (e.g., a tile in the viewing pane 50) relative to all the information on the slide (shown in the viewing pane 52). This represents a significant improvement over conventional manual and computer-based methods, as prior art requires pathologists to manually map the location of target cells on the slide. In embodiments of the present invention, for example, if tiles are shown as a reduced or relative reduced image, the display version of the slide portion may be based on compressed image data. Advantageously, the amount of data transmission required can be reduced because it is not necessary to transmit the entire available data regarding the image to the displaying computer. In some embodiments, the total size of an uncompressed slide may approach, for example, 20 gigabytes, so significant time and data transfer savings can be achieved by avoiding the need to present the entire uncompressed slide image to the pathologist simultaneously. In embodiments, the uncompressed slide image may be streamed to the pathologist's display during the re-viewing process to allow for an initial re-view before receiving the entire uncompressed image. To further avoid delays, the streaming may occur in the order in which the tiles on the slide are presented to the pathologist, with respect to the tiles on the slide, so that while the pathologist is re-viewing earlier, for example, more important tiles, the streaming of tiles to be presented later may be downloaded.When a user performs a zoom operation, additional data may be provided to the user's computer to enable the relevant portion of the image displayed on the tile as a result of the zoom to be displayed at a resolution appropriate for re-viewing. Advantageously, this may allow a pathologist viewing the display to view details about the cell nuclei sufficient for diagnosis, for example, in such a magnified view. In embodiments, various tiles may be individually zoomed in and out by the pathologist to provide customization in the pathologist's re-viewing of the slide images.

[0092] Referring to Figure 6, in another embodiment of the present invention, the system allows the user to view the relationship between two different tiles in the context of a larger overall slide image. For example, as shown, the user can select tiles 41 and 22, and the slide area from which each of these tiles is derived is highlighted in the right viewing pane 52.

[0093] Referring to Figure 11, in an embodiment, the system may be configured to mark (e.g., with a flag) portions of the digitized slide that have been previously identified by the computer as “positive” for abnormality and / or “negative” for abnormality. The system is configured to present information about positive and negative regions in various arbitrary configurations. This feature allows pathologists to easily “jump back” to diagnostically significant cells as they review the slides. In an embodiment, other markings, e.g., markings for areas with particularly high scores, markings given specific classifications, markings for combined tiles and / or their components, and markings related to such other determinations described herein, as performed by a computer system and / or neural network, may be made available for presentation to the pathologist.

[0094] For example, in one embodiment, the system provides a toolbar or similar menu 56 that allows the user to select actions to output a specific figure and switch between various figures. In one embodiment, the user may choose to display all “positive” areas on the digitized slide. As shown in Figure 11, when the user selects to display positive areas on the slide, the system displays a mark on such slide areas, for example, in the form of a rectangle 58 or other image marker. The user may click on any rectangle (e.g., 58) to enlarge the image of the marked positive area. In embodiments of the present invention, the marks identifying the positive areas may be a first color or a different color or visually distinguishable (for example, red or indicated by flashing or dashed lines).

[0095] In embodiments of the present invention, the system is configured to similarly mark portions of the digitized slide that are "negative" for anomalies. For example, referring to Figure 12, the user can choose to mark areas of the digitized slide that are "negative" (e.g., from an option in menu 56), thereby the system displays a mark on the negative slide area in the form of, for example, a rectangle 60 or other image marker. The user can click on one of the rectangles (e.g., 60) to enlarge the image of the negative area. In embodiments of the present invention, the mark identifying the negative area is a second color (e.g., blue).

[0096] In embodiments of the present invention, the user may switch the markers displayed on the screen, or the user may select the option to display both a positive marker (e.g., 58) and a negative marker (e.g., 60) simultaneously. For example, as shown in Figure 13, the digitized image of the sample is provided with positive and negative markers that mark positive and negative slide regions (e.g., 28, 60), respectively.

[0097] It is understood that "positive" and "negative" regions can be determined by any of the various techniques in various embodiments of the present invention. For example, when analyzing glandular epithelium such as tissue obtained from the upper gastrointestinal tract, a dye (e.g., cdx2) that binds to abnormal columnar cells and stains such abnormal cells with a specific color (e.g., brown) is commonly used. The system can then identify abnormal columnar cells by searching for the color (e.g., brown) of the dye in question. To identify unstained (i.e., negative) columnar cells, a system using an image recognition system such as a classifier and / or a trained neural network computer identifies the columnar cells based on morphological features. Thus, the system stores information about stained regions (obtained by color analysis) and information about unstained columnar cells (e.g., obtained by computer image analysis).

[0098] For example, in an embodiment of the present invention, the system is configured to visually present the ratio of positive areas to negative areas on a slide and to calculate the ratio value based on the spatial relationship. In an embodiment of the present invention, the system queries a digitized image (or a database containing information based on the digitized image) for all positive and negative areas of the digitized slide. The system then assembles each positive area and presents them in a first area of ​​the display. The system similarly assembles each negative area and presents them in a second area of ​​the display, adjacent to or near the first area, in such a manner that a viewer can observe the overall image of the positive areas relative to the overall image of the negative areas.

[0099] For example, referring to Figure 14, a display screen having an upper viewing pane 62 and a lower viewing pane 64 according to an embodiment of the present invention is shown. A dividing line 66 may exist separating the upper viewing pane 62 and the lower viewing pane 64. As shown, the dividing line 66 is provided with markers that identify percentage values ​​of slides along a continuum. For example, a 10% marker 68 and equally spaced 20% markers 70 are shown on the dividing line 66. It should be understood that only about 20% of the overall negative image is shown in Figure 14, and the entire negative region (i.e., up to 100%) can be observed by scrolling to the right. As shown, the aggregation of positive cells 72 spatially represents about 12% of the aggregation of negative cells 74 (i.e., occupies a significantly smaller space along the x-axis compared to the negative region). This provides the pathologist with a visual indication of how likely a specimen may be positive.

[0100] In embodiments of the present invention, the system provides a search window function that allows the user to dynamically create sub-sections or similar user-defined search areas within the overall slide image. For example, referring to Figure 15, the system allows the user to draw a search window 76 using a computer mouse or via similar commands. In embodiments of the present invention, the size of the search window 76 is customizable by the user.

[0101] In embodiments of the present invention, upon initiating a search, the system displays an accompanying toolbar 78 within or near the browsing pane 52. In the embodiment shown in Figure 15, the toolbar 78 is located in the upper right corner of the right browsing pane 52. The toolbar 78 includes control icons, such as navigation tools, which, when selected by the user, allow the user to navigate within the search window 76.

[0102] In embodiments of the present invention, the system tracks and remembers each instance in which the user navigates to a given region within the search window 76. For example, referring to Figure 16, the system provides a user-defined map 80 of the search window 76, in which regions navigated by the user are shaded, and regions navigated more than once are shaded in a darker color than regions navigated only once (e.g., 82). Thus, in embodiments of the present invention, the map 80 provides the user with a visual guide of which regions within the search window 76 have been revisited. In embodiments of the present invention, the system darkens the region in the map 80 each time the user traverses a given region within the search window 76. In this regard, the user is presented with a visual indication of which regions have been revisited more often than others. In exemplary embodiments, regions may be selected and marked by the user (e.g., a pathologist), for example, as potential areas of interest, as areas not of such potential interest, and as areas requiring further revisitation. In this embodiment, a user may share a selected set of tiles or regions with others, for example, by automatically composing an email or by automatically inserting it into a password-protected file directory.

[0103] In embodiments of the present invention, the system is configured to analyze slides containing fixed cell block segments. For example, the system digitizes slides containing cell block slices attached to the slide and generates a gallery of tiles as described herein. As described, since the cutting of cell blocks is performed in close proximity and in succession, two adjacent cell block slices will usually be very similar to each other, and may even appear identical. Thus, in embodiments of the present invention, the system provides a pathologist with the ability to compare the reaction of identical tissue structures to two different stains or immunohistochemical markers.

[0104] For example, referring to Figure 17, a display having a first viewing pane 84 and an adjacent second viewing pane 86 is shown. In embodiments of the present invention, a first cell block slice (or a segment thereof) is displayed in the first viewing pane 84, and a second cell block slice is displayed in the second viewing pane 86. The second cell block slice may be cut in conjunction with the first cell block slice or near the first cell block slice. Thus, the first and second cell block slices contain substantially the same tissue and cellular structure. The system may display a first cell block stained with a first dye and a second cell block stained with a second dye in a juxtaposed manner, allowing a pathologist to view the same tissue region under two different staining conditions.

[0105] In embodiments of the present invention, the system further provides the user with the ability to align and move the images displayed in the first browsing pane 84 and the second browsing pane 86. For example, as shown in Figure 17, the system provides the user with the option to select anchor points on each image in the respective browsing panes 84 and 86. As illustrated, the user may, for example, select corresponding points in either pane 84 or 86, simultaneously position a first image anchor 88 on a point in the first image and a second image anchor 90 on a point in the second image. In this way, both images are simultaneously moved and scaled by the user to enable various comparison and navigation options.

[0106] While the present invention has been described in conjunction with the embodiments outlined above, it will be apparent to those skilled in the art that many alternatives, variations, and modifications are obvious. Therefore, the exemplary embodiments of the present invention described above are for illustrative purposes only and are not limiting. Various modifications can be made without departing from the spirit and scope of the invention.

Claims

1. A computer implementation method for dynamically presenting sample cells in an embodiment that demonstrates diagnostic significance, a. A step of accessing a digitized cell specimen stored on a storage medium, b. The step of identifying a first abnormality associated with at least one first cell in the digitized cell specimen and a second abnormality associated with at least one second cell in the digitized cell specimen through the use of a classifier, c. A step of calculating a first score based on the first abnormality characteristic associated with at least one of the first cells, d. A step of calculating a second score based on the second abnormality characteristic associated with the second cell, e. A step of generating a first target region of the digitized cell specimen surrounding the first at least one cell, wherein the size of the first target region is calculated by the processor based on the first score, f. A second target region of the digitized cell specimen is generated, the size of which is calculated by the processor based on the second score, wherein the size of the second target region is calculated by the processor based on the second score. g. The first target region is determined to be in contact with the second target region based on at least one of the following: (i) the determination that the first target region and the second target region each include a common outer perimeter, and (ii) the determination that the first target region and the second target region each include a common inner perimeter, wherein the target region is substantially rectangular in shape, and the dimensions of each side of the target region are greater than the minimum distance between the outer perimeter boundaries of adjacent cells in the sample, and less than the distance between cell clusters in the sample. h. Selectively, if the determination is that the first target region is in contact with the second target region, i. The first target region and the second target region are combined with the third target region of the digitized cell specimen which includes the first target region and the second target region, and ii. Display on the electronic display a combination image tile showing at least one first cell, at least one second cell, and the third target region. Steps and Equipped with, If the determination is that the first target region does not come into contact with the second target region, then a first image tile showing at least one first cell and the first target region is displayed on the electronic display, and a second image tile showing at least one second cell and the second target region is displayed on the electronic display. This method ensures that the combination image tile representing the third target area is larger than the first image tile or the second image tile, and that the larger tile is displayed first in the linear shape presentation.

2. The method according to claim 1, wherein the display comprises the linear presentation of image tiles.

3. The method according to claim 2, wherein, in the presentation of the linear shape of the tiles, the first image tile is presented before the second image tile.

4. A system for dynamically generating image tiles by the method described in Claim 1, Electronic displays and A storage medium for storing digitized cell specimens containing cells, A processor operably connected to the storage medium and the electronic display, Equipped with, A system wherein the processor is configured to perform the method according to claim 1.

5. The method according to claim 1, wherein the digitized cell specimen comprises at least one slice of a cell block.

Images