A system for processing whole slide images (WSI) of biopsies

Abstract

A system and method for processing whole-slide images (WSIs) of biopsies, such as core needle or vacuum-assisted biopsies, is provided. A skeleton of the detected tissue shape is created and a skeleton path is determined. An image is generated that includes a line representing the skeleton path, with different line representations along the line representing different tissue pathologies. A pathology summary for the entire line is also prepared. This provides the information desired by the pathologist in the most convenient format and presentation.

Description

Technical Field

[0001] The present invention relates to the processing of whole slide images, such as for use in core needle biopsy or vacuum assisted biopsy. Background Art

[0002] Whole slide images, WSIs, are digitized images of slide specimens, which enable the automatic detection and interpretation of features of tissue specimens.

[0003] Current practice in computer-aided detection and / or diagnostic analysis of whole-slide images is to use simple dot markers or polygons on top of the clinical image to represent regions of interest. Probability maps can also be displayed to provide more information, but the disadvantage is that they obscure the original clinical image.

[0004] The development of digital and computational pathology has enabled the application of these methods to create new medical fields that take advantage of whole slide images (WSIs). At the same time, advances in computer vision technology, especially deep learning, have enabled the analysis of large quantities of clinical images, making it possible and useful to integrate such technologies into the clinical pathology workflow.

[0005] Such automated analysis has been applied, for example, to the detection and grading of prostate cancer, or the detection of breast lymph node metastases.

[0006] In some specific cases, the medical professional is not interested in a two-dimensional representation (e.g., a polygon) of the detected affected tissue area, but only in the detection along a specific axis. In particular, core needle biopsy or vacuum-assisted biopsy is used to extract a column of cells, thereby enabling the analysis of cell structure along a path (such as depth) through the location of the tissue body. The use of biopsy in pathology follows well-defined protocols. These protocols assume that the biopsy is a single-dimensional length of tissue, whose width is irrelevant for diagnostic purposes. In fact, for core needle biopsy and vacuum-assisted biopsy, the main and only relevant axis is the length of the biopsy, and most protocols recommend reporting the length and line coverage of tumor tissue (or other type of diseased tissue under study).

[0007] Displaying 2D images of elongated tissue, such as those obtained from core needle or vacuum-assisted biopsies, is inconvenient because the view of the original specimen becomes cluttered. Many pathologists report that this hinders their workflow. Furthermore, the displayed 2D information does not correspond to the information required to be reported according to official medical protocols, particularly length and coverage, as described above.

[0008] It is desirable to enable desired information to be extracted from whole-slide images WSI by automated image analysis. Summary of the Invention

[0009] The invention is defined by the claims.

[0010] According to an example according to one aspect of the present invention, a system for processing a whole slide image WSI of a biopsy is provided, comprising a processor adapted to:

[0011] Detection of tissue associated with a biopsy in the WSI;

[0012] creating a skeleton of the shape of the detected tissue;

[0013] The skeleton path is selected as the longest continuous path through the tissue;

[0014] determining the histopathology of the examined tissue; and

[0015] Generate an image, including:

[0016] a line representing a skeleton path with different line representations along the line representing different tissue pathologies; and

[0017] A summary of the pathology for the entire line.

[0018] This system enables automatic determination of desired information from a biopsy as a summary of the pathology associated with a line along the biopsy and an image representation of the line conveying the pathology information. The line is displayed, for example, on top of the original WSI image, making all the original image content visible as well as the summary information provided by the line. The line is essentially a one-dimensional representation of the pathology information. It is important to note that a line is generally not straight, so the single dimension of a line is not straight. In addition, the thickness of the line can vary along its length to convey pathology information (as discussed below), so the line can have a two-dimensional component even though it is primarily a one-dimensional representation. This is what is meant by "essentially one-dimensional."

[0019] Therefore, the present invention combines the use of digitized slides in pathology with the application of image analysis algorithms for skeleton detection and for pathology detection. These can be, for example, deep learning based detectors (e.g. tissue and pathology detectors).

[0020] The line comprises, for example, the centerline of the biopsy obtained by deriving the position, form and length of the centerline from the WSI of the biopsy. Pathology information, such as identification of the diseased area, is encoded into the line representation and derives tumor coverage information for pathology reporting.

[0021] Display of the biopsy line representation enables the pathologist to confirm the pathology summary by assessing the quality of both tissue detection and tumor (or other tissue pathology) detection.

[0022] Different representations include, for example, different colors or line thicknesses. Thus, the appearance of the line encodes information related to the tissue pathology.

[0023] The pathology summary includes the biopsy length and the portion of the line occupied by a specific type of histopathology. It may only include the type of histopathology (diseased or non-diseased), but there may be more. The pathology summary can also present the individual lengths that make up the portion of the line occupied by a specific type of histopathology. An example of a specific type of histopathology is tumor tissue.

[0024] Generating the line can include overlaying the detected tissue pathologies on the skeleton path and selecting a representative tissue pathology for each point along the skeleton path. Thus, the line conveys different pathology types.

[0025] For example, if tumor tissue is present at or perpendicular to a corresponding point along the line, then the representative tissue pathology is selected as tumor tissue. Thus, the line is, for example, the midline, and for the detection of tumor tissue, the detection is based on the tissue at the point along the line and the tissue samples on each side of the point.

[0026] A first machine learning algorithm may be used for tissue detection, and one or more second machine learning algorithms may be used for detecting tissue pathology.

[0027] The system preferably further comprises a display for displaying the image with the pathology summary alongside the line.

[0028] The present invention also provides a computer-implemented method for processing a whole slide image (WSI) of a biopsy, comprising:

[0029] Detection of tissue associated with a biopsy in the WSI;

[0030] creating a skeleton of the shape of the detected tissue;

[0031] The skeleton path is selected as the longest continuous path through the tissue;

[0032] determining the histopathology of the examined tissue; and

[0033] Generate an image, including:

[0034] a line representing a skeleton path with different line representations along the line representing different tissue pathologies; and

[0035] A summary of the pathology for the entire line.

[0036] The method may include generating an image with lines of different colors representing different tissue pathologies.The method also includes generating a pathology summary including the biopsy length and the portion of the line occupied by tumor tissue.

[0037] An image may be generated by overlaying or juxtaposing the determined histopathologies on a skeletal path and selecting an associated histopathology for each point along the skeletal path.

[0038] The invention also provides a computer program comprising computer program code means adapted to implement the method defined above, when the program is run on a computer.

[0039] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] For a better understanding of the invention, and to show more clearly how it may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0041] Figure 1 Whole-slide images and regional outlines of single-core needle biopsies are shown;

[0042] Figure 2 The region outline and skeleton are shown;

[0043] Figure 3 The skeleton forming the skeleton path and the pruned skeleton are shown;

[0044] Figure 4 The skeleton path and the skeleton path overlaid within the region outline are shown;

[0045] Figure 5 The region outline with the skeleton path and the skeleton path combined with the pathology image are shown;

[0046] Figure 6 shows a skeleton path combined with a pathology image, and shows how the pathology information is projected onto the skeleton path;

[0047] Figure 7 Shows that it is possible to Figure 6 The pathological summary for the entire line is obtained by analyzing the information;

[0048] Figure 8 An example of a WSI with multiple samples is shown;

[0049] Figure 9 It shows how to use the method described above applied to the first example of the present invention to modify Figure 8 images;

[0050] Figure 10 A system for processing whole slide images of a biopsy is shown; and

[0051] Figure 11 A computer-implemented method for processing whole-slide images of a biopsy is shown. DETAILED DESCRIPTION

[0052] The present invention will be described with reference to the accompanying drawings.

[0053] It should be understood that the detailed description and specific examples, although indicating exemplary embodiments of the apparatus, system, and method, are for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0054] The present invention provides a system and method for processing whole-slide images (WSIs) of biopsies, such as core needle or vacuum-assisted biopsies. A skeleton of the detected tissue shape is created and a skeleton path is determined. An image is generated that includes a line representing the skeleton path, with different line representations along the line representing different tissue pathologies. A pathology summary for the entire line is also prepared. This provides the information desired by the pathologist in the most convenient format and presentation.

[0055] The present invention relates to elongated biopsies where pathological information at different locations along the elongated length of the biopsy is of interest. As an example of the type of biopsy to which the present invention may be applied, Figure 1 On the left is shown a whole slide image WSI 10 of a single core needle biopsy. There may be multiple biopsies in one WSI, but the invention will be explained using a single biopsy in a WSI.

[0056] The steps performed in the processing of WSI will be discussed. These steps are performed as image analysis steps of WSI.

[0057] The first step is to detect all organizations present on the WSI. This creates Figure 1 The region outlines 12 are shown in the middle right figure. If there are multiple biopsies, separate tissue regions corresponding to the individual biopsies (equivalent to separate samples) are identified.

[0058] For the biopsies identified, e.g. Figure 2 As shown in FIG, a skeleton 20 of a biopsy is identified. Figure 2 The left image of shows the region outline 12, and the right image shows the final skeleton 20. A skeleton is a series of nodes and vertices connecting the nodes, which represent identifiable branches in the shape of the tissue sample.

[0059] The skeleton is then pruned to remove any branches of the skeleton that differ from the longest possible path. Figure 3The left figure shows the skeleton 20, and the right figure shows the trimmed skeleton 30. The trimmed skeleton defines a skeleton path 32, which is the longest continuous path through the tissue. It follows the midline, that is, each point along the skeleton path 32 is located at the center of the sample width at that point along the sample length.

[0060] The skeleton path 32 thus defines the full extent of the biopsy.

[0061] Figure 4 The left image of shows the skeleton path 32 , and the right image shows the skeleton path 32 overlaid within the region outline 12 .

[0062] One or more relevant pathologies are then identified in each sample (ie, within the region outline 12).

[0063] Figure 5 The left image of shows the region outline 12 with a skeleton path 32, and the right image shows the skeleton path 32 combined with a pathology image 50. Thus, points along the skeleton path 32 are aligned with the associated pathology information.

[0064] Figure 6 The left figure shows the skeleton path combined with the pathology image 50, and the right figure shows how the pathology information is projected onto the skeleton path.

[0065] The result is a line 60 representing the skeleton path, but with different line representations along the line representing different tissue pathologies. For example, region 62 may correspond to an area of ​​tumor tissue (e.g., indicated in red), while other areas do not have tumor tissue (e.g., indicated in green).

[0066] Whether tumor tissue or normal tissue is represented along the line can be determined based on the most sensitive detection method by which any tumor tissue detected along a path perpendicular to a point along line 60 causes that point to be indicative of tumor tissue. If multiple pathologies are detected, a weighting scheme can determine which pathology is identified at that point along line 60.

[0067] Figure 7 Shows that it is possible to Figure 6 This shows the coverage percentage (as a percentage of the line length, in this example 80%), the total biopsy length (in this example 7 mm) and the total corresponding physical length of the tumor tissue (in this example 5.6 mm).

[0068] This example has only two classes (tumor or normal), but there could be more.

[0069] Figure 8 An example of a WSI with multiple samples is shown.

[0070] Figure 9 It is shown how an image is modified using the method described above applied to a first sample 90 within a WSI. This image is displayed on a display device.

[0071] A line 60 is overlaid on the WSI. It has different colors or thicknesses (or other differences, such as markings, dot patterns, etc.) along its length to represent different tissue pathologies. In this example, a region outline 12 is also shown.

[0072] A pathology summary is displayed beside the line, such as the tumor length 92 for individual tumor portions along the length of the line and an overall summary 94 (such as Figure 7 middle).

[0073] The pathologist can zoom in to examine a specific area of ​​the WSI.Then, the line 60 can, for example, disappear so that the examination is unobstructed.

[0074] Figure 10 A system for processing a whole slide image (WSI) of a biopsy, such as a core needle biopsy, is shown. The system includes a processor 100 adapted to perform the analysis steps described above. This involves detecting tissue present in the WSI, creating a skeleton of the shape of the detected tissue, selecting the skeleton path that is the longest continuous path through the tissue, determining the histopathology of the detected tissue, and generating an image.

[0075] The image is displayed on a display device 102. It includes the original WSI with a line added to represent the skeleton path, with different line representations along the line representing different tissue pathologies. A pathology summary for the entire line is also displayed.

[0076] The first machine learning algorithm MLA1 is used, for example, in the tissue detection step. Such algorithms are known.

[0077] As an example, see the article "tissueloc: Whole slide digital pathology image tissue localization" by Pingjun Chen et al., Department of Biomedical Engineering, University of Florida, DOI: 10.21105 / joss.01148.

[0078] The one or more second machine learning algorithms MLA2 are used, for example, to detect tissue pathology.

[0079] As an example, see Gabriele Campanella et al., “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images,” Nature Medicine 25, 1301-1309 (2019).

[0080] For further reference, see Wouter Bulten et al.: “Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study,” DOI: https: / / doi.org / 10.1016 / S1470-2045(19)30739-9.

[0081] Algorithms for identifying skeletons from shapes in images are also well known. For example, EP 2 772 882 discloses creating skeletons from biopsy images and determining lengths from the skeletons.

[0082] Figure 11 A computer-implemented method for processing a whole slide image (WSI) of a biopsy is shown. The method comprises:

[0083] In step 110 , the presence of tissue associated with a biopsy in the WSI is detected;

[0084] In step 112 , a skeleton of the shape of the detected tissue is created;

[0085] In step 114 , a skeleton path is selected that is the longest continuous path through the tissue;

[0086] In step 116, the histopathology of the detected tissue is determined; and

[0087] In step 118, an image is generated for display.

[0088] The image is generated by creating a line representing a skeleton path in step 120 , with different line representations along the line representing different tissue pathologies, and creating a pathology summary for the entire line in step 122 .

[0089] As described above, the system utilizes a processor to perform data and image processing. The processor can be implemented in various ways using software and / or hardware to perform the various functions required. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware that performs some functions and one or more programmed microprocessors and associated circuits that perform other functions.

[0090] Examples of circuits that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0091] In various implementations, a processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the desired functions. The various storage media may be fixed within the processor or controller or may be removable so that one or more programs stored thereon can be loaded into the processor.

[0092] Variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed disclosure, based on a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0093] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0094] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium supplied with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. (Optional)

[0095] If the term "adapted to" is used in the claims or the description, it should be noted that the term "adapted to" is intended to be equivalent to the term "configured to".

[0096] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A system for processing a whole slide image (WSI) of a biopsy, comprising a processor (100), the processor being adapted to: detecting the presence of tissue associated with a biopsy in the WSI; creating a skeleton (20) of the detected shape of the tissue; selecting a skeletal path (32) that is the longest continuous path through the tissue; determining the histopathology of the detected tissue; as well as Generate an image, the image comprising: a line (60) representing the skeleton path, having different line representations representing different tissue pathologies along the line; as well as A pathology summary (94) for the entire line, wherein the pathology summary includes the biopsy length and the portion of the line occupied by a specific type of tissue pathology. 2 . The system of claim 1 , wherein the different representations include different colors or line thicknesses.

3. The system according to claim 1 or 2, wherein the specific type of tissue pathology is tumor tissue.

4. The system of claim 1 or 2, wherein generating the line (60) comprises overlaying the detected tissue pathology on the skeleton path (32) and selecting a representative tissue pathology for each point along the skeleton path. 5 . The system of claim 4 , wherein the representative tissue pathology is selected as tumor tissue if tumor tissue is present at or perpendicular to the corresponding point along the line.

6. The system according to claim 1 or 2, comprising a first machine learning algorithm (ML1) for tissue detection.

7. The system of claim 6, comprising one or more second machine learning algorithms (ML2) for detecting tissue pathology.

8. The system according to claim 1 or 2, comprising a display (102) for displaying the image with the pathology summary next to the line.

9. A computer-implemented method for processing a whole slide image (WSI) of a biopsy, comprising: (110) detecting the presence of tissue associated with a biopsy in the WSI; (112) creating a skeleton of the detected shape of the tissue; (114) selecting a skeletal path that is the longest continuous path through the tissue; (116) determining the histopathology of the detected tissue; as well as (118) Generate an image, the image comprising: a line (60) representing the skeleton path, having different line representations representing different tissue pathologies along the line; as well as A pathology summary for the entire line, wherein the pathology summary includes the biopsy length and the portion of the line occupied by tumor tissue.

10. The method of claim 9, comprising generating an image with said lines having different colors, said different colors representing different tissue pathologies.

11. A method according to claim 9 or 10, comprising generating the image by overlaying or juxtaposing the determined tissue pathology on the skeleton path and selecting an associated tissue pathology for each point along the skeleton path.

12. A method according to claim 9 or 10, comprising displaying the image with the pathology summary alongside the line.

13. A computer program medium comprising computer program code means adapted to implement the method according to any one of claims 9 to 12 when said program is run on a computer.

Images