Methods, systems, and computer-readable media for determining anatomical regions of tissue

Abstract

A system and computer-implemented method are provided for identifying one or more anatomical regions in a series of pathology sections using a series of images representing digitized versions of the series. Annotations are obtained for at least two reference images, each annotation representing a anatomical region in a respective reference image. Annotations are then generated for intermediate images between the reference images based on bidirectional image registration. The two annotations are then propagated to each intermediate image, and the annotations are then combined to obtain a combined annotation for each intermediate image. This approach is well-suited for generating annotations for a series of pathology sections containing tissue sections of interest having complex three-dimensional shapes.

Description

Technical Field

[0001] The present invention relates to pathology and digital pathology, and more particularly to a system and computer-implemented method for determining one or more anatomical regions of tissue in a series of pathology slides. The present invention further relates to controlling a tissue dissection system based on one or more annotations generated by the system or method. The present invention further relates to a computer-readable medium comprising instructions for causing a processor system to perform the computer-implemented method. Background Art

[0002] In the field of pathology, surgeons often remove tissue samples from patients for examination by the pathology department of a hospital or elsewhere. This tissue sample can be embedded in paraffin wax to form what is often called a "tissue block." Pathologists can then cut the tissue block into very thin slices, place them on slides, and examine them under a microscope. This allows the pathologist to confirm a diagnosis.

[0003] Pathology has various specialized uses. For example, pathology can be used to guide treatment decisions for individual cancer patients, for example, using so-called genomic-based diagnostic tests. This may involve isolating cancer cells in tissue, which is often prepared on a glass slide, and using the isolated cancer cells in a polymerase chain reaction (PCR) process to "amplify" the DNA and / or RNA, enabling the detection or quantification of specific genetic aberrations of interest.

[0004] Common methods for separating cancer cells from surrounding non-cancerous cells or separating tissue of interest from surrounding tissue include, but are not limited to, manual dissection, such as cutting and / or lifting tissue with a blade, and (semi-)automatic dissection using milling tools, lasers, dissolving fluids, etc.

[0005] The tissue area to be dissected can be determined using the technology of digital pathology. This digital pathology is based on the digitization of pathology slides, which refers to converting tissue slides placed on slides into digital images ("digital slides") that can be viewed, managed, shared and analyzed on a display (e.g., a computer monitor). Therefore, a pathologist can specify tissue extraction areas by digitally annotating the image, such as using a mouse, pen or touch screen. For example, a pathologist can indicate these areas by selecting and placing geometric primitives (e.g., lines, circles, rectangles or other types of polygons) in the image, and then adjust their size and / or shape. Additionally or alternatively, a pathologist can define these geometric primitives by, for example, so-called "free form" tools, or can simply draw freely in the image, etc. It is also known to annotate morphological structures in images automatically or semi-automatically. If these areas are associated with morphological structures, such technology can be used to automatically or semi-automatically determine tissue extraction areas.

[0006] Typically, annotating tissue anatomical regions in images in which the tissue is stained with a stain to increase the visibility of certain types of biological material, which in turn may help identify tissue of interest, such as cancer cells. For example, hematoxylin-eosin (H&E) may be applied to the tissue to obtain a stained "reference" slice in which the tissue of interest can be better identified. At the same time, it is best to dissect the tissue of interest from unstained slices, which are also called "extracted" slices. Therefore, it may be necessary to transfer annotations representing tissue anatomical regions from a stained reference slice to an unstained extracted slice, or generally from one image to another.

[0007] US20180267290A1 describes a method for selecting a sample removal region of an unstained sample for removal for molecular diagnosis. The method comprises the following steps: a) selecting a reference removal region in a reference image of a reference slice of an object, wherein biological material in the reference slice is stained, b) obtaining a digital sample image of the sample slice of the object under an imaging setting, wherein the biological material in the sample slice is unstained, c) registering the digital sample image with the reference image to translate the reference removal region in the reference image to the digital sample image, and d) determining a sample removal region of the digital sample image based on the translated reference removal region.

[0008] Disadvantageously, US20180267290A1 is not robust in cases where the biological material has different shapes and / or positions across the slice. Summary of the Invention

[0009] It is an object of the present invention to more robustly transfer annotations identifying anatomical regions of tissue from a reference image to one or more extracted images.

[0010] According to a first aspect of the present invention, there is provided a computer-implemented method for determining one or more anatomical regions of tissue in a series of pathological sections using a series of images representing a digitized form of the series of pathological sections. The method comprises:

[0011] - obtaining a first annotation that identifies a first tissue anatomical region in a first image of the series of images;

[0012] - obtaining a second annotation that identifies a second tissue anatomical region in a second image of the series of images, wherein the first image and the second image are separated in the series of images by a series of intermediate images; and

[0013] - generating annotations for one or more intermediate images in the series of intermediate images by performing the following operations on each of the one or more images:

[0014] - determining a first set of registration parameters, said first set of registration parameters representing at least a partial registration of the intermediate image with the first image;

[0015] - determining a second set of registration parameters representing at least a partial registration of the intermediate image with the second image;

[0016] - propagating the first annotation from the first image to the intermediate image using the first set of registration parameters;

[0017] - propagating the second annotation from the second image to the intermediate image using the second set of registration parameters; and

[0018] - Combining the propagated first annotation and the second annotation to obtain the annotation in the intermediate image.

[0019] According to another aspect of the present invention, a system is provided for determining one or more anatomical regions of tissue in a series of pathology sections using a series of images representing a digitized form of the series of pathology sections.

[0020] The system includes:

[0021] - a data interface for accessing the series of images;

[0022] - Another data interface for accessing annotation input data, which defines:

[0023] - a first annotation that identifies a first tissue anatomical region in a first image of the series of images, and

[0024] - a second annotation identifying a second tissue anatomical region in a second image of the series of images, wherein the first image and the second image are separated in the series of images by a series of intermediate images; and

[0025] - A processor subsystem configured to:

[0026] - generating annotations for one or more intermediate images in the series of intermediate images by performing the following operations on each of the one or more images:

[0027] - determining a first set of registration parameters, said first set of registration parameters representing at least a partial registration of the intermediate image with the first image;

[0028] - determining a second set of registration parameters representing at least a partial registration of the intermediate image with the second image;

[0029] - propagating the first annotation from the first image to the intermediate image using the first set of registration parameters;

[0030] - propagating the second annotation from the second image to the intermediate image using the second set of registration parameters; and

[0031] - Combining the propagated first annotation and the second annotation to obtain the annotation in the intermediate image.

[0032] According to another aspect of the present invention, a computer-readable medium is provided, the computer-readable medium including transitory or non-transitory data representing instructions arranged to cause a processor system to perform a computer-implemented method.

[0033] The above measures are based on the understanding that the shape and position of the tissue of interest can vary significantly across a series of pathology sections, and therefore across a series of images representing the digitized form of that series of pathology sections. For example, a tumor may have a complex and irregular three-dimensional shape, and the two-dimensional intersections of this three-dimensional shape may vary significantly between different pathology sections. The tissue of interest may also have a complex shape, such as in the case of a prostate biopsy.

[0034] Therefore, if annotations of tissue anatomical regions are obtained for one of the images (e.g., a "reference" image), it may be difficult to transfer these annotations to the other "extracted" images. Here, the reference image may correspond to an H&E-stained pathology slide, or any other representation that facilitates obtaining annotations of tissue-extracted regions, while the extracted image may correspond to an unstained pathology slide, a Nuclear Fast Red (NFR)-stained pathology slide, or any other representation that may not allow for annotations of tissue-extracted regions or is more difficult to obtain than the reference image.

[0035] The above-described measures involve obtaining annotations that identify respective anatomical regions of tissue in at least two reference images. The two reference images are separated in a series of images by a series of intermediate images, which represent digitized versions of a series of intermediate pathology sections. The annotations can be obtained in a manner known per se, for example, by manual annotation by a pathologist or, in some embodiments, by semi-automatic or automatic annotation.

[0036] The annotation of at least one intermediate image can be obtained by at least partially registering each reference image with the intermediate image. Such registrations are known per se and generally express a correspondence between the image contents of the images and are therefore also referred to as "image registrations". Here, the term "at least partially" means that at least part of the reference image is registered with the intermediate image (e.g. a spatial sub-region, or a selected image feature), etc. As is known per se, each registration can be defined by a set of parameters. For example, the registration can be a parametric registration, but can also include registrations that are conventionally considered to be non-parametric registrations, e.g. registrations that are defined by a deformation field or something similar. In the latter embodiment, the registration parameters can define a deformation vector.

[0037] After obtaining both annotations, each annotation can be transferred to the intermediate image based on its respective registration. This transfer may typically involve deforming the annotation according to the registration. For example, if elastic registration is used, the elastic registration may indicate a deformation of the image content, which can be applied in the same way to the annotation overlaid on the image content. Thus, two annotations can be obtained for the intermediate image and then combined into one, for example using morphological operations, such as taking into account shape overlap, or distance transformations or shape context.

[0038] Compared to known techniques for transferring annotations from a reference image to an extracted image, the above approach transfers two or more annotations obtained for the respective reference images to both sides of the intermediate image in the series. Therefore, annotations can be transferred from either side of the intermediate image. Consequently, the combined annotation effectively represents a bidirectional interpolation of such annotations, rather than simply an extrapolation from one side. This interpolation can better account for significant variations in the shape and / or position of the tissue of interest across a series of pathology sections. In particular, if the shape and / or position of the tissue of interest varies significantly, the two annotations may also have different shapes and / or positions, for example, if a pathologist annotated each reference image differently to account for these variations. This represents additional information that can be taken into account when combining the two annotations. For example, the combined annotation may consist solely of the overlap between the two annotations, representing a logical sum and thus a "conservative" annotation that is more likely to encompass only the tissue of interest than either annotation alone. Consequently, a more robust annotation can be obtained compared to extrapolating annotations from one side.

[0039] Optionally, the first set of registration parameters and the second set of registration parameters each define a deformable registration, and propagating the first annotation to the intermediate image and propagating the second annotation to the intermediate image each include deforming the respective annotation based on the respective set of registration parameters. Such a deformable registration is also called an elastic registration, and typically expresses the correspondence between the image contents of the images not only in terms of translation and / or rotation (e.g., as an affine transformation), but also in terms of local deformation (e.g., a non-rigid transformation). After obtaining such elastic registration parameters, both annotations can be deformed according to the deformation calculated for the underlying image content. Thereby, each deformed annotation can better match the image content in the intermediate image.

[0040] Optionally, the method further includes determining a quality of fit of the annotations in the intermediate image, and if the quality of fit does not meet a quality of fit criterion, prompting the user to provide a third annotation in a third image included in the series of intermediate images between the first and second images. Such quality of fit can be determined in a variety of ways, such as by comparing the combined annotation with image content in the intermediate image, and / or by comparing the two propagated annotations. If the quality of fit is deemed insufficient, this may indicate that the shape and / or position of the tissue of interest differs significantly between one or both of the reference and extracted images, resulting in inaccurate propagation of either or both annotations. Therefore, it may be desirable to obtain annotations for a closer image (e.g., between the two reference images) that may be more similar to the intermediate image in terms of the shape and / or position of the tissue of interest. Therefore, the user may be prompted, for example, using an on-screen warning or in any other manner, to provide the third annotation in a third image included in the series of intermediate images between the first and second images. In some embodiments, this may involve having to stain and re-digitize the corresponding pathology slide, effectively converting the extracted slide into a reference slide, and obtaining another reference image between the two original reference images after re-digitization.

[0041] In some embodiments, obtaining a second annotation in the second image may be necessary only if the quality of fit achieved by propagating the first annotation to the intermediate image does not meet a fit quality criterion. For example, the methods and systems may first obtain the first annotation, determine a first set of registration parameters, propagate the first annotation from the first image to the intermediate image using the first set of registration parameters, and determine the fit quality of the resulting propagated annotation. In these embodiments, the fit quality may be determined based on, for example, user input received from a user, which may indicate the fit quality. If the fit quality is deemed insufficient (e.g., does not meet the fit quality criterion), the methods and systems may obtain a second annotation, for example, by prompting the user to provide a second annotation in the second image. After obtaining the second annotation, the methods and systems may determine a second set of registration parameters, propagate the second annotation from the second image to the intermediate image using the second set of registration parameters, and combine the propagated first and second annotations to obtain an annotation in the intermediate image. In other words, the functionality associated with the second annotation and the combination of the first and second annotations is conditional, as the functionality is only invoked if the fit quality generated by propagating a single annotation is insufficient.

[0042] Optionally, prompting the user includes indicating a position of the third image in the series of intermediate images or a position of the corresponding pathology slice in the series of pathology slices. For example, the user may be prompted to annotate the third image, indicating that the third image is substantially midway between the series of intermediate images, or in other words, substantially midway between the first and second images in the series of images, and thus midway between the corresponding series of intermediate pathology slices.

[0043] Optionally, determining the fit quality of the annotations in the intermediate image includes determining a first fit quality of the propagated first annotation and a second fit quality of the propagated second annotation, and the method further includes determining a position of a third image based on a comparison of the first fit quality and the second fit quality. The position of the third image to be annotated can be adaptively determined based on the respective fit qualities. For example, if one propagated annotation generates a poorer fit quality than another propagated annotation, this may indicate that the shape and / or position of the tissue of interest differs significantly between the intermediate image and the corresponding reference image, and therefore, it may be desirable to obtain annotations on one side of the intermediate image that are closer to the intermediate image. Therefore, when generating annotations for the intermediate image, the annotations for the third image can replace the annotations for the earlier reference image.

[0044] Optionally,

[0045] - the first image and the third image are separated in the series of images by an intermediate image of the first subseries;

[0046] - the second image and the third image are separated in the series of images by an intermediate image of the second subseries; and

[0047] The method further includes:

[0048] - generating an annotation for one or more intermediate images of the first subseries of intermediate images based on the first annotation and the third annotation; and

[0049] - generating an annotation for one or more intermediate images of the second subseries of intermediate images based on the second annotation and the third annotation.

[0050] The above measures provide an iterative process in which, if the quality of fit of the combined propagated annotations is deemed insufficient for a particular intermediate image, the annotation of a third image between the first and second images is obtained, after which the "annotation-propagation" process is repeated for both sub-series individually. This iteration can be repeated until all or a sufficient number of extracted images have been annotated, at which point the iterative process is stopped. Advantageously, the iterative process is adaptive, in that the user is asked to annotate another image only if the quality of fit is deemed insufficient (which may involve staining the corresponding pathology slide and re-digitizing the stained slide). Otherwise, the iterative process is based on "annotation-propagation" elsewhere.

[0051] Optionally, determining the quality of fit includes determining a difference in shape, or an overlap in surface area, between the propagated first annotation and the propagated second annotation. Such a measure of fit quality may be used in addition to or as an alternative to a fit quality based on comparing the annotation to the image content and considering that if two annotations are similar or substantially identical in shape and / or size, then the propagated annotation may be considered reliable.

[0052] Optionally, determining the quality of the fit includes determining a magnitude of a deformation of the propagated first annotation and / or the propagated second annotation. A larger deformation may indicate a larger difference in shape of the tissue of interest between the respective reference image and the intermediate image, and thus a greater likelihood that the propagated annotation is insufficient to fit the intermediate image.

[0053] Optionally, the quality of the fit is determined based on: i) the similarity and / or difference between the image data underlying the first annotation in the first image and the image data underlying the first annotation propagated in the intermediate image, and / or ii) the similarity and / or difference between the image data underlying the second annotation in the second image and the image data underlying the second annotation propagated in the intermediate image. This image-based quality of fit can express the reliability and / or accuracy of the respective registrations and can be combined with the above-mentioned quality of fit based on shape differences or surface area overlap. It will be appreciated that, alternatively, any other known measure of the reliability and / or accuracy of the respective registrations can be used to determine the quality of the fit.

[0054] Optionally, the first set of registration parameters and the second set of registration parameters are determined by one of the following: feature-based image registration, image registration using an active contour model, image registration using shape context, and image registration using a level set method. These registration techniques are known per se and have been found to be suitable not only for image registration but also for propagating annotations to one or more intermediate images.

[0055] Optionally, the first image, the second image, and the third image are pathological sections stained with hematoxylin and eosin (H&E), or pathological sections stained with another stain that is removable or non-removable from the pathological sections.

[0056] Optionally, the series of intermediate images are pathology sections that are unstained in paraffin, not stained in paraffin, and / or stained with Nuclear Fast Red (NFR) or another stain that is removable from the pathology section.

[0057] Optionally, the method further includes controlling a tissue dissection system to dissect tissue from one or more pathology slices corresponding to the one or more intermediate images based on the one or more annotations generated for the one or more intermediate images. Thus, the generated annotations can be used to automatically or semi-automatically extract tissue from the corresponding pathology extraction slices.

[0058] Optionally, determining the first set of registration parameters comprises determining individual registrations between respective pairs of images in the series of images, wherein the individual registrations together constitute a chain of registrations between the first image and the intermediate image. Thus, instead of directly determining the registration parameters between the first image and the intermediate image, the registration between the first image and the intermediate image may be obtained as a superposition of individual registrations between pairs of images between the two images. Such registrations between pairs of images (e.g., consecutive pairs of images, or pairs of images formed by each second or third image in a series of images) may be more robust because the distance between the images may be smaller than the distance between the first image and the intermediate image, and therefore it is desirable that the similarity between the pairs of images is higher than the similarity between the first image and the intermediate image, which may result in a more robust overall registration. The above-described "chain-based" registration method may also be used, mutatis mutandis, to determine the second set of registration parameters.

[0059] Optionally, the system further includes a user interface subsystem, including:

[0060] - a display output for showing the respective image on a display;

[0061] - a user input interface for receiving user input data from a user input device operable by a user;

[0062] The processor subsystem is configured to enable a user, through the user interface subsystem, to provide a first annotation identifying a first tissue anatomical region in a first image and to provide a second annotation identifying a second tissue anatomical region in a second image of the series of images.

[0063] Thus, the system may be configured to enable a user to manually define the first annotation and the second annotation. In such an embodiment, the annotation input data may be data generated by the processor subsystem as a result of the user annotation. In such an embodiment, the further data interface may be an internal data interface of the processor subsystem, such as a memory interface.

[0064] Optionally, the system is configured to generate annotation input data by applying an annotation technique to the first image and the second image to obtain the first annotation and the second annotation. The annotation technique may be an automatic annotation technique of a known type, but may also be a semi-automatic annotation technique, with the user being able to provide additional user input via the user interface subsystem.

[0065] Optionally, the system is part of a tissue dissection system or is configured to interface with a tissue dissection system.

[0066] It will be appreciated by those skilled in the art that two or more of the above-described embodiments, implementations, and / or alternative aspects of the invention may be combined in any way deemed useful.

[0067] Modifications and variations of any computer-implemented method and / or any computer program product, corresponding to modifications and variations of the corresponding system described, can be made by those skilled in the art on the basis of this description, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0069] Figure 1 Tissue material embedded in paraffin blocks to form tissue blocks is shown;

[0070] Figure 2 The sectioning of a tissue block to obtain a series of pathology sections is illustrated, wherein each pathology section contains a respective tissue portion;

[0071] Figure 3 A series of pathological sections are shown as an example, wherein the tissue in one pathological section is stained with H&E, and the tissue in the other pathological sections is not stained with H&E;

[0072] Figure 4 The annotation of H&E-stained pathology sections and the transfer of annotation to pathology sections not stained with H&E are illustrated;

[0073] Figure 5 A tissue block including a tissue strand is shown;

[0074] Figure 6The sectioning of a tissue block to obtain a series of pathology sections is illustrated, wherein each pathology section contains a respective tissue portion;

[0075] Figure 7 The annotation of several H&E-stained pathological sections is illustrated as a reference section when transferring the annotation to one or more extracted sections. The pathological section located between the two reference sections is not stained with H&E.

[0076] Figure 8 The transfer of annotations from multiple H&E-stained pathology sections to an intermediate pathology section that was not stained with H&E is illustrated;

[0077] Figure 9 A system for determining anatomical regions of tissue is shown;

[0078] Figure 10 A method of determining anatomical regions of tissue is shown; and

[0079] Figure 11 A computer-readable medium including non-transitory data is shown.

[0080] It should be noted that items with the same reference numerals in different figures have the same structural features and the same functions or the same signals. If the function and / or structure of such an item has been explained, there is no need to explain it repeatedly in the detailed description.

[0081] Reference Signs

[0082] The following list of reference numerals is provided to facilitate the explanation of the drawings and should not be construed as limiting the claims.

[0083] 10, 11 Organization blocks

[0084] 15 Cutting

[0085] 20, 21 Organizational Materials

[0086] 30-86 Pathological sections (* = tissue containing H&E staining)

[0087] 90-95% registration between paired images

[0088] 100-107 tissue sections (not stained with H&E)

[0089] 150, 151 tissue sections (H&E stained)

[0090] 200-203 Annotations representing anatomical regions of tissue

[0091] 250-254 Transfer Notes

[0092] 300 System for determining anatomical regions of tissue

[0093] 320 data interface

[0094] 340 processor subsystem

[0095] 342, 344 Internal Data Communications

[0096] 360 User Interface Subsystem

[0097] 370 Display Output

[0098] 380 User Input Interface

[0099] 400 Data Storage

[0100] 410 Image Data

[0101] 412 Comment output data

[0102] 420 Display

[0103] 422 Display data

[0104] 440 User Input Devices

[0105] 442 User input data

[0106] 500 Methods for determining anatomical regions of tissue

[0107] 510 Obtaining a first annotation in a first image

[0108] 520 Obtain a second annotation in the second image

[0109] 530 Generate annotations for intermediate images

[0110] 540 Determine the first set of registration parameters

[0111] 550 Determine the second set of registration parameters

[0112] 560 Propagate the first annotation to the intermediate image

[0113] 570 Propagate the second annotation to the intermediate image

[0114] 580 Combining the first and second comments propagated

[0115] 600 Computer readable medium

[0116] 610 Command Data DETAILED DESCRIPTION

[0117] Figure 1A tissue block 10 is shown containing tissue material 20, which may have been removed from a patient during surgery, a biopsy, or a similar procedure, and which may contain diseased tissue, such as a lesion, or any other type of tissue of interest. To obtain the tissue block 10, the tissue material 20 may be impregnated with paraffin and embedded therein. To enable pathological or histological examination of the tissue of interest, the tissue block may be sectioned, and the resulting tissue section or sections may be transferred to separate glass slides, which can then be examined not only by light microscopy but also digitized, for example, using a so-called digital slide scanner, known per se in the field of digital pathology. Figure 2 The tissue block 10 is slicing, and the resulting tissue slices are mounted on glass slides (both made of Figure 2 The arrow 15 in FIG. 1 shows the result obtained, i.e., a series of pathological sections 30-34. Figure 2 , each of a series of pathology sections 30 - 34 is shown as including a tissue portion 100 .

[0118] Figure 3 A series of pathology sections 40*-45 are illustrated, wherein the tissue in the first pathology section 40* is stained with hematoxylin-eosin (H&E). This type of staining is well known in the fields of pathology and histology and can be used to enhance the visibility of certain types of biological materials, which in turn facilitates identification of a tissue of interest, such as tissue section 150, or a portion of tissue section 150.

[0119] Figure 3 The remaining pathology sections 41-45 that were not stained with H&E are shown. Generally speaking, a series of pathology sections can include an H&E stained pathology section (also referred to as a "reference section" for reasons further described), and one or more pathology sections that are unstained or stained with Nuclear Fast Red (NFR) or with a similar stain that does not affect the polymerase chain reaction (PCR) process and / or can be removed before PCR. Pathology sections containing H&E stained tissue are also referred to here and elsewhere as "stained pathology sections" and are marked with an asterisk in the figure. Pathology sections containing unstained (in paraffin or not) or NFR stained tissue can be used for tissue dissection, which is also called tissue extraction, and the corresponding sections are called "extraction sections."

[0120] Figure 4Annotation of an H&E stained pathology slide 50* is illustrated. Such annotations can be made on the physical slide 50* itself, for example using a marker, but are now typically applied to a digitized form of the pathology slide, such as an image thereof. Therefore, any reference to annotations applied (or generated, transferred, etc.) to a pathology slide should be understood to include annotations applied (or generated, transferred, etc.) digitally to the digitized form of the pathology slide. Such digital annotations are known per se and can be performed by a user (e.g., a pathologist) by operating a digital pathology software package, or by a (semi-)automatic algorithm. It should also be noted that such digital annotations can also, in principle, be obtained by physically marking the pathology slide and digitizing the pathology slide and its annotations.

[0121] Generally speaking, the annotations 200 can be used to mark tissue of interest, and in many cases, to specifically mark anatomical regions of tissue. Figure 4 As illustrated in , the shape of the annotation 200 need not correspond to the shape of an anatomical structure, e.g., a tissue portion 151 that is or contains the tissue of interest, but may do so. For example, the annotation 200 may include a portion of the surrounding paraffin, or may extend beyond the paraffin, or may extend to an area that previously contained paraffin but has been removed prior to digitization of the tissue section.

[0122] Due to H&E staining, it may be preferable not to perform tissue extraction on the reference section 50*, but on one of the extraction sections. Figure 4 As indicated by the arrow 70 in FIG, the annotation 200 can be transferred from the reference slice 50* to the extracted slice 51 to obtain the annotation 250 in the extracted slice 51. It is known that the annotation is transferred by translating the digital annotation based on the image registration between the corresponding images of the two slices. That is, since the image registration can determine the correspondence between the image data of the reference slice 50* and the image data of the extracted slice 51, and thus determine the correspondence between the tissue portion 151 and the tissue portion 101, the annotation 200 can be transferred based on the correspondence of the underlying image data. Note that Figure 4 It is shown that annotation 250 is translated, rotated, and deformed compared to annotation 200, which is the result of image registration that also includes translation, rotation, and deformation components, for example, by components estimated as affine transformations. It should also be noted that reference slice 50* can be labeled as a "reference" slice by being the pathology slice of the initial annotation, for example, by representing a reference for transferring annotation 200 to one or more extracted slices.

[0123] Figure 5 Similar to Figure 1, but shows a tissue block 11 comprising a tissue bundle, whereby the tissue material 21 has a curled shape. The tissue of interest may often have such a complex shape, for example for biological reasons, but also when obtaining tissue sections from a tissue bundle, such as in the case of a prostate biopsy.

[0124] Figure 6 Similar to Figure 2 , which shows that the tissue block 11 was sectioned and the resulting tissue sections were mounted on glass slides. Figure 6 The arrow 15 in the figure indicates a series of pathological sections 60-65, each of which may contain a respective tissue section 102-107. Figure 6 and Figure 2 When the tissue is curled, the curly shape of the tissue causes the tissue portions 102-107 to have a significantly different appearance in each pathology slice 60-65, e.g., having different shapes. Consequently, image registration between pathology slices may be suboptimal, e.g., resulting in erroneous correspondences. This problem may be exacerbated when the tissue of interest is located in a homogenous tissue region. In this case, image registration cannot be guided by the tissue of interest itself, e.g., due to its different shape, nor by the surrounding tissue region, as the lack of spatial detail does not allow image registration to determine correspondences in the surrounding tissue region.

[0125] Thus, if a pathology slide contains H&E-stained tissue and is annotated so as to effectively represent the reference slide, transferring the annotation to the extracted slide may also be suboptimal because it may result in the transferred annotation not matching the tissue of interest in the extracted slide. Consequently, subsequent tissue dissection based on this annotation may be suboptimal because not enough tissue of interest may be extracted, or too much tissue of no interest may be present.

[0126] Figure 7 The following illustrates annotations for a plurality of H&E stained pathology sections according to some embodiments of the present invention. A series of pathology sections 70*-76* are shown, wherein sections 70*, 73*, and 76* are H&E stained pathology sections that contain respective annotations 201-203 and thus may represent reference sections. Such annotations may be provided by the user or as referenced by the user. Figure 9 As explained, the system can prompt the user to provide annotations on selected slices that may already be H&E stained or still require H&E staining, or use a (semi-)automatic algorithm to determine annotations on selected H&E stained slices. After obtaining annotations for reference slices, where the reference slices are separated in a series of pathology slices 70*-76* by the middle slices of the respective subseries (e.g., slices 71 and 72, and slices 74 and 75), the annotations can be performed as follows. Figure 8These annotations are transferred to the intermediate slices in the manner illustrated in , so that the tissue extraction region can be identified in each of said slices.

[0127] Further references Figure 7 ,Note that the transfer of annotations can involve at least two reference slices, such as the outer slices 70* and 76*, and optionally extend one or more intermediate reference slices, such as Figure 7 The slice 73* in the .

[0128] Figure 8 The transfer of annotations from reference slices 80*, 83* and 86* to extracted slices 81, 82, 84 and 85 is illustrated. As elsewhere, reference is made here to slices, but it will be understood that this applies to the digitized form of the slices, i.e. the corresponding images. The transfer may comprise the following steps, which may be applied separately (e.g., in parallel or sequentially) to each slice of the sub-series of intermediate slices. For example, for a first sub-series of intermediate slices 81, 82, annotations 251, 252 may be generated for the respective slices by registering each intermediate slice 81, 82 with each of the two nearest reference slices, i.e., the first reference slice 80* and the second reference slice 83*. Such "bidirectional" registration may involve direct registration between the respective slices, e.g., direct registration between the first reference slice 80* and the intermediate slice 81 and direct registration between the second reference slice 83* and the intermediate slice 81. Alternatively, as Figure 8 As also shown in , the registration of the intermediate slices with each reference slice can be performed by a registration chain. To this end, each pair of consecutive slices can be registered with each other, such as Figure 8 90-95 in the figure, which represent the registration between the respective pair of slices. The registration between the reference slices and the respective intermediate slices can then be obtained as a superposition of the individual registrations. For example, the intermediate slice 81 can be registered with the first reference slice 80* by registration 90, and with the second reference slice 83* by superposition of registration 91 and registration 92. Then, whether directly or through a registration chain, a first set of registration parameters and a second set of registration parameters can be generated for each intermediate slice 81, 82, the first set of registration parameters representing at least a partial registration of the intermediate slices 81, 82 with the first reference slice 80*, and the second set of registration parameters representing at least a partial registration of the intermediate slices 81, 82 with the second reference slice 83*.

[0129] The annotations in both reference slices 80*, 83* can then be propagated and thus transferred to the intermediate slices 81, 82 in the manner described elsewhere (also in Figure 8), and are then combined to obtain combined annotations 251, 252 in the respective intermediate slices 81, 82. The above process can also be performed on the second sub-series of intermediate slices 84, 85 in substantially the same manner, except that the second reference slice 83* and the third reference slice 86* are now used as the two nearest reference slices, thereby obtaining combined annotations 253, 254 for the intermediate slices 84, 85.

[0130] In general, annotations can be transferred in the manner described above by H&E staining the tissue of at least two pathology sections. In a series of pathology sections, these slices can be the first and last slice, or generally any two slices separated by a series of intermediate slices representing extracted slices. After digitization, the H&E stained slices can be annotated to identify the tissue anatomical region, thereby becoming a reference slice for the intermediate extracted slices. The annotations in the two reference slices can then be transferred to each or at least one of the intermediate extracted slices, which may involve a bidirectional registration, for example, from the intermediate slices "forward" to one reference slice and "backward" to the other reference slice. Such a bidirectional registration can involve the performance of two known image registrations, such as feature-based image registration, image registration using active contour models, image registration using shape context, or image registration using level set methods. After obtaining the forward and backward registrations, each annotation can be propagated to the intermediate slices by adjusting the shape and / or position of the annotations to be consistent with the underlying image data, thereby obtaining a "forward annotation" and a "backward annotation", the adjectives "forward" and "backward" here referring to the origin of the respective annotations in a series of pathological slices, for example, earlier or later in the series. The two annotations can then be combined, for example based on techniques such as distance transforms, shape contexts, or by morphological operations, for example using the overlap between the two annotations. A specific embodiment can be that the two annotations can be combined as a logical "and" (i.e., the intersection of the two annotations) or a logical "or" (i.e., the union of the two annotations).

[0131] In another specific embodiment, a signed distance transform can be applied to each transferred annotation to obtain two signed distance transformed images. The two signed distance transformed images can then be added together, and the combined annotation region can be determined as the region where the added pixel values ​​are positive.

[0132] If two annotations are too different, for example in terms of shape, the system generating the annotations (e.g. Figure 9system) may suggest staining the intermediate slice with H&E and annotating it. This slice can then be used as a reference slice, thereby reducing the average distance between reference slices in a series of pathology slices, thereby reducing the possibility that the two annotations differ too much. In general, a measure of fit quality can be used to evaluate the differences between annotations and / or between image data to determine whether the intermediate slice needs to be stained and annotated. In a specific embodiment, the fit quality can be expressed as the size of the deformation of the propagated first annotation and / or the propagated second annotation, for example, as the size of the non-rigid component of the transformation defined by a set of transformation parameters. Additionally or alternatively, the fit quality can be expressed as the difference in shape, or the overlap in surface area, between the propagated first annotation and the propagated second annotation. For example, the difference in shape can be quantified by the difference in the contours of the annotations.

[0133] refer to Figure 8 and other described techniques may be implemented by a suitably configured system or computer-implemented method.

[0134] Figure 9 A system 300 is shown for determining one or more tissue anatomical regions in a series of pathology sections using a series of images in digitized form representing the series of pathology sections. The system 300 includes a data interface 320 configured to access image data 410 of the series of images. Figure 9 In the embodiment of the present invention, the data interface 320 is shown as being connected to an external data storage 400 including image data 410. For example, the data storage 400 may consist of or be part of a picture archiving and communication system (PACS) of a hospital information system (HIS), the system 300 may be connected to or contained in the data storage 400, or may be any other type of data storage, such as a hard drive, an SSD, or a combination thereof. Thus, the system 300 may obtain access to the image data 410 through external data communication. Alternatively, the image data 410 may be accessed from an internal data storage (not shown) of the system 300. In general, the data interface 320 may take various forms, such as a network interface to a local area network (LAN) or a wide area network (WAN), such as the Internet, a storage interface to an internal or external data storage, and the like.

[0135] The system 300 is further shown as including a processor subsystem 340, which is configured to communicate internally with the data interface 320 via data communication 342. The processor subsystem 340 is further shown as communicating internally with a user interface subsystem 360 via data communication 344. The user interface subsystem 360 can be configured to enable a user to interact with the system 300 during operation, such as using a graphical user interface. The user interface subsystem 360 is shown as including a user input interface 380, which is configured to receive user input data 442 from a user-operable user input device 440. The user input device 440 can take a variety of forms, including but not limited to a computer mouse, a touch screen, a keyboard, a microphone, etc. Figure 9 The user input device shown is a computer mouse 080. Generally speaking, the user input interface 380 may be of a type corresponding to the type of user input device 440, ie, may be of a type of user device interface 380 corresponding thereto.

[0136] The user interface subsystem 360 is further shown to include a display output interface 370 configured to provide display data 422 to a display 420 for visualizing the output of the system 300. Figure 9 In the embodiment of FIG, the display is an external display 420. Alternatively, the display may be an internal display. Note that, instead of display output interface 370, user interface subsystem 360 may also include another type of output interface configured to present output data to the user in a sensory perceptible manner.

[0137] The processor subsystem 340 may be configured to establish a user interface enabling a user to annotate image data during operation of the system 300 and use of the user interface subsystem 360, and in particular to provide a first annotation identifying a first tissue anatomical region in a first image of a series of images, and a second annotation identifying a second tissue anatomical region in a second image of a series of images, wherein the first image and the second image are separated by a series of intermediate images in the series of images. Note that in some embodiments, the first annotation and the second annotation may be obtained (semi-)automatically by the processor subsystem 340. In this case, the role of the user changes and the user data (inputs and outputs, 442 and 422) may be adapted to the different roles. In general, manual, semi-automatic or automatic annotation may generate annotation input data that defines the first annotation and the second annotation in a computer-readable manner, which the processor subsystem 340 may access through another data interface (in Figure 9The processor subsystem 340 may further be configured to access the annotation input data in a manner described elsewhere (e.g., with reference to FIG. 1 ). For example, the other data interface may be a memory interface of the processor subsystem, or, in the case where the annotation input data is stored in the data memory 400, the other data interface may be the same as or of the same type as the data interface 320. Figure 8 and 10 ) generates annotations for one or more intermediate images in a series of intermediate images. The result may be annotation output data 412 stored in the data memory 400 and / or used by the tissue dissection system.

[0138] Despite Figure 9 Although not shown, in some embodiments, system 300 may include an interface to a tissue dissection system to control the tissue dissection system to dissect tissue from one or more pathological slices corresponding to the one or more intermediate images based on the one or more annotations generated for the one or more intermediate images. In other embodiments, system 300 may be part of such a tissue dissection system, for example, in the form of a subsystem thereof. Furthermore, in some embodiments, processor subsystem 340 may be configured to determine the quality of fit of the annotations in the intermediate images and, if the fit quality does not meet a fit quality criterion, prompt a user to provide a third annotation in a third image included in the series of intermediate images between the first image and the second image. For example, processor subsystem 340 may generate a screen warning via user interface subsystem 360 for display on display 420. In some embodiments, processor subsystem 340 may also indicate a recommended location for the third image in the series of images via user interface subsystem 360, for example, as an on-screen message specifying the location or by highlighting a visual representation of a particular slice on display 420. Generally speaking, the processor subsystem 340 can provide instructions to be executed to the user via the user interface subsystem 360 to provide a third annotation using the third reference image of the intermediate third reference slice. In some embodiments, once the newly stained slice is digitized again, the processor subsystem 340 can prompt the user to annotate the slice via the user interface subsystem 360. After obtaining the annotation, the processor subsystem 340 can then use the newly stained slice as a reference slice to generate an annotation for the extracted slice.

[0139] In general, the system 300 can be embodied as or in a single device or apparatus, such as a workstation or pathology system, such as a tissue dissection system or pathology system connected to such a tissue dissection system. The device or apparatus may include one or more microprocessors, such as a CPU and / or GPU, which together represent a processor subsystem and can execute appropriate software. The software may have been downloaded and / or stored in a corresponding memory, such as a volatile memory such as RAM, or a non-volatile memory such as Flash. Alternatively, the functional units of the system, such as a data interface, a user input interface, a display output interface, and a processor subsystem, can be implemented in the form of programmable logic in the device or apparatus, such as in the form of a field programmable gate array (FPGA). In general, each functional unit of the system can be implemented in the form of a circuit. Note that the system 300 can also be implemented in a distributed manner, for example, involving different devices or apparatuses. For example, the distribution can be in accordance with a client-server model, for example, using a server and a thin client. For example, computationally complex operations such as image registration and annotation propagation can be performed by one or more servers, such as one or more cloud-based servers or high-performance computing systems, while annotations can be generated on the client side, such as by a user operating a user input device connected to the client side. In such an embodiment, the processor subsystem of system 300 can be represented as a microprocessor of the distributed servers and / or clients.

[0140] Figure 10A method 500 is shown for identifying one or more histoanatomical regions in a series of pathological sections using a series of images representing a digitized form of the series of pathological sections. The method 500 is shown as including, in a step entitled "Obtaining a first annotation in a first image," obtaining 510 a first annotation identifying a first histoanatomical region in a first image in the series of images. The method 500 is further shown as including, in a step entitled "Obtaining a second annotation in a second image," obtaining 520 a second annotation identifying a second histoanatomical region in a second image in the series of images, wherein the first and second images are separated by a series of intermediate images in the series of images. The method 500 is further shown as including, in a step entitled "Generating annotations for intermediate images," generating 530 annotations for one or more intermediate images in the series of intermediate images by performing the following steps 540-580 for each of the one or more images. That is, step 530 is shown as including, in a step entitled "Determining a first set of registration parameters," determining 540 a first set of registration parameters representing at least partial registration of the intermediate image with the first image. Step 530 is further shown as including, in a step entitled "Determining a Second Set of Registration Parameters," determining 550 a second set of registration parameters representing at least a partial registration of the intermediate image with the second image. This step 530 is further shown as including, in a step entitled "Propagating the First Annotation to the Intermediate Image," propagating 560 the first annotation from the first image to the intermediate image using the first set of registration parameters. This step 530 is further shown as including, in a step entitled "Propagating the Second Annotation to the Intermediate Image," propagating 570 the second annotation from the second image to the intermediate image using the second set of registration parameters. Step 530 is further shown as including, in a step entitled "Combining the Propagated First and Second Annotations," combining 580 the propagated first and second annotations to obtain the annotation in the intermediate image.

[0141] It is understandable that, generally speaking, Figure 10 The steps of the method 500 may be performed in any suitable order, for example, serially, simultaneously, or a combination thereof, but where applicable, a specific order may be necessary, for example, due to input / output relationships. The method 500 may be implemented on a computer, as a computer-implemented method, dedicated hardware, or a combination of both. Figure 11 As further illustrated in FIG6 , computer instructions (e.g., executable code) can be stored on a computer-readable medium 600, for example, in the form of a series of machine-readable physical marks 610 and / or in the form of a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a transient or non-transitory manner. Examples of computer-readable media include memory devices, optical storage devices, cloud storage devices, and the like. Figure 11 An optical disc 600 is shown.

[0142] It should be noted that the embodiments described above illustrate rather than limit the present invention, and that those skilled in the art will be able to devise many alternative embodiments. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The use of the verb "comprise" and its variations does not exclude the presence of elements or steps other than those recited in the claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. When expressions such as "at least one" precede a list or group of elements, they indicate a selection of all elements or any subset thereof from the list or group. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, A and B, A and C, B and C, or all of A, B, and C. The present invention may be implemented by means of hardware comprising several different elements, as well as by means of a suitably programmed computer. In a device claim enumerating several means, several of these means may be implemented by the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A computer-implemented method (500) for determining one or more anatomical regions of tissue in a series of pathology sections (30-86) using a series of images representing a digitized version of the series of pathology sections, the method comprising: - obtaining (510) a first annotation (201) for identifying a first tissue anatomical region in a first image of the series of images; - obtaining (520) a second annotation (202), the second annotation (201) being used to identify a second tissue anatomical region in a second image of the series of images, wherein the first image and the second image are separated in the series of images by a series of intermediate images; as well as - generating (530) annotations (251, 252) for one or more intermediate images in the series of intermediate images by performing the following operations on each of the one or more images: - determining (540) a first set of registration parameters, said first set of registration parameters representing at least a partial registration of said intermediate image with said first image; - determining (550) a second set of registration parameters representing at least a partial registration of the intermediate image with the second image; - propagating (560) the first annotation from the first image to the intermediate image using the first set of registration parameters; - propagating (570) the second annotation from the second image to the intermediate image using the second set of registration parameters; as well as - combining (580) the propagated first annotation and the second annotation to obtain an annotation in said intermediate image.

2. The method (500) of claim 1, wherein: - the first set of registration parameters and the second set of registration parameters each define a deformable registration; as well as - Propagating the first annotation (201) to the intermediate image and propagating the second annotation (202) to the intermediate image each comprises deforming the respective annotation based on a respective set of registration parameters.

3. The method (500) of claim 1, further comprising - determining a quality of fit of said annotations in said intermediate image; and - if the quality of fit does not meet the fit quality criterion, prompting the user to provide a third annotation in a third image comprised in the series of intermediate images between the first image and the second image.

4. The method (500) according to claim 3, wherein prompting the user comprises indicating a position of the third image in the series of intermediate images or a position of a corresponding pathological slice in the series of pathological slices.

5. The method (500) according to claim 4, wherein - determining the quality of fit of the annotation in the intermediate image comprises determining a first quality of fit of the propagated first annotation and a second quality of fit of the propagated second annotation; and - The method further comprises determining the position of the third image based on a comparison of the first quality of fit and the second quality of fit.

6. The method (500) according to any one of claims 3 to 5, wherein: - said first image and said third image are separated in the series of images by an intermediate image of a first subseries; - said second image and said third image are separated in the series of images by an intermediate image of a second subseries; as well as Wherein, the method further comprises: - generating an annotation for one or more intermediate images of the first subseries of intermediate images based on the first annotation and the third annotation; and - generating an annotation for one or more intermediate images of the second subseries of intermediate images based on the second annotation and the third annotation.

7. The method (500) according to any one of claims 3 to 5, wherein determining the quality of the fit comprises determining a difference in shape, or an overlap in surface area, between the propagated first annotation and the propagated second annotation.

8. The method (500) of claim 2, further comprising: - determining a quality of fit of the annotation in the intermediate image by determining a magnitude of a deformation of the propagated first annotation and / or of the propagated second annotation; as well as - if the quality of fit does not meet the fit quality criterion, prompting the user to provide a third annotation in a third image comprised in the series of intermediate images between the first image and the second image.

9. The method (500) according to any one of claims 1 to 5, wherein the first set of registration parameters and the second set of registration parameters are determined by one of the following: - Feature-based image registration; - Image registration using active contour models; - Image registration using shape context; and -Image registration using level set methods.

10. The method (500) according to any one of claims 1 to 5, wherein the first image and the second image are hematoxylin-eosin (H&E) stained pathological sections.

11. The method (500) according to any one of claims 1 to 5, wherein the series of intermediate images are pathological sections unstained in paraffin, not stained in paraffin and / or stained with Nuclear Fast Red (NFR).

12. The method (500) according to any one of claims 1 to 5, further comprising controlling a tissue dissection system for dissecting tissue from one or more pathological slices corresponding to the one or more intermediate images based on the one or more annotations generated for the one or more intermediate images.

13. A non-transitory computer-readable medium (600) comprising data (610) representing instructions arranged to cause a processor system to perform a computer-implemented method according to any one of claims 1 to 5.

14. A system (300) for determining one or more anatomical regions of tissue in a series of pathology sections (30-86) using a series of images representing the series of pathology sections in digitized form, the system comprising: - a data interface (320) for accessing the series of images; - another data interface (360) for accessing annotation input data, said annotation input data defining: - a first annotation (201) identifying a first tissue anatomical region in a first image of the series of images, and - a second annotation (202) identifying a second tissue anatomical region in a second image of the series of images, wherein the first image and the second image are separated in the series of images by a series of intermediate images; as well as - a processor subsystem (340) configured to: - generating annotations (251, 252) for one or more intermediate images in the series of intermediate images by performing the following operations on each of the one or more images; - determining a first set of registration parameters, said first set of registration parameters representing at least a partial registration of said intermediate image with said first image; - determining a second set of registration parameters, said second set of registration parameters representing at least a partial registration of said intermediate image with said second image; - propagating the first annotation from the first image to the intermediate image using the first set of registration parameters; - propagating the second annotation from the second image to the intermediate image using the second set of registration parameters; and - combining the propagated first annotation and the second annotation to obtain the annotation in the intermediate image.

15. The system (300) of claim 14, wherein the system is part of a tissue dissection system or is configured to interface with a tissue dissection system.

Images