Generate annotation data for tissue images

Abstract

Currently, there is an interest in applying machine learning techniques to automatically analyze digital pathology images. Machine learning techniques often rely on training with real-world image input. The quality and amount of the training data determines the quality of the detector as expressed in the ratio of true and false positives and the robustness to changes in the appearance of the input image. The present application proposes obtaining image data of the same sample before and after at least one re-staining step (first with a structure-revealing stain and secondly with a biomarker-revealing stain). Portions of the first and second image data that have a good registration relationship are selected, and the probability of detecting a desired candidate object (such as a nucleus) and the probability of a biomarker-revealing stain presenting annotation data suitable for training a machine learning algorithm on the first and / or second image data is provided.

Description

Technical Field

[0001] The present invention relates to an apparatus for automatically generating annotation data for tissue images, and a related computer-implemented medical image annotation method, a method for generating annotation data for tissue images, a method for training a machine learning model, and a computer program unit, a computer-readable medium, and a computer data structure. Background Art

[0002] Currently, there is interest in applying machine learning techniques to automatically analyze digital pathology images. Machine learning techniques often rely on training with real-world image input. The quality and amount of training data determines the quality of the detector, as expressed by the ratio of true and false positives, and the robustness to changes in the appearance of the input images.

[0003] Ground truth training images (or image patches) are typically provided from manually annotated images annotated by pathologists. Although this may be feasible for larger regions of pathology images (such as tissue types), annotating a large number of cellular objects is time consuming as typically thousands of objects need to be annotated.

[0004] US2017 / 0103521 A1 discusses a method for identifying biomarker-positive tumor cells.

[0005] A. Can et al.: "Techniques for Cellular Analysis for Lifescience Applications", in "Analytical Chemistry" (in "American Chemical Society", June 1, 2018, Volume 89, Pages 1-29) provides an overview of the technique and summary description of several steps required for quantification of multi-tissue-based biomarkers, including tissue staining with target-specific antibodies labeled with fluorescent agents, image capture, preprocessing, registration, segmentation, and subcellular quantification. It is described that this provides the flexibility to quantify biomarkers in more than one cellular compartment, thereby maximizing the amount of data present in a tissue image and enabling a broader analysis of the role of biomarkers in predicting response to treatment and patient survival.

[0006] However, such methods can be improved. Summary of the invention

[0007] It would be advantageous to have an improved method for automatic annotation of organizational images.

[0008] According to a first aspect, an apparatus for automatically generating annotation data for a tissue image is provided.

[0009] According to a second aspect, a computer-implemented medical image annotation method for automatically generating annotation data for a tissue image is provided.

[0010] According to a first embodiment, there is provided an apparatus for automatically generating annotation data of a tissue image, comprising: an input unit and a processor.

[0011] The input unit is configured to: acquire first image data, the first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure; and acquire second image data, the second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample.

[0012] The processor is configured to: detect multiple candidate objects in the first image data, and corresponding multiple detection probabilities for the detected candidate objects; generate staining confidence data for the second image data, the staining confidence data indicating the likelihood that the corresponding portion of the tissue sample includes the biomarker targeted by the second stain; and generate image annotation data including representations of the multiple candidate objects, wherein individual candidate objects among the multiple candidate objects are annotated to indicate the presence of the biomarker in the corresponding portion of the second image data if the individual candidate objects reach a combined confidence level based on the corresponding multiple detection probabilities for the detected candidate objects, the corresponding staining confidence data, and a corresponding alignment quality threshold between the first image data and the second image data of the candidate objects.

[0013] The effect is that a reliable training data set that can be used as input to a machine learning algorithm can be provided much more easily. In the case of applying such a data processing method to the first and second stained images, a single re-staining step of the tissue sample is necessary to provide ground truth data for the training of the machine learning algorithm (although more re-staining steps can be provided). In addition, since the generation of the image annotation data is based on a corresponding registration quality threshold between the first and second image data of the candidate object, image annotation data is not generated for patches with poor registration. Therefore, the training data set can be generated with much higher computational efficiency (for example, this means that it can be implemented with much lower computational latency or with much less advanced computing hardware).

[0014] Optionally, the processor is further configured to generate a registration quality map between the first image data and the second image data, wherein the processor is configured to annotate a candidate object among the plurality of candidate objects to indicate the presence of a biomarker when the registration quality map indicates that the registration between corresponding locations of the candidate object in the first image data and the second image data exceeds a registration quality threshold.

[0015] The effect is that if the location of the candidate object is poorly registered between the first and second images, the computationally expensive operation of generating annotations for the candidate object does not occur, thereby increasing computational efficiency.

[0016] Optionally, the processor is further configured to generate the image annotation data for the individual candidate objects among the plurality of candidate objects, the image annotation data comprising a positive annotation for a first candidate object among the plurality of candidate objects and a negative annotation for a second candidate object among the plurality of candidate objects.

[0017] The effect is that the generated image annotation data is ground truth data that can be provided as suitable for providing reliable input into a machine learning based detector.

[0018] Optionally, the processor is further configured to generate the staining confidence data by: using the second image data to generate a histogram of staining confidence, and providing a plurality of staining confidence results corresponding to the positions of the plurality of candidate objects in the first image data as the corresponding staining confidence data.

[0019] The effect is that the decision between positive and negative candidate objects in the second image data has a high accuracy because it contains staining information from more objects.

[0020] Optionally, the processor is configured to generate the staining confidence data by: generating a first channel of the second image data for use as a positive control, generating a second channel of the second image data for use as a negative control. The processor is also configured to generate a histogram of staining confidence, including generating a first histogram channel using the first channel of the second image data, generating a second histogram channel using the second channel of the second image data, and generating the histogram of staining confidence using the first histogram channel and the second histogram channel.

[0021] The effect is that a histogram of pixel colours (DAB channel values) of the second image data may be plotted to provide a transformation curve to match the staining values ​​to staining confidence values, which improves the accuracy of the generated image annotation data.

[0022] Optionally, the processor is configured to detect a plurality of candidate objects in the first image data using a kernel detector, and to detect a corresponding plurality of detection probabilities for the detected candidate objects.

[0023] The effect is that the annotations of candidate objects in the generated image annotation data can be weighted based on their likelihood to identify a cell nucleus. This means that the location of a cell nucleus revealed in the first image data can be spatially correlated with a corresponding marker staining in the second image data. In the generated annotation data, less emphasis can be placed on marker staining in the second image data that has a poor likelihood of identifying a cell nucleus, for example, thereby improving the reliability of the generated image annotation data.

[0024] The processor is configured to: segment the multiple candidate objects in the first image data, transfer the segmentation of the multiple candidate objects in the first image data to the second image data based on the alignment between the first image data and the second image data, and sum the corresponding staining confidence data in the positions of the multiple candidate objects in the second image data.

[0025] Optionally, the device further comprises an output unit.

[0026] The output unit is configured to display the image annotation data on a graphical user interface, optionally in combination with the first image data and / or the second image data.

[0027] The effect is that the generation of image annotation data can be easily monitored by a user of the apparatus and compared with the original first and second image data to provide independent verification of the reliability of the generated training data.

[0028] According to a second embodiment, there is provided a computer-implemented medical image annotation method for automatically generating annotation data for a tissue image, comprising:

[0029] a) acquiring first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure;

[0030] b) acquiring second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample;

[0031] c) detecting a plurality of candidate objects in the first image data, and a plurality of corresponding detection probabilities of the detected candidate objects;

[0032] d) generating staining confidence data for the second image data, the staining confidence data representing a likelihood that the corresponding portion of the tissue sample includes the biomarker targeted by the second stain; and

[0033] e) generating image annotation data comprising representations of the plurality of candidate objects, wherein individual candidate objects among the plurality of candidate objects are annotated to indicate the presence of a biomarker in a corresponding portion of the second image data if the individual candidate objects reach a combined confidence level based on the corresponding plurality of detection probabilities of the detected candidate objects, the corresponding staining confidence data, and a corresponding registration quality threshold between the first image data and the second image data of the candidate objects.

[0034] Optionally, the first stain is a hematoxylin and eosin (H&E) stain and the second stain is an immunohistochemistry (IHC) stain.

[0035] According to a third aspect, there is provided a method for generating annotation data for a tissue image, comprising:

[0036] a1) Collect tissue samples from patients;

[0037] a2) performing a first staining protocol on the tissue sample using a first stain;

[0038] a3) using a digital microscope imager to acquire first image data of the tissue sample;

[0039] a4) performing a washing protocol on said tissue sample to substantially remove said first stain;

[0040] a5) performing a second staining protocol on the tissue sample using a second stain; and

[0041] a6) acquiring second image data of the tissue sample using the digital microscope imager; and

[0042] A computer-implemented medical image annotation method for automatically generating annotation data for a tissue image according to the second aspect is performed.

[0043] The effect is that a large amount of image annotation data can be automatically generated by performing only one sample re-staining step (although the technique can be applied to third, fourth, and further input image data of the re-stained sample).

[0044] According to a fourth aspect, there is provided a method for training a machine learning model, comprising:

[0045] f1) collecting image annotation data generated according to the second aspect;

[0046] f2) training a machine learning model using the image annotation data; and

[0047] f3) Output the trained machine learning model.

[0048] The effect is that since it is possible to produce the generated image annotation data using a much larger range of sample input data, a machine learning model trained using the generated image annotation data has a higher accuracy.

[0049] According to a fifth aspect, a computer program element is provided, which, when executed by a processor, is adapted to perform the method of the first aspect or the fourth aspect.

[0050] According to a sixth aspect, there is provided a computer readable medium storing the computer program element of the fifth aspect.

[0051] According to a seventh aspect, a computer data structure comprises image annotation data generated according to a method according to the first or fourth aspect as claimed.

[0052] In the following application, the term "annotation data" refers to metadata describing the properties of the input image data. For example, the annotation data may include a list of point coordinates in the first and / or second input image data, wherein each item in the list is marked with a logical "1" or a logical "0", which defines that the pixel at the given point coordinate meets the detection criteria or does not meet the detection criteria. For example, the annotation data may define that a particular pixel represents a region of candidate nuclei having a strong match for a given biomarker test. Of course, the annotation data may define any region of image data that meets the conditions. In addition, the annotation data may be probabilistic rather than binary. In one option (suitable for display to a human operator), the annotation data is displayed on the output image as a colored region, but this is not necessary. The annotation data may be provided to define several conditions of reference coordinates in the input image data, and these conditions may be independent or mutually exclusive. The automated machine learning algorithm may read the point coordinates of the annotation data defining the region of cells that meet the specific biomarker conditions, and refer to the input image data of the image to which the annotation data refers. The automated machine learning algorithm may then import the portion of the input image data defined as meeting the specific biomarker conditions as an example of an input image that meets the specific biomarker conditions. The automated machine learning algorithm can then engage in training operations with reference to input image data that meets specific biomarker conditions will cease in this way, and the annotation data can be used in combination with the image data to provide more accurate training.

[0053] In the following applications, the "first image data" and the "second image data" are digital representations of images of biological samples, such as histopathological samples obtained using, for example, digital bright field microscopy, digital fluorescence microscopy, etc. Therefore, the first and second image data can be directly acquired from at least the digital microscope, or can be downloaded from a PACS image storage system.

[0054] In the following applications, the term "candidate object" can refer to a region of tissue in a pathological sample, an individual cell, a cell nucleus, a tubule, a specific type of cell such as a lymphocyte, an erythrocyte, a monocyte, a dendritic cell, a microglia. The annotation method discussed in this application has a wide application based on a specific staining technique applied to a precursor tissue sample.

[0055] In the following application, the term "staining confidence data" reflects the probability that a stain exists at a specific pixel or arbitrary region of the second input image data. A staining confidence map is an image corresponding to the second image data showing the probability that a specific stain exists in the second image data.

[0056] In the following applications, a "stain for indicating tissue structure" is, for example, a conventional hematoxylin and eosin (H&E) stain, but other stains that enable identification of tissue structure may be applied, such as periodic acid Schiff's (for kidney or liver biopsies), Perls' Prussian Blue Iron (for liver staining), Gomori's Trichomes (for muscle and collagen structure), Alcian Blue (for atherosclerosis detection). The skilled artisan will appreciate that many types of tissue structure revealing stains may be applied.

[0057] In the following application, "a stain for revealing the presence of a biomarker" may, for example, refer to an immunohistochemistry (IHC) marker. Examples of such markers are CD3, CD20, CD30, CD3, RCC, keratin, protein). The skilled person will appreciate that many types of biomarker revealing stains may be applied.

[0058] The following applications refer to techniques that can be applied in histochemical staining.

[0059] Thus, the idea of ​​the invention is to obtain image data of the same sample before and after at least one re-staining operation (firstly with a structure revealing stain and secondly with a biomarker revealing stain). Selecting portions of the first and second image data that have a good registration relationship and that provide a probability of detecting a desired candidate object (such as a nucleus) and a probability of a biomarker revealing stain presents annotation data suitable for training a machine learning algorithm on the first and / or second image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Exemplary embodiments of the present invention will be described hereinafter with reference to the following drawings:

[0061] Figure 1 An apparatus according to the first aspect is schematically illustrated.

[0062] Figure 2 The application of the method according to the second aspect is schematically illustrated.

[0063] Figure 3 Shown is an experimentally obtained example of a registration quality map superimposed on the original H&E image.

[0064] Figure 4 a) Schematic representation of a slide obtained from an experiment for a negative control.

[0065] Figure 4 b) Schematic representation of a slide obtained from an experiment for a positive control.

[0066] Figure 4 c) Schematic representation of the separated DAB channel obtained for the negative control experiment.

[0067] Figure 4 d) Schematic representation of the separated DAB channel obtained from the experiment for the positive control.

[0068] Figure 5 a) illustrates an experimentally obtained histogram of DAB channel values ​​following color deconvolution.

[0069] Figure 5 b) illustrates the use Figure 5 a) Transformation curve defined by the experimentally obtained histogram.

[0070] Figure 6 Illustration of an experimentally acquired DAB channel of a region of interest of a biomarker staining image.

[0071] Figure 7 The generated staining confidence values ​​for the DAB channel corresponding to the biomarker staining image are illustrated.

[0072] Figure 8 An experimentally obtained staining confidence map including labeling within and around the cell nucleus border obtained according to the method of the present application is shown.

[0073] Fig. 9 a) Schematic representation of H&E tissue images obtained experimentally.

[0074] Fig. 9 b) Fig. 9 a) Line representation of nuclear segmentation data of experimentally obtained H&E tissue images is shown at the same scale and origin.

[0075] Fig. 9c) Schematic representation of IHC tissue images obtained from experiments including CD3 / 20 stain.

[0076] Fig. 9 d) with Fig. 9 c) Line representation of nuclear staining markers of experimentally obtained IHC tissue images are shown at the same scale and origin.

[0077] Fig.10 A pathology image including an overlay of annotation data generated according to example techniques discussed in this application is illustrated.

[0078] Fig.11 a) shows an IHC staining pathology image obtained from another experiment.

[0079] Fig.11 b) The diagram shows Fig.11 a) Same scale and origin Fig.11 a) Annotation data in the form of line overlays of the image.

[0080] Fig.12 A method according to the fourth aspect is schematically illustrated.

[0081] Fig.13 An apparatus for training a data model using annotation data obtained according to the techniques described herein to generate a computer data structure comprising image annotation data according to the seventh aspect is schematically illustrated. DETAILED DESCRIPTION

[0082] Establishing large annotated datasets for training and validating image analysis algorithms used in computational pathology is an important task. For deep learning, the availability of large annotated datasets is important for the success of such algorithms in this case, and the datasets can involve hundreds or thousands of pathology slides. Applying computational recognition processes to such large image datasets may involve excessive computer time. In the case where individual units in the dataset must be identified and classified (annotated), such large datasets will be difficult to accurately manipulate using human annotators.

[0083] Therefore, it has been proposed to annotate large data sets using biomarker staining. In this method, biomarkers that are specifically bound to the cell or tissue type that needs to be annotated are used. By analyzing the image data of such biomarker-stained tissues, annotation masks (annotation data) can be efficiently created by computer processing methods. For example, the annotation data can then be used as input for training algorithms on H&E images. However, this technique requires near-perfect alignment between the image in the H&E image and the biomarker-stained image. If there is no near-perfect alignment between the H&E image and the biomarker-stained image, the annotation data will not correctly train the deep learning algorithm.

[0084] Maintaining high tissue quality is important for image alignment, and extensive tissue re-staining cycles are undesirable. Therefore, there is a need to overcome this limitation.

[0085] It is suggested to use so-called re-staining so that the same tissue slide can be used to enable removal of the H&E stain and then application of the biomarker stain. According to this technique, misalignment between the H&E staining image and the biomarker staining image (for example) is minimized.

[0086] The use of a single re-staining step for providing annotation data (true state data) for training deep learning algorithms suggests the use of biomarkers that stain a class of objects that is a subset of the initial set of candidate objects. A possibility map is generated to indicate the probability that the candidate object belongs to the staining class (class of interest). The possibility map is based at least on the registration quality (between the tissue structure image and the biomarker image), the probability of the initial detection result (of the candidate object in the tissue structure image), and the probability of the biomarker presence (in the biomarker image). The possibility map can be generated based on any portion size of the input image and can refer to, for example, a cell, a nucleus, or an area of ​​arbitrarily placed patches in the image. Although it is noted above that a single re-staining step for providing annotation data is preferred, the present invention is not so limited, and a second, third, fourth, and subsequent re-staining step can be provided, such as targeting different biomarkers, but there is the potential for subsequent re-staining steps to enjoy lower registration quality.

[0087] Based on the generated likelihood map, positive and negative samples can be selected to form a training set for a deep learning algorithm.

[0088] Hence, according to a first aspect, an apparatus 10 for automatically generating annotation data for a tissue image is provided.

[0089] Figure 1 An apparatus 10 according to the first aspect is illustrated. The apparatus comprises an input unit 12 and a processor 14.

[0090] The input unit 12 is configured to acquire first image data 18 including a representation of a tissue sample stained with a first stain for indicating tissue structure and to acquire second image data 20 including a representation of a tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample.

[0091] The processor 14 is configured to: detect a plurality of candidate objects in the first image data 18, and a corresponding plurality of detection probabilities for the detected candidate objects; generate staining confidence data for the second image data 20 representing a likelihood that a corresponding portion of the tissue sample includes a biomarker targeted by a second stain; and generate image annotation data 22 including a representation of the plurality of candidate objects, wherein individual candidate objects of the plurality of candidate objects are annotated to indicate the presence of the biomarker in the corresponding portion of the second image data if the individual candidate objects reach a combined confidence based on the corresponding plurality of detection probabilities for the detected candidate objects, the corresponding staining confidence data, and a corresponding registration quality threshold between the first and second image data of the candidate objects.

[0092] Optionally, the device 10 comprises an output unit 16 .

[0093] An exemplary implementation of the apparatus 10 is to use a personal computer (PC). In this example, the PC acquires the first image data 18 and the second image data 20 from a computer network (e.g., a hospital's PACS system), and in this way, the input unit 12 can be a network interface. The processor 14 is a combination of a general-purpose PC data processor 14a (such as an Intel i5 (TM) processor), a random access memory (RAM) 14b, and an optional graphics processing unit (GPU) image processing accelerator, and a hard disk drive (14d). The processor 14 can be configured to load a program configured to implement the present invention according to program instructions stored on the hard disk drive 14d loaded into the RAM 14b, and execute the program according to the program instructions. The data processor 14a can acquire the first image data 18 and the second image data 20 from the input unit 12, the RAM 14b and / or the hard disk drive 14d, and generate annotation data that can be stored on the hard disk drive 14d and transmitted via the output unit 16 (e.g., to a GUI displayed on a monitor, to an external PACS server through a network interface, etc.). However, it will be appreciated by the skilled person that this is merely an example implementation and that the apparatus 10 may also be implemented as a large-scale computer cluster, optionally accelerated using a GPU array and / or FPGA and / or, for example, a dedicated deep learning processing array.

[0094] Figure 2 The data processing performed using the apparatus 10 for generating annotation data is schematically illustrated.

[0095] Provided are first image data 24a comprising a representation of the tissue structure of a tissue sample (typically by an H&E staining process) and second image data 24b comprising a representation of the same sample tissue stained with a stain for revealing the presence of biomarkers in the tissue sample. The image data may be in a proprietary or open source image format. For example, an image file associated with a 20x scan of a 20mm x 15mm tissue sample may be approximately 4GB in size, optionally compressed using a format such as JPEG2000.

[0096] This example is explained using an example of a detector trained to detect a broad class of lymphocytes based on H&E stained nuclear detectors, including T lymphocytes (stained using the CD-3 marker), however it will be appreciated that many alternative combinations of stains may be used.

[0097] Figure 2 Step 1 of illustrates the registration of first image data (e.g., H&E image) with second image data (e.g., IHC image). The images are registered using an automatic registration algorithm known to those skilled in the art. The images can be registered for mutually corresponding patches or, for example, between any calculated portions of the first and second images.

[0098] Briefly turn to Figure 3 , shows an experimentally obtained example of a registration quality map superimposed on an original H&E image. Portion 23 is highlighted as an area with relatively low registration quality, while region 25 is illustrated as an area with relatively good registration quality. Therefore, the square patch in portion 23 is excluded from the annotation data generation process. It will be appreciated that the generation of an image-based registration quality map is not substantial, and that registration quality information of the coordinates of the image data is stored in many ways.

[0099] Figure 2 Step 2 of FIG. 1 illustrates the registration quality metric Q for the coordinates L1 and L2 of the registration patch T1. L1 and Q L2 (when compared to the second image data). In this case, the combined registration quality of the pixels in the patches between the same portion of the first and second image data is reasonably good, and a patch pair T1 and T2 in the first and second image data, respectively, is selected that can be used in subsequent data processing. By rejecting patches with poor registration quality, the final annotated dataset is more accurate, resulting in more accurate training of the algorithm using the annotated dataset. Furthermore, performing the present invention on corresponding patches of the first and second image data with poor registration is a waste of computational effort.

[0100] Figure 2Step 3 of illustrates the generation of staining confidence data using the second image data. For example, a portion (patch) T2 that is considered to be well registered to a corresponding portion (patch) T1 in the first image data is analyzed to generate staining confidence data.

[0101] In this example, the second image data may include an IHC image with a CD marker. Thus, a staining confidence map is generated that reflects the probability of the presence of a specific biomarker marker in portion T2 of the IHC image. For example, the staining confidence map is a transform of the color deconvolved DAB channel image.

[0102] Briefly go to Figure 4 , the staining confidence map is generated by mapping staining values ​​in a portion T2 of the second image data corresponding to T1 in the first image data to confidence values. Optionally, the confidence values ​​may be on a scale between 0 and 1, but it will be appreciated that any numerical value may be applied. For example, DAB deconvolution is used to generate Figure 4 A negative control (NC) slide is shown, and Figure 4 b) shows the slide of the positive control (PC). As expected, Figure 4 The isolated DAB channel of the negative control shown in c) is almost blank, and Figure 4 The separated DAB channel of the positive control shown in d) occurs over regions expressing biomarkers for the presence of the condition being tested. Therefore, the pixel values ​​of the positive and negative control slides are used to calibrate the mapping curves used to generate the staining confidence map.

[0103] exist Figure 5 An example of such a calibration process is illustrated in a). A histogram of DAB channel values ​​has three curves plotted. The x-axis of the histogram 30 represents the color value bins of the DAB channel, and the y-axis 32 of the histogram represents the number of pixels in the portion (or patch) of the second image data that is present in the associated color value bin of the DAB channel.

[0104] The first histogram 34 represents a histogram of positive control values. The second histogram 36 represents a histogram of negative control values. Figure 5 b) illustrates a staining confidence map with a transformation curve 38 indexed to the pixel bin axis 30 of the histogram. The transformation curve 38 is generated to match the staining values ​​of the deconvolution channel of the second image data to a staining confidence value between 0 and 1. Of course, this is an example of a transformation curve for generating a distinction between positive control values ​​and negative control values ​​for processing the second image data. Optionally, the transformation curve 38 may include a transition region where pixels are not classified as positive or negative control values. This has the effect of reducing the noise caused by the pixel values ​​near the transition boundary between the positive control value and the negative control value.

[0105] Figure 6 Experimentally acquired DAB channel data for a region of interest of a biomarker staining image is shown. Figure 7 The pixel values ​​are mapped to Figure 5 b) After the dye confidence function Figure 6 DAB channel data. Therefore, Figure 7 A "heat map" of regions having a low probability of representing a stained portion of the second image data and regions having a high probability of representing a stained portion of the second image data is provided.

[0106] Figure 8 An experimentally obtained IHC image including staining confidence markers within and around cell nucleus boundaries obtained according to the methods of the present application is illustrated.

[0107] Fig. 9 a) Schematic representation of H&E tissue images obtained experimentally.

[0108] Fig. 9 b) Fig. 9 a) Line representation of nuclear segmentation data of experimentally obtained H&E tissue images is shown at the same scale and origin.

[0109] Fig. 9 c) Schematic representation of IHC tissue images obtained from experiments including CD3 / 20 stain.

[0110] Fig. 9 d) with Fig. 9 c) Line representation of nuclear staining markers of experimentally obtained IHC tissue images are shown at the same scale and origin.

[0111] The nuclear segmentation from the corresponding H&E image (first image data) is transferred to Figure 8 The corresponding position in the IHC (second image data). Figure 8 It is possible to distinguish positions 42 with relatively weak staining (indicative of negative samples) and positions of nuclei with relatively strong staining 44 indicative of positive samples. Figure 8 The positions of the markers were examined within and around the nuclear borders as indicated in .

[0112] Reference again Figure 2 , step 2, detecting a plurality of candidate objects in the first image data. Optionally, a kernel detector is used to detect a plurality of candidate objects in the first image data. For example, the kernels may be segmented using a multi-reference level set (MRL) algorithm, a random forest algorithm, or a graph cut algorithm, or any other machine learning based training algorithm. A skilled person will be able to apply other forms of kernel detector algorithms. In this example, the kernel detector is applied to patch T1. Based on Figure 2Based on the registration result calculated in step 1, the corresponding nuclear segmentation coordinates are transferred to the patch T2 of the second image data. This provides the correspondence between the nuclei detected in the tissue structure image (first image data) and the tissue biomarker image (second image data).

[0113] In this example, the staining confidence values ​​in and around each detected cell nucleus in the first image data are summed. Combined with the nuclear detection confidence value (an example of a detection probability for a detected candidate in the first image data), a likelihood score value is defined for the candidate nucleus to be positive for the biomarker targeted in the second image data (in this case, CD3 / 20 positive).

[0114]

[0115] Equation (1) defines the integral of the staining likelihood value over the nuclear area, divided by the total area of ​​the nucleus. This integral is added by the nuclear likelihood value. The sum is then divided by two to obtain the average (integral) value for the total likelihood of the nucleus as being positive for the marker.

[0116] In the above, the term "nucleus" means the area within and around each nuclear outline. It will be appreciated that the above expression is a figure of merit for determining the likelihood score of an individual candidate nucleus as positive for a given biomarker. However, many expressions can be derived. In variations, only the nuclear staining itself can be considered. In cases where the candidate of interest is not a nucleus or a nuclear region, a different segmentation approach may be required. For example, detecting fibrous structures in lung tissue may require a different segmentation algorithm.

[0117] return Figure 2 ,The fourth step involves selecting candidate objects based on the corresponding staining confidence data. In this example, the corresponding staining confidence data is the nuclear level likelihood score, but different staining protocols with different corresponding staining confidence data will be applied.

[0118] In order to select the positive and negative samples for the detector for foreground training, we select samples with sufficiently high corresponding staining confidence (P CD) are selected from candidate objects (nuclei) with corresponding staining confidence. For example, the corresponding staining confidence of the candidate object can be higher than a predefined threshold. Multiple candidate objects of interest in a patch of the first image data T1 with staining confidence data higher than the first threshold can be optionally marked as, for example, positive samples. This is an example of a first annotation data type. Optionally, candidate objects in a patch of the first image data T1 with low corresponding staining confidence (less than a second staining correspondence threshold) are marked as, for example, negative samples. This is an example of a second annotation data type. Optionally, candidate objects in a patch of the first image data T1 with uncertain corresponding staining confidence can be marked as, for example, uncertain samples. This is an example of a third annotation data type. Therefore, the annotation data can include multiple different types of annotations, each with different biological significance.

[0119] Thus, generated annotation data comprising one or more of the first, second and third annotation data types has been generated. It will be appreciated that the annotation data need not be provided as an image, but may be provided as a data structure having a list of image coordinates, and associated annotations in a particular order. Optionally, the annotation data may define an annotation type for a portion of an image. In other words, the annotation data may be interpreted with reference to the corresponding image and superimposed on the corresponding image (and for example the first or second image data) to enable the annotation to be displayed to a human observer. In practice, the generated annotation data is supplied to a machine learning algorithm in combination with the first and / or second image data that has been used to generate the annotation data, so that the machine learning algorithm can use the first and / or second image data to improve its model.

[0120] Fig.10 An example of an experimental acquisition of an IHC stained image is illustrated, with a staining confidence map for annotating regions of the image with a staining confidence greater than a certain threshold. For example, region 50 indicates a weakly stained region in the IHC image (second image data), which has therefore not been marked in the staining confidence map. Region 52 is more strongly stained and has been marked in the staining confidence map.

[0121] Fig.11 a) shows an IHC staining pathology image obtained from another experiment.

[0122] Fig.11 b) Fig.11 a) Same scale and origin as shown Fig.11 a). The annotation data includes a large box 53 representing a positive sample and smaller dots representing a negative sample, for example. For example, the area indicated by the outline 51 is optionally marked as "unknown".

[0123] Thus, the process can be fully automated, for example into batch mode, prior to processing. The user of the system only has to select a batch of slide image pairs (eg, H&E and IHC), and the system will automatically CD mark the true location of the verified lymphocytes.

[0124] The apparatus 10 optionally includes an output unit 16. The output unit 16 is capable of outputting the generated annotation data. One output data format is, for example, a data structure comprising a list of pixel positions of annotated pixels. More complex data formats include hierarchical data structures containing lists of pixel positions, and whether the pixel positions have been annotated using, for example, a specific annotation type. Optionally, the output unit 16 is configured to interpret the output data structure and generate an overlay of the pixel annotation onto the first and / or second image data. Therefore, the output unit 16 may include an output display such as a monitor, and the output unit 16 may generate a display signal to display the annotated output image to the user. Therefore or in combination, the output unit 16 may be configured to transfer the generated annotation data to a PACS server or archive, optionally together with the first and / or second image data, for future use and training of a machine learning algorithm.

[0125] According to a second aspect, there is provided a computer-implemented medical image annotation method for automatically generating annotation data for a tissue image, comprising:

[0126] a) acquiring 70 first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure;

[0127] b) acquiring 72 second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in said tissue sample;

[0128] c) detecting 74 a plurality of candidate objects in the first image data, and a corresponding plurality of detection probabilities of the detected candidate objects;

[0129] d) generating 78 stain confidence data for the second image data representing a likelihood that the corresponding portion of the tissue sample includes the biomarker targeted by the second stain; and

[0130] e) generating 80 image annotation data comprising representations of a plurality of candidate objects, wherein individual candidate objects in the plurality of candidate objects are annotated to indicate the presence of a biomarker in a corresponding portion of the second image data if the individual candidate objects reach a combined confidence level based on a corresponding plurality of detection probabilities for the detected candidate objects, corresponding staining confidence data, and a corresponding registration quality threshold between the candidate objects' first and second image data.

[0131] Optionally, the first stain is a hematoxylin and eosin (H&E) stain and the second stain is an immunohistochemistry (IHC) stain.

[0132] Fig.12 A method according to the second aspect is illustrated.

[0133] According to a third aspect, there is provided a method for generating annotation data for a tissue image, comprising:

[0134] a1) Collect tissue samples from 82 patients;

[0135] a2) performing 84 a first staining protocol on the tissue sample using a first stain;

[0136] a3) collecting 86 first image data of the tissue sample using a digital microscope imager;

[0137] a4) performing 88 a washing protocol on the tissue sample to substantially remove the first stain;

[0138] a5) performing 90 a second staining protocol on the tissue sample using a second stain; and a6) acquiring 92 second image data of the tissue sample using the digital microscope imager; and

[0139] A computer-implemented medical image annotation method for automatically generating annotation data for a tissue image according to the second aspect is performed.

[0140] The method according to the third aspect defines the precursor steps that laboratory professionals will undertake. For example, a tissue biopsy will be obtained from a patient and prepared for histopathological analysis. The tissue sample is stained with a first stain that can reveal tissue structure (such as H&E). The tissue slide is an image that generates the first image data. Subsequently, the tissue sample is processed by removing its cover glass, washing off the H&E stain, pre-treating the slide and using an IHC stain (such as CD marker) for the expected biomarker to immunostain it and cover glass the sample. Therefore, the second image data is obtained, including a representation of the tissue sample stained with a second stain for revealing the presence of the biomarker in the tissue sample. Therefore, the first and second image data obtained according to the method can be the input of the computer-intimate method according to the second aspect. It will be appreciated that the method according to the second aspect is not limited to performing staining, and in some cases, in addition to the tissue sample stained with the first stain that can reveal tissue structure, three, four, or more staining protocols can be obtained.

[0141] According to aspects, there is provided a method for computer correction of training a machine learning model, comprising:

[0142] f1) acquiring image annotation data generated according to the computer-implemented method according to the second aspect;

[0143] f2) training a machine learning model using the image annotation data; and

[0144] f3) Output the trained machine learning model.

[0145] therefore, Fig.13 Aspects of an apparatus for training a machine learning model are illustrated. The machine learning model collects image data 90 such as H&E images and / or IHC staining images and corresponding annotation data 92 generated according to a computer-implemented method according to the second aspect. The machine learning engine is configured to identify the structure in the relationship between the corresponding annotation data 92 and the pixels of the corresponding image data 90, and updates the configuration of its layers according to the type of machine learning algorithm deployed. For example, the machine learning model 94 may include an autoencoder algorithm, a sparse autoencoder algorithm, a variation autoencoder algorithm, a demonized autoencoder algorithm, a deep belief network, a convolutional or neural network, a deconvolutional network, or a deconvolutional or inverse graph network. The machine learning model 94 is configured to output a trained machine learning model 96 that is capable of identifying candidate objects in input image data similar to the input image data from an image for which annotation data has been provided.

[0146] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, which is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments on a suitable system.

[0147] Therefore, the computer program element can be stored on a computer unit, which can also be part of an embodiment of the present invention. The computing unit can be suitable for executing the steps of the method described above or inducing the execution of the steps of the method described above. In addition, it can be suitable for operating the components of the device described above. The computing unit can be suitable for automatically operating and / or executing the user's commands. The computer program can be loaded into the working memory of a data processor. The data processor can thus be equipped to perform the method of the present invention.

[0148] This exemplary embodiment of the invention covers both a computer program that right from the beginning uses the invention and a computer program that by means of an up-date turns an existing program into a program that uses the invention.

[0149] Furthermore, the computer program element can provide all necessary steps for implementing the procedures of an exemplary embodiment of the method as described above.

[0150] According to another exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is proposed, wherein the computer readable medium has a computer program element stored on the computer readable medium, the computer program element being described in the preceding section.

[0151] The computer program may be stored / distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0152] However, the computer program may also be present on a network like the World Wide Web and can be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, wherein the computer program element is arranged to perform a method according to one of the previously described embodiments of the invention.

[0153] It must be noted that embodiments of the present invention are described with reference to different subject matters. Specifically, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art will appreciate from the above and following descriptions that, unless otherwise indicated, in addition to any combination of features belonging to one type of subject matter, any combination between features relating to different subject matters is also considered to be disclosed by the present application. However, all features can be aligned to provide a synergistic effect that exceeds the simple sum of the features.

[0154] Although the present invention has been described and illustrated in detail in the drawings and the foregoing description, such description and illustration are to be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the description, and the appended claims.

[0155] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. Although certain features are recited in mutually different dependent claims, this does not indicate that a combination of these features cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for automatically generating annotation data of a tissue image, comprising: - input unit (12); as well as - a processor (14); wherein the input unit (12) is configured to: acquire first image data, the first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure; and acquire second image data, the second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample; and wherein the processor (14) is configured to: detect a plurality of candidate objects in the first image data, and a corresponding plurality of detection probabilities for the detected candidate objects; generate staining confidence data for the second image data, the staining confidence data representing a likelihood that a corresponding portion of the tissue sample includes the biomarker targeted by the second stain; and generate image annotation data including representations of the plurality of candidate objects, wherein individual candidate objects from the plurality of candidate objects are annotated to indicate the presence of the biomarker in the corresponding portion of the second image data if the individual candidate objects reach a combined confidence level based on the corresponding plurality of detection probabilities for the detected candidate objects, the corresponding staining confidence data, and a corresponding registration quality threshold for registration between the first image data and the second image data of the candidate objects; wherein the processor (14) is configured to: segment the plurality of candidate objects in the first image data; transfer the segmentation of the plurality of candidate objects in the first image data to the second image data based on the registration between the first image data and the second image data; and sum the corresponding staining confidence data in the locations of the plurality of candidate objects in the second image data, wherein the summed staining confidence data can be used together with the plurality of detection probabilities of the detected candidate objects to determine a likelihood score value that the candidate object is positive for the biomarker targeted by the second stain; and The processor (14) is further configured to generate an alignment quality map between the first image data and the second image data, wherein the processor is configured to annotate a candidate object among the plurality of candidate objects to indicate the presence of a biomarker only when the alignment quality map indicates that the alignment between corresponding locations of the candidate object in the first image data and the second image data exceeds an alignment quality threshold.

2. The device (10) according to claim 1, in, The processor (14) is also configured to generate the staining confidence data by: using the second image data to generate a histogram of staining confidence, and providing a plurality of staining confidence results corresponding to the positions of the plurality of candidate objects in the first image data as the corresponding staining confidence data.

3. The device (10) according to claim 1, in, The processor (14) is further configured to generate the image annotation data of the individual candidate objects among the plurality of candidate objects, the image annotation data comprising a positive annotation of a first candidate object and a negative annotation of a second candidate object among the plurality of candidate objects.

4. The device (10) according to one of claims 1 to 3, in, The processor (14) is configured to generate the staining confidence data by: generating a first channel of the second image data to be used as a positive control; generating a second channel of the second image data to be used as a negative control, and Wherein, the processor is also configured to generate a histogram of staining confidence, including using the first channel of the second image data to generate a first histogram channel, using the second channel of the second image data to generate a second histogram channel, and using the first histogram channel and the second histogram channel to generate the histogram of staining confidence.

5. The device (10) according to one of claims 1 to 3, in, The processor (14) is configured to detect a plurality of candidate objects in the first image data using a kernel detector and the corresponding plurality of detection probabilities of the detected candidate objects.

6. The device (10) according to one of claims 1 to 3, further comprising: - output unit (16); Therein, the output unit is configured to display the image annotation data on a graphical user interface, optionally in combination with the first image data and / or the second image data.

7. A computer-implemented medical image annotation method for automatically generating annotation data for a tissue image, comprising: a) acquiring (70) first image data, the first image data comprising a representation of a tissue sample stained with a first stain for indicating tissue structure; b) acquiring (72) second image data, the second image data comprising a representation of the tissue sample stained with a second stain for revealing the presence of a biomarker in the tissue sample; c) detecting (74) a plurality of candidate objects in the first image data, and a corresponding plurality of detection probabilities for the detected candidate objects; d) generating (78) staining confidence data for the second image data, the staining confidence data representing a likelihood that a corresponding portion of the tissue sample includes the biomarker targeted by the second stain, the generating (78) staining confidence data for the second image data comprising: segmenting the plurality of candidate objects in the first image data, transferring the segmentation of the plurality of candidate objects in the first image data to the second image data based on an alignment between the first image data and the second image data, and summing the corresponding staining confidence data in the locations of the plurality of candidate objects in the second image data, wherein the summed staining confidence data is used together with the plurality of detection probabilities of the detected candidate objects to determine a likelihood score value that the candidate object is positive for the biomarker targeted by the second stain; and e) generating (80) image annotation data including representations of the plurality of candidate objects, wherein individual candidate objects among the plurality of candidate objects are annotated to indicate the presence of a biomarker in a corresponding portion of the second image data if an individual candidate object reaches a combined confidence level based on the corresponding plurality of detection probabilities of the detected candidate objects, the corresponding staining confidence data, and a corresponding registration quality threshold of the registration between the first image data and the second image data of the candidate objects, and wherein generating the image annotation data further comprises generating a registration quality map between the first image data and the second image data, and annotating a candidate object among the plurality of candidate objects to indicate the presence of a biomarker only if the registration quality map indicates that the registration between the corresponding positions of the candidate objects in the first image data and the second image data exceeds the registration quality threshold.

8. The computer-implemented medical image annotation method of claim 7, in, The first stain is a hematoxylin and eosin stain, and the second stain is an immunohistochemical stain.

9. A method for generating annotation data for a tissue image, comprising: a1) Collect tissue samples from patients (82); a2) performing (84) a first staining protocol on the tissue sample using a first stain for indicating tissue structure; a3) acquiring (86) first image data of the tissue sample using a digital microscope imager; a4) performing (88) a washing protocol on the tissue sample to remove the first stain; a5) performing (90) a second staining protocol on the tissue sample using a second stain for revealing the presence of a biomarker in the tissue sample; and a6) acquiring (92) second image data of the tissue sample using the digital microscope imager; and A computer-implemented medical image annotation method for automatically generating annotation data for a tissue image according to one of claims 7 or 8 is performed.

10. A method for training a machine learning model, comprising: f1) acquiring image annotation data generated according to the computer-implemented medical image annotation method as recited in claim 7 or 8; f2) using the image annotation data to train a machine learning model; and f3) Output the trained machine learning model.

11. A computer program product, which, when executed by a processor, is adapted to perform the computer-implemented medical image annotation method according to claim 7 or 8.

12. A computer-readable medium storing the computer program product according to claim 11.

Images