Detection of the presence of different antinuclear antibody fluorescence pattern types and device for this purpose
Patent Information
- Application Number
- DE502019012332
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2019-07-19
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2039-07-19
Description
[0001] The invention relates to a method and a device for detecting the potential presence of different cellular fluorescence pattern types on a biological cell substrate comprising human epithelioma cells (HEp cells), wherein the fluorescence pattern types comprise different antinuclear antibody fluorescence pattern types. The invention further relates to a method for detecting the potential presence of different cellular fluorescence pattern types on a biological cell substrate comprising human epithelioma cells using digital image processing, as well as a computing unit, a data network device, a computer program product, and a data carrier signal therefor.
[0002] For the purposes of medical diagnostics, and in particular for the detection of autoantibodies in a liquid patient sample such as blood or blood serum, it is a known method to incubate a biological substrate containing human epithelioma cells with the liquid patient sample or with a diluted liquid patient sample. Depending on the presence of certain primary autoantibodies in the patient sample, these primary antibodies bind to different areas of the cells. The biological cell substrate is then further incubated with secondary antibodies labeled with a so-called fluorescent dye. Such secondary antibodies can, in turn, bind to the primary antibodies bound to the cell substrate.After irradiating the substrate with excitation radiation, fluorescence of the aforementioned fluorescent dye is produced, so that the primary antibodies of the patient sample, which are bound to the cells of the cell substrate and to which secondary antibodies containing the fluorescent dye are bound, can be localized in a corresponding fluorescence image as a fluorescence pattern. Depending on the patient's disease or the presence of specific primary antibodies in the patient sample, different specific fluorescence patterns result. Such detection of primary antibodies, known as autoantibodies, which are directed against the cell nuclei or the antigens of the cell nucleus, results in so-called antinuclear antibody fluorescence patterns. These are also called ANA fluorescence patterns or an ANA pattern.
[0003] The naming of such autoantigens has been based either on biochemical characteristics (DNA, histones, ribonucleoproteins: RNP) or on diseases associated with the autoantibodies.
[0004] Indirect immunofluorescence (IIFT) with human epithelial cells or human epithelioma cells, the task is to identify and classify the resulting different types of fluorescence patterns. A single pattern does not necessarily have to be present in a fluorescence image; several patterns can also be present simultaneously.
[0005] Antibodies against nuclear antigens are directed against various nuclear components (biochemical substances of the nucleus). These include nucleic acids, nuclear proteins, and ribonucleoproteins.
[0006] The different resulting antinuclear antibody fluorescence pattern types are divided into different pattern types. One approach to this can be found on the website www.anapatterns.org, which, among other things, lists and distinguishes different nuclear patterns. An example classification according to this scheme is shown in the Figure 24 illustrated.
[0007] Document DE102006027516A1 discloses a method using an indicator dye as a so-called counterstain and a fluorescent dye to visualize antibody-specific binding to a substrate. Relevant structures, such as cells in the mitotic stage, are identified by the counterstain. Individual images of such relevant structures are then extracted from the fluorescence images and fed to a classifier.
[0008] The document C.-H. LIN ET AL: "SAT0646 Development and validation of a deep learning algorithm for classifying anti-nuclear antibody patterns in indirect immunofluorescence images", SATURDAY, 16 JUNE 2018, June 1, 2018 (2018-06-01), discloses a classification of fluorescence images using a convolutional neural network.
[0009] Das Dokument Infantino M, Meacci F, Grossi V, Manfredi M, Benucci M, Merone M, Soda P. "The burden of the variability introduced by the HEp-2 assay kit and the CAD system in ANA indirect immunofluorescence test" Immunol Res.2017 Feb;65(1):345-354, describes an immunofluorescence method in which fluorescence images of a biological sample are recorded in two different color channels, DAPI and FITC, wherein the image of the DAPI color channel is analyzed to segment cell areas, wherein partial images of the FITC color channel are then selected such that a minimum number of certain cell types are present in the partial images of the FITC color channel, wherein the selected partial images of the FITC color channel are further divided into a group of positive and a group of negative partial images taking into account their signal intensity, and wherein the positive partial images from the FITC color channel are then subjected to a pattern classification in order to distinguish between different ANA fluorescence patterns.
[0010] The document "The HEp-2 cell as target for multiparametric autoantibody analyses: automation and standardization" by Hiemann, Rico, Roggenbuck, Dirk, Sack, Ulrich, Anderer, Ursula, and Conrad, Karsten, in LaboratoriumsMedizin, vol. 35, no. 6, 2011, pp. 351-361, also discloses a method in which fluorescence patterns are detected using pattern recognition in fluorescence images of a single color channel.
[0011] The document Egerer, K., Roggenbuck, D., Hiemann, R. et al. "Automated evaluation of autoantibodies on human epithelial-2 cells as an approach to standardize cell-based immunofluorescence tests", Arthritis Res Ther 12, R40 (2010), also discloses a method in which fluorescence patterns are detected by pattern recognition in fluorescence images of a single color channel, in particular using a convolutional neural network.
[0012] The object of the present invention is to provide a method by means of digital image processing which automatically detects respective potential presences of respective different antinuclear antibody fluorescence pattern types in a fluorescence image.
[0013] The object of the invention is achieved by the proposed method according to claim 1, the proposed computing unit according to claim 6, the data network device according to claim 7, the proposed computer program product according to claim 8 and the proposed data carrier signal according to claim 9.
[0014] A method is proposed for detecting the respective potential presence of respective different cellular fluorescence pattern types on a biological cell substrate comprising human epithelioma cells, wherein the cellular fluorescence pattern types comprise several different antinuclear antibody fluorescence pattern types. The method comprises various steps. The cell substrate is incubated with a liquid patient sample that potentially contains primary antibodies. The liquid patient sample is preferably diluted patient blood, particularly preferably diluted patient blood serum. The cell substrate is further incubated with a first fluorescent dye. In addition, the cell substrate is further incubated with secondary antibodies labeled with a second fluorescent dye. The cell substrate is preferably further irradiated with excitation radiation.Furthermore, a first image is captured, which represents a staining of the cell substrate by the first fluorescent dye. Furthermore, a second image is captured, which represents a staining of the cell substrate by the second fluorescent dye.
[0015] Due to the staining of the cell substrate with the first fluorescent dye, a staining of the cell substrate results in the first image, which allows the optical identification of different cellular regions based on their position in the first image. The first image is preferably spatially aligned in the same way as the second image. Such a first fluorescent dye binds nonspecifically to cellular regions due to the presence of primary antibodies in a patient sample and thus allows, in principle, the identification of different cell regions in the first image.
[0016] Because the secondary antibodies can bind to primary antibodies that originate from the liquid patient sample and are bound to specific areas of the cell substrate or the cell nucleus, depending on the presence of different types of primary antibodies, different antibody fluorescence patterns can be present and then detected in the second image.
[0017] The method according to the invention further comprises detecting respective image segments, each representing at least one mitotic cell of sufficient quality, based on the first image, wherein a mitotic cell has sufficient quality when it is in metaphase. Sub-images of the first image and corresponding sub-images of the second image are then selected based on the detected image segments, each representing at least one mitotic cell (MZX, MZY) of sufficient quality. Furthermore, the actual presence of respective cellular fluorescence pattern types is detected by means of a convolutional neural network based on the selected sub-images of the first image and the selected sub-images of the second image.Furthermore, the convolutional neural network determines respective confidence measures for the respective actual presences of the respective fluorescence pattern types based on the selected partial images of the first image and the selected partial images of the second image. For each partial image tuple comprising a partial image of the first image and a corresponding partial image of the second image, the convolutional neural network determines respective partial image confidence measures for the respective actual partial image presences of respective cellular fluorescence pattern types, such that the convolutional neural network simultaneously evaluates corresponding partial images of different color channels. Finally, the respective confidence measures for the respective actual presences of the respective fluorescence pattern types are determined based on the partial image confidence measures.
[0018] To detect different ANA patterns, it would in principle be possible to analyze an entire fluorescence image or the entire second image with a large number of human epithelioma cells (HEp cells). To identify the different ANA patterns, this entire fluorescence image could be fed to a single convolutional neural network to detect the presence of each fluorescence pattern type. The convolutional neural network (CNN) would then have to evaluate all HEp cells in the image to identify the different antinuclear antibody fluorescence pattern types. While it is advantageous for accurate detection to examine as many HEp cells as possible for statistical reasons, this approach also has several disadvantages. Firstly, the computational effort required to process an entire fluorescence image with a large number of HEp cells would be very high.Furthermore, processing such a large overall image with a large number of HEp cells would represent a large degree of freedom in terms of abstract image information. If the convolutional neural network were trained with such large, entire fluorescence images during a training phase, the amount and variance of abstract image information might be too large for the convolutional neural network to converge sufficiently to a state during the training phase to enable reliable detection of different fluorescence pattern types or classes.
[0019] Therefore, the invention proposes that, based on the first image, which is stained nonspecifically by an antibody binding with the first fluorescent dye, specific image segments are first detected, each representing at least one mitotic cell, in order to then select those partial images of the first image and those partial images of the second image that have a corresponding image segment with at least one mitotic cell. A mitotic cell is also referred to as a mitotic cell. In particular, the image segments are smaller in spatial extent than the partial images.
[0020] This results in the advantage that the convolutional neural network only needs to process a sub-image of the second image with antibody-specific fluorescent dye staining and the corresponding sub-image of the first image, rather than the entire first image and the entire second image at once. Therefore, the convolutional neural network can be trained to the size of such sub-images and only needs to evaluate sub-images with a limited number of HEp cells, and thus only a limited amount of abstract image information. Furthermore, the proposed method is particularly advantageous because only those sub-images are selected that also contain a mitotic cell.Because the partial images contain a mitotic cell, they are particularly well-suited for differentiating between different nuclear patterns. For the detection of specific nuclear patterns, not only so-called interphase cells must be examined for their staining by the fluorescent dye, but also mitotic cells. This allows for differentiation between certain fluorescence pattern types. Ensuring that a partial image contains an image segment containing a mitotic cell increases the quality of detection of the respective antinuclear antibody fluorescence pattern types.
[0021] In particular, the proposed method is particularly advantageous because the convolutional neural network, by ensuring that the partial images have corresponding segments with at least one mitotic cell, can also detect a possible staining of a metaphase plate of a mitotic cell for detecting the respective presence of the respective fluorescence pattern types in order to distinguish between different antinuclear antibody fluorescence pattern types.
[0022] Since in the method according to the invention the convolutional neural network not only processes the selected partial image of the second image, which exhibits patient-specific staining with the second fluorescent dye, but also a corresponding partial image of the first image, which exhibits patient-nonspecific staining of the cell substrate with the first fluorescent dye, the convolutional neural network can rely on localization information of cell regions or cell nucleus regions present in the first partial image and simultaneously detect the actual patient-specific fluorescence patterns in the partial images of the second image. The convolutional neural network thus simultaneously evaluates corresponding partial fluorescence images of different color channels.
[0023] Advantageous embodiments of the invention are the subject of the dependent claims and are explained in more detail in the following description with partial reference to the figure.
[0024] For the purposes of this application, a mitotic cell is of sufficient quality or is valid if it is in a correct or valid mitotic stage. A correct or valid mitotic stage is called metaphase. A mitotic cell that is not in metaphase is therefore an incorrect or invalid mitotic cell.
[0025] Preferably, the respective confidence measures are output.
[0026] Preferably, the convolutional neural network has an output layer that generates a respective feature map for a respective cellular fluorescence pattern type, in particular for a respective partial image tuple. Furthermore, the convolutional neural network determines a respective confidence measure, in particular for a respective partial image tuple, based on a respective feature map. A partial image tuple comprises a partial image of the first image and a corresponding partial image of the second image.
[0027] Preferably, the method further comprises: segmenting the first image into image segments of different segment classes, wherein preferably at least a first segment class represents a cell stage of a cell and wherein preferably at least a second segment class represents a cell region within a cell, further determining a brightness value for at least one fluorescence pattern type based on one or more image segments of at least one specific segment class, and further verifying the confidence measure of the fluorescence pattern type based on the brightness value of the fluorescence pattern type.
[0028] Preferably, the confidence measure is verified on the basis of the brightness value and depending on a threshold value specified by a user.
[0029] Preferably, the method further comprises: dividing the second image into a set of partial images according to a predetermined division scheme, selecting partial images of the second image on the basis of the detected image segments and further selecting corresponding partial images of the first image and detecting respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network on the basis of the selected partial images of the second image and on the basis of the selected partial images of the first image.
[0030] Furthermore, a device according to the invention is proposed for detecting the respective potential presence of respective different cellular fluorescence pattern types on a biological cell substrate comprising human epithelioma cells by means of digital image processing. The device comprises: a holding device for the biological substrate, which has been incubated with a liquid patient sample potentially comprising primary antibodies, further with a first fluorescent dye, and further with secondary antibodies labeled with a second fluorescent dye. The device further comprises: at least one image acquisition unit for acquiring a first image representing a staining of the cell substrate by the first fluorescent dye, and for acquiring a second image representing a staining of the cell substrate by the second dye.The device is characterized by at least one computing unit which is designed to detect respective image segments on the basis of the first image, each representing at least one mitotic cell of sufficient quality, wherein a mitotic cell has sufficient quality when it is in metaphase, further to select partial images of the first image and corresponding partial images of the second image on the basis of the detected image segments, each representing at least one mitotic cell (MZX, MZY) of sufficient quality, and further to detect respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network on the basis of the selected partial images of the first image and the selected partial images of the second image.The computing unit is further configured to determine respective confidence measures for the respective actual presences of the respective fluorescence pattern types by means of the convolutional neural network based on the selected partial images of the first image and the selected partial images of the second image. For a respective partial image tuple comprising a partial image of the first image and a corresponding partial image of the second image, respective partial image confidence measures for respective actual partial image presences of respective cellular fluorescence pattern types are determined by means of the convolutional neural network, such that the convolutional neural network simultaneously evaluates mutually corresponding partial images of different color channels. Furthermore, the computing unit is configured to determine the respective confidence measures for the respective actual presences of the respective fluorescence pattern types based on the partial image confidence measures.
[0031] Furthermore, a computing unit is proposed which is designed to receive a first image during digital image processing, which represents a staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, and to receive a second image, which represents a staining of the biological cell substrate by a second fluorescent dye.The computing unit is characterized in that it is further designed to detect, on the basis of the first image, respective image segments which each represent at least one mitotic cell of sufficient quality, wherein a mitotic cell has sufficient quality when it is in metaphase, to select, on the basis of the detected image segments, partial images of the first image and corresponding partial images of the second image which each represent at least one mitotic cell (MZX, MZY) of sufficient quality, and to detect, on the basis of the selected partial images of the first image and the selected partial images of the second image, respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network.The computing unit is further configured to determine respective confidence measures for the respective actual presences of the respective fluorescence pattern types by means of the convolutional neural network based on the selected partial images of the first image and the selected partial images of the second image. For a respective partial image tuple comprising a partial image of the first image and a corresponding partial image of the second image, respective partial image confidence measures for respective actual partial image presences of respective cellular fluorescence pattern types are determined by means of the convolutional neural network, such that the convolutional neural network simultaneously evaluates mutually corresponding partial images of different color channels. Furthermore, the computing unit is configured to determine the respective confidence measures for the respective actual presences of the respective fluorescence pattern types based on the partial image confidence measures.
[0032] Furthermore, a data network device is proposed, which has at least one data interface for receiving a first image representing a staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, and for receiving a second image representing a staining of the biological cell substrate by a second fluorescent dye. The data network device further comprises a computing unit according to the invention.
[0033] Furthermore, a computer program product is proposed which comprises instructions which, when executed by a computer, cause the computer to carry out a method for digital image processing, comprising: receiving a first image which represents a staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, receiving a second image which represents a staining of the biological cell substrate by a second fluorescent dye, detecting respective image segments which each represent at least one mitotic cell of sufficient quality, on the basis of the first image, wherein a mitotic cell has sufficient quality if it is in metaphase,Selecting partial images of the first image and corresponding partial images of the second image based on the detected image segments, each of which represents at least one mitotic cell of sufficient quality, and detecting respective actual presences of respective cellular fluorescence pattern types using a convolutional neural network based on the selected partial images of the first image and the selected partial images of the second image. The method is characterized by the steps: determining respective confidence measures for the respective actual presences of the respective fluorescence pattern types using the convolutional neural network based on the selected partial images of the first image and the selected partial images of the second image, wherein for a respective partial image tuple comprising a partial image of the first image and a corresponding partial image of the second image,respective partial image confidence measures for respective actual partial image presences of respective cellular fluorescence pattern types are determined by means of the convolutional neural network, so that the convolutional neural network evaluates corresponding partial images of different color channels simultaneously, and finally determining the respective confidence measures for the respective actual presences of the respective fluorescence pattern types on the basis of the partial image confidence measures.
[0034] A data carrier signal which transmits the computer program product is also proposed.
[0035] In the following, the invention is explained in more detail with reference to specific embodiments without limiting the general inventive concept with reference to the figures. Figure 1 Steps of the method according to the invention according to a preferred embodiment, Figure 2processing of partial images by a convolutional neural network according to a preferred embodiment, Figure 3 processing of several partial images by a convolutional neural network according to a preferred embodiment, Figure 4 Steps for segmenting the first image and verifying confidence measures based on determined brightness values according to a preferred embodiment, Figure 5 detailed steps of a method for segmenting the first image according to a preferred embodiment, Figure 6 a proposed device according to a preferred embodiment, Figure 7A a proposed computing unit according to a preferred embodiment, Figure 7B a proposed data network device according to a preferred embodiment, Figure 7C a proposed computer program product and a data carrier signal according to a preferred embodiment, Figure 8a fluorescence image superimposed from two color channels, Figure 9 a first fluorescence image of a first color channel, Figure 10 a second fluorescence image of a second color channel, Figure 11 different exemplary nuclear antibody fluorescence pattern types, Figure 12a a segmented fluorescence image, Figure 12b a legend to Figure 12a , Figure 13 a division of the fluorescence image into partial images and a highlighting of different mitotic cells, Figure 14 a first partial image, a second partial image and a segmented partial image, Figure 15 different feature maps for different fluorescence pattern types or fluorescence pattern classes, Figure 16a the second part of the image Figure 14 , Figure 16b a feature map of a specific fluorescence pattern type as well as an associated overlay image, Figure 16c a feature map for another specific fluorescence pattern type as well as an associated overlay image, Figure 17a a first further fluorescence image, Figure 17b a highlighted interphase cell as well as a highlighted mitotic cell to the fluorescence image from Figure 17a , Figure 18 a second additional fluorescence pattern image, Figure 19a a highlighted interphase cell to the fluorescence pattern image from Figure 18 , Figure 19b a highlighted mitotic cell to the fluorescence pattern image from Figure 18 , Figure 20 a third additional fluorescence pattern image with highlighted partial images and highlighted mitotic cells, Figure 21 a first and a second partial image as well as a segmented partial image to the fluorescence pattern image from Figure 20 , Figure 22 different feature maps for the partial images from the Figure 21 , Figure 23a the second part of the Figure 21 , Figure 23b a feature map for a specific fluorescence pattern type and a resulting overlay image to the partial image from Figure 23a , Figure 23can enlarged view of a mitotic cell from the second part of Figure 23a or from Figure 20 , Figure 24 a classification scheme for different fluorescence pattern types, Figure 25 a layer of a convolutional neural network for detecting fluorescence pattern types according to a preferred embodiment, Figure 26 a convolutional neural network for image segmentation according to a preferred embodiment, Figure 27 a table with experimental results.
[0036] The Figure 8 shows a fluorescence pattern image B superimposed from two color channels, which has a gray coloration representing a coloration of a cell substrate in a so-called red channel as a first color channel by a first fluorescent dye and also a superimposed coloration of the cell substrate in a so-called green channel as a second color channel by a second fluorescent dye.
[0037] Due to an incubation of the cell substrate shown here, comprising HEp cells, with a first fluorescent dye, preferably propidium iodide, certain cell areas are colored, as shown in image B1 in Figure 9 The HEp cells are preferably so-called HEp2 cells.
[0038] Due to an incubation of the cell substrate shown here with a diluted patient-specific blood serum and the resulting binding of primary autoantibodies, a further incubation of the cell substrate with secondary antibodies, which are labelled with a second fluorescent dye, results in a staining in a green channel, which in the Figure 10 shown as a second image B2. The second fluorescent dye is preferably fluorescein isothiocyanate (FITC).
[0039] It is evident that the staining by the first fluorescent dye in the first image B1 allows for the basic detection of cell structures or cell regions. Furthermore, a differentiated pattern-like staining of cell regions in the second image B2 of the green channel allows for a differentiated detection of the presence of different types of fluorescent patterns.
[0040] In the second picture B2 from the Figure 10 It is not necessary that only a single stained pattern or a single antinuclear antibody fluorescence pattern of a certain pattern type is present; several such antinuclear antibody fluorescence pattern types can also occur.
[0041] The first image B1 and the second image B2 can be captured by means of at least one image capture unit in the form of one or more cameras K1 and K2 of a device V1 from Figure 6captured. In the event that the first image B1 in the first color channel is captured as image information BI1 by a first image capture unit K1 and that the second image B2 in the second color channel is captured as image information B2 by a separate second image capture unit K2, the images B1 and B2 can be spatially aligned with one another before further processing so that certain image regions of the first image B1 with their corresponding pixel indices correspond to certain image regions of the second image with the same pixel indices. However, it is also possible, by appropriately configuring the device V1 with regard to optical components and optical filters, to capture the two images B1 and B2 by means of a single image capture unit or a single camera which has a sensor which can capture images in the two color channels separately from one another, for example a color sensor.
[0042] The Figure 11 shows different antinuclear antibody fluorescence pattern types BM1,..., BM8. Pattern type BM1 represents a so-called negative staining, in the case that none of the other antinuclear antibody fluorescence pattern types to be detected is present. This corresponds to pattern AC-0 from the Figure 24 with the pattern index n=1 used here as an example. The exemplary pattern classification from the Figure 24 comes from the source www.anapatterns.org / trees-full.php.
[0043] The pattern BM2 is a so-called homogeneous coloring or the pattern type homogeneous, which in the Figure 24 referred to as AC-1 with the pattern index n=2 used here as an example.
[0044] The pattern type of pattern BM3 is a so-called centromere staining, which is found in the Figure 24 referred to as AC-3 with the pattern index n=3 used here as an example.
[0045] The pattern type BM4 is a so-called dense fine speckle pattern, which is known as pattern type AC-2 in the Figure 24 is denoted by the pattern index n=4 used here as an example.
[0046] The pattern type BM5 is a so-called fine or coarse speckled pattern, which is used in the Figure 24 referred to as AC4 or AC5 with the pattern index n=5 used here as an example.
[0047] The pattern type BM6 represents so-called nuclear dots, which are found in the Figure 24 is designated as pattern type AC-6,7 with the pattern index n=6 used here as an example.
[0048] The pattern type BM7 is a so-called nucleolar pattern type, which is found in the Figure 24 designated AC-8, 9,10 with the sample index n=7 used here as an example.
[0049] The pattern type BM8 is a so-called nuclear marginal or nuclear membrane pattern type, which is found in the Figure 8is designated as pattern type AC-11,12 with the pattern index n=8 used here as an example.
[0050] The different antinuclear antibody fluorescence pattern types thus include homogeneous patterns (AC-1), speckled patterns (AC-2, 4, 5), a centromere pattern (AC-3), nuclear dot patterns (AC-6, 7), nucleolar patterns (AC-8, 9, 10), and the nuclear rim fluorescence pattern type (AC-11, 12). Another fluorescence pattern type class is the so-called negative class (AC-0), in the case of a patient sample without specific primary antinuclear antibodies.
[0051] Image B from the Figure 8 is an exemplary fluorescence image, for which the second image B2 from the green channel in the Figure 10exhibits both a homogeneous pattern (AC-1) and finely or coarsely speckled patterns (AC-4, AC-5). The following section will discuss how the convolutional neural network proposed here can detect these two pattern types.
[0052] Figure 1 shows the essential steps of the method V according to the invention. In a first step SC1, the cell substrate is incubated as previously explained.
[0053] In a step SC2, the first image and the second image are captured so that a first image information BI1 and a second image information BI2 are provided.
[0054] In the next step SC3, based on the first image or the first image information BI1, respective image segments are detected, each representing at least one mitotic cell. This results in so-called segmentation information SI.
[0055] Such segmentation information SI is stored as an image SB in the Figure 12a shown. The Figure 12b shows different gray value levels for the different segment classes s=1 ...S for S=6, which represent the different segment classes in the image SB from the Figure 12aThe first segment class s=1 is a so-called background class. A second class s=2 indicates a cell edge. A third class s=3 indicates a poor mitotic cell or a mitotic cell of insufficient quality, alternatively also called an invalid mitotic cell. For the purposes of this application, a mitotic cell has insufficient quality or is invalid if it is in an incorrect or invalid mitotic stage. Another class s=4 indicates an interphase cell. Another class s=5 indicates an area with nucleoli. Another class s=6 indicates a so-called good mitotic cell or a mitotic cell of sufficient quality, also called valid mitotic cells. For the purposes of this application, a mitotic cell has sufficient quality or is valid if it is in a correct or valid mitotic stage. A correct or valid mitotic stage is the so-called metaphase.A mitotic cell that is not in a metaphase is therefore an incorrect or invalid mitotic cell.
[0056] According to the Figure 1 In a step SC4, partial images of the first image and corresponding partial images of the second image are then selected on the basis of the detected image segments, each of which represents at least one mitotic cell.
[0057] The Figure 13 The second image B2 is shown again, which is divided into subimages TBA, TBB, TBC, TBD, and further subimages. White squares indicate mitotic cells in the VMZ that are present and of sufficient quality. The subimage TBD is not selected here because it does not contain any mitotic cells in the MZ. White circles indicate mitotic cells in the IMZ of insufficient quality.
[0058] The division of the second image B2 into corresponding partial images can preferably be carried out in such a way that from a position information of mitotic cells from the image SB the Figure 12a It is concluded whether such a mitotic cell of sufficient quality is present within a partial image of the first image and the corresponding part of the second image. The detection of image segments, each representing at least one mitotic cell of sufficient quality, is carried out on the basis of the first image B1 from the Figure 9 .
[0059] The Figure 13 shows a corresponding division into partial images and detected image segments, adopted for the second image B2, which show the mitotic cells VMZ of sufficient quality.
[0060] For this purpose, the first image is preferably divided into a set of partial images according to a predetermined division scheme and then such a partial image of the first partial image is selected which has such image segments which represent mitotic cells of sufficient quality, so that then corresponding partial images of the second image can also be selected for the selected partial images of the first image.
[0061] According to the Figure 1 the selection of the partial images takes place in step SC4, so that a first partial image information TBI1 of selected partial images of the first image and a second partial image information TBI2 of selected partial images of the second image, which correspond to the partial images of the first image, are obtained.
[0062] In step SC5, the actual presence of the respective antinuclear antibody fluorescence pattern types is then detected using a convolutional neural network (CNN) based on the selected partial images of the first image and the selected partial images of the second image. The detection result is then output as detection information DI.
[0063] By focusing on image segments with a mitotic cell of sufficient quality and by further selecting the partial images depending on those detected image segments which each represent at least one mitotic cell of sufficient quality, it is ensured that the mitotic cell to be observed or the mitotic cell present in the partial image is at a correct stage in order to represent a valid mitotic cell with regard to its coloration of its metaphase plate, which provides valid and reliable information for the detection of the different antinuclear antibody fluorescence pattern types.
[0064] Figure 2shows essential steps of the convolutional neural network in one embodiment CNN1 for determining detection information DI1 regarding the presence of antinuclear antibody fluorescence pattern types for a first partial image TB11 and a second partial image TB12. The partial images TB11 and TB12 thus represent a partial image tuple. The partial images TB11, TB12 are preferably normalized to a value range or gray value range from 0 to 1.
[0065] The Figure 14 shows an exemplary first partial image TB11 from the first image B1 and a corresponding second partial image TB12 from the second image B2. Figure 13 and the corresponding representation of the second image B2, it becomes clear that the second partial image TB12 is the partial image TBA.
[0066] In the Figure 14Furthermore, a segmented partial image TBS is shown for the partial images TB11 and TB12 shown here. It is clearly visible that the VMZ regions represent valid mitotic cells or mitotic cells of sufficient quality.
[0067] Such partial images TB11, TB12 are then created according to the Figure 2 processed by the Convolutional Neural Network CNN1 to determine a detection information DI1, which for N different classes, here with index n=1... N and N=8, each outputs a value which lies in the value range Dn ∈ {0,1} as a Boolean value detects or indicates the presence of the pattern. The detection information DI1 for all N pattern types or pattern classes with the values D11, ..., D1N then corresponds to a vector D 1 → = D 11 ⋮ D 1 N .
[0068] According to the Figure 2Processing then takes place over several layers L1,..., LP of a convolutional neural network CNN1, where each layer L1,..., LP has at least one convolutional layer.
[0069] The convolutional neural network CNN1 then determines a respective feature map FM1,..., FMN for each of the N=8 classes or N=8 fluorescence pattern types. Based on each feature map FM1,..., FMN, a respective confidence measure P1,..., PN or a respective prediction value P1,..., PN is then determined based on the respective actual presence of the respective fluorescence pattern type with index n=1...N.
[0070] This is preferably done by using a pooling layer PL or a pooling layer function PL, which preferably performs a so-called global average pooling, to reduce a feature map FM1 to a single scalar value or an averaged value, which is preferably also called a logit, as a value LO1.
[0071] The logit values LO1,.., LON of the N classes or fluorescence pattern types are then each individually subjected to a so-called sigmoid function SF in order to determine a respective prediction value or a respective confidence measure Pn with index n for a respective class n based on the respective feature map FMn with index n. This structure of the convolutional neural network therefore results in the advantage that a separate detection channel is available for each individual fluorescence pattern type and that not only a single pattern or a single pattern type can be detected as present, but also multiple pattern types simultaneously.
[0072] Determining respective confidence measures Pn as prediction information or presence information PI1 then enables an output of prediction values or confidence measures in the form of a vector P 1 → = P 1 ⋮ P N with a range of values 0 ≤ Pn ≤ 1.
[0073] If such confidence measures Pn are determined and preferably also output, this is advantageous because a user can use a respective confidence measure as an indicator for the presence of a respective pattern in a later final diagnosis.
[0074] The part TCNN1 of the Convolutional Neural Network CNN1, which is used to determine the confidence measures Pn based on the partial images TB11, TB12, can be applied separately to respective partial image tuples TB11, TB12, consisting of a first partial image of the first color channel TB11 and a corresponding second partial image of the second color channel TB12, as will be explained in detail later.
[0075] The Figure 2further shows further steps by means of which, on the basis of the confidence measures or prediction values, which can be summarized as prediction information PI1, and by means of which detection results or detection values D11,..., D1N can be determined, which in turn can be summarized as detection information DI1.
[0076] A confidence measure Pn can then be evaluated using a threshold function TS1, preferably applying a user-specified threshold value or a threshold value T1 specified in some other way, to determine the corresponding detection information DI1. Preferably, the detection value D11 can assume the value 1, thus indicating the presence of the pattern with the index n=1, if the confidence measure Pn exceeds a threshold value T1 of 0.5.
[0077] An application of further threshold operations TS1,..., TSN using respective individual threshold values T1,..., TN, preferably for the respective patterns, then enables the determination of the detection values D11,..., D1N.
[0078] The Convolutional Neural Network CNN1 from Figure 2 has a partial convolutional neural network TCNN1, which will be used later in the Figure 3 applies.
[0079] The Figure 15 shows exemplary feature maps FM1,..., FM8 for the different fluorescence pattern types or example patterns, which are shown in Figure 11 and examples of the corresponding fluorescence pattern types from the Figure 24 with the corresponding labels. The feature map FM2 represents an activation with respect to a homogeneous pattern for the partial images TB11 and TB12 from the Figure 14The degree of brightness in feature map FM2 indicates a probable presence of a homogeneous fluorescence pattern type in the partial image TB12. Furthermore, feature map FM5 represents a probable presence of finely speckled or coarsely speckled patterns in the corresponding partial image TB12. For the other feature maps FM1, FM3, FM4, FM6, FM7, FM8, no or a relatively low activation results. The feature maps FM1,..., FM8 from the Figure 15 were originally determined with a resolution of 8x8 pixels and then interpolated to the resolution of 512x512 pixels of the second sub-image TB12.
[0080] The Figure 16a shows again the second partial image TB12 from the Figure 14 together with the feature map FM5 in the Figure 16band an overlay of this feature map FM5 over the sub-image TB12 as an overlaid feature map OLM5. It is clearly evident here that the proposed convolutional neural network succeeded in generating a feature map FM5 that highlights or detects cell nuclei with fine or coarse speckles.
[0081] The Figure 16c shows the feature map FM2 from the Figure 15 for homogeneous patterns and one via the drawing TB12 of the Figure 16a superimposed feature map OLM2. Here, too, it is evident that the proposed convolutional neural network succeeded in highlighting or detecting from the TB12 subimage those regions that represent cell nuclei with a homogeneous coloration or a homogeneous pattern type.
[0082] Looking at the Figure 14and the segmented partial image TBS, it becomes clear that the valid mitotic cells VMZ are the cells which are responsible for the detection of the homogeneous pattern according to the Figure 16c in the overlaid feature map OLM2. Even if in principle the interphase cells, which are shown in the overlaid feature map OLM5 from the Figure 16b were selected, a homogeneous staining may be present, but this is masked by the speckled staining of these interphase cells, so that a detection of a homogeneous pattern on these interphase cells alone from the Figure 16b in the overlaid feature map OLM5 cannot be done reliably. However, by ensuring that the selected drawing files TB11 and TB12 from the Figure 14in any case have valid mitotic cells VMZ or the segmented image TBS has such valid mitotic cells VMZ, the homogeneous pattern staining or the homogeneous fluorescence pattern type can advantageously be recognized on these mitotic cells, as can be seen by a joint examination of the overlaid feature map OLM2 from the Figure 16c and the segmented partial image TBS from the Figure 14 becomes apparent.
[0083] The Figure 3 illustrates further advantageous embodiments of the method according to the invention. The Convolutional Neural Network CNN2 shown here uses the partial Convolution Neural Network TCNN1 from the Figure 2 to determine for respective partial image tuples TB11, TB12,..., TBJ1, TBJ2 with tuple index j=1...J respective partial image confidence measures PI1, ..., PIJ, as previously in the Figure 2presented and explained in detail, in order to then determine the respective actual presences of the respective fluorescence pattern types and the corresponding confidence measures based on the partial image confidence measures as a presence information or prediction information PI. Here, a partial image confidence measure PI corresponds to a vector Pj with index j. The information PI corresponds to a vector P ‾ → = P ‾ 1 ⋮ P ‾ N with a respective confidence measure Pn for a respective fluorescence pattern type with index n.
[0084] The resulting partial image confidence measures Pj with index j=1... J are then averaged in an averaging step MS over all selected j=1...J partial images or partial image tuples according to P ‾ → = 1 J ∑ i = 1 J P ι → .
[0085] These confidence measures P ‾ → = P ‾ 1 ⋮ P ‾ N can then be output as the prediction information or confidence information PI.
[0086] This embodiment of the method according to the invention is advantageous because a Convolutional Neural Network CNN2 does not simultaneously extract all the information of an entire first and an entire second image B1, B2 from the Figure 9 and 10 all at once, but the partial image tuples can be evaluated individually in separate, separate processing paths.
[0087] The determined confidence measures PI or P ‾ → = P ‾ 1 ⋮ P ‾ N in respective threshold value assessment steps TS1,..., TSN using preferably predeterminable or otherwise predefined threshold values T1,..., TN in order to generate a corresponding detection information DI with corresponding detection values D1,..., DN with index n=1... N as D 1 → = D 1 ⋮ D N to spend.
[0088] The Figure 5 shows details of step SC3 from the Figure 1for detecting image segments, each representing at least one mitotic cell of sufficient quality, based on the first image BI1. This is done using a segmentation convolutional neural network (SEG-CNN). The segmentation convolutional neural network (SEG-CNN) has steps or layers LA1, ..., LAQ, which in turn can be understood as a convolutional neural network. The segmentation is thus performed using another convolutional neural network (SEG-CNN), which differs from the convolutional neural network (CNN2) for detecting the respective actual presence of each of the cellular fluorescence pattern types using a convolutional neural network based on the selected partial images of the first image and the selected partial images of the second image.The division of the segmentation task and the detection task into two different convolutional neural networks is advantageous because the convolutional neural network for detection does not have to be trained for the purpose of segmentation and thus the convolutional neural network can be trained particularly specifically for the purpose of segmentation.
[0089] According to the Figure 5 the first image, e.g. image B1 from the Figure 9 , or the corresponding image information BI1 is fed to a Convolutional Neural Network CNNS for the purpose of segmentation.
[0090] In the Figure 5The segmentation convolutional neural network SEG-CNN is shown, which segments an input image using multiple layers LA1, LAQ. Preferably, the image data BI1 is normalized to a value range from 0 to 1. Preferably, a so-called downscaling also occurs in an optional step DWS. Preferably, a first image of the image data BI1 has a dimensionality of 2400×1700 pixels. This first image can then be reduced to a size of 800×800 pixels by so-called downsampling. The further processing steps are explained in detail here for the exemplary case in which a DWS downscaling of the first image to a size of 800×800 pixels occurred.However, this is not absolutely necessary; the segmentation convolutional neural network SEG-CNN can also be dimensioned so that it can process the first image with a size of 2400x1700 pixels without downscaling.
[0091] The Convolutional Neural Network CNNS has several layers LA1,..., LAQ, each of these layers has at least one convolutional layer.
[0092] The Convolutional Neural Network CNNS then generates for each of the segment classes s=1... S with here for example S=6 classes, as in Figure 12b indexed by the legend, a respective activation map M1,..., MS. The activation maps M1, ..., MS each preferably have the same size or resolution, preferably 800x800 pixels, as the first image or the first image data BI1 after downscaling DWS.
[0093] The activation map M1 for the first segment class s=1 indicates, in its first pixel PI1, the value present there, to what degree the corresponding pixel of the first image B1 or the image information BI1 belongs to this first segment class. Corresponding individual pixels PI1,...PIS of the activation maps M1,...MS each indicate the degree to which the pixel located at the same location of the first image B1 or the first image information BI1 belongs to the corresponding class. By linking the values of the activation maps M1, M2,..., MS at the same pixel location or the corresponding values of this corresponding pixel location PI1, PI2,..., PIS using a so-called softmax function SMX, a probability map PM1 can then be created in order to generate a value for a corresponding pixel position PIX1, which indicates the probability with which, in a value range from 0 to 1, the corresponding pixel of the first image B1 belongs to the first class with s=1. This can be determined accordingly for the further S classes with s=1...S using a separately applied softmax function SMX for corresponding pixels, and corresponding probability maps PMs can be entered for the s=1...S segment classes. The axes x and y shown here indicate respective indices for the corresponding pixel positions within the probability maps PM1, ..., PMS. The probability maps PM1, ..., PMS preferably have the same size or resolution, preferably 800x800 pixels, as the first image orthe first image data BI1 after downscaling DWS and how the activation maps M1, ... , MS.
[0094] A segmentation map SM is then determined, which has a value from the value range s ∈ {1, ..., S} at a corresponding pixel position PIX1 and indicates to which segment class s=1...S the corresponding pixel of this position PIX1 belongs. For this purpose, the corresponding pixel values of the corresponding pixel position PIX1 are linked across all probability maps PM1,..., PMS using an Argmax function. A pixel of the segmentation map SM with the index x, y then contains the index value of that probability map PM1,..., PMS for which the corresponding probability value was highest at the pixel position. This can be determined according to SM x y = argmax s ∈ 1 … S PM s x y .
[0095] The segmentation map SM then contains for each individual pixel a value from the value range s=1... S, which assigns the corresponding pixel to a respective segment class.
[0096] The segmentation map SM preferably has the same size or resolution of preferably 800×800 pixels as the first image or the first image data BI1 after the downscaling DWS, as do the activation maps M1, ..., MS and the probability maps PM1, ..., PMS.
[0097] In the event that a so-called downscaling of the first image B1 or the first image information BI was carried out in an optional step DWS before the segmentation, a so-called upscaling of the segmentation map SM from preferably 800x800 pixels back to the original image resolution of the image B1, for example 2400 x 1700 pixels, can then preferably be carried out in a preferably carried out step UP.
[0098] The segmentation map SM can then be further processed to merge partial areas or segment areas using digital image processing in an image processing step BV using conventional image processing measures such as eroding, delation and / or contour matching, in order to then arrive at a modified segmentation map SM', which can be output as segmentation information SI. Such information with a segmentation SI is in the Figure 12a shown.
[0099] The Figure 5 illustrates how the segmentation of the first image into image segments of different segment classes can be carried out, whereby the segment classes are as in the Figure 12b illustrated can be selected. Preferably, at least one first segment class represents a cell stage of a cell, and at least one second segment class further represents a cell region within a cell.
[0100] The segmentation information SI, as shown by way of example as the segmentation image SB in Figure 12a, can then be used to determine at least one brightness value for at least one fluorescence pattern type based on one or more image segments of at least one specific segment class and to then verify at least one confidence measure of a fluorescence pattern type based on at least one such brightness value.
[0101] The Figure 5 the possibility of extracting at least one brightness value for at least one fluorescence pattern type from the second image B2 based on the second image B2 and the segment information SI. This is done in a brightness determination step SH.
[0102] It is therefore carried out according to the Figure 5determining a brightness value for at least one fluorescence pattern type based on one or more image segments of at least one specific segment class, and verifying the confidence measure of the fluorescence pattern type based on the brightness value of the fluorescence pattern type.
[0103] For the different n=1... N classes, corresponding brightness values h with index n=1..N as vector h → = h 1 ⋮ h N determined and output as brightness information HI.
[0104] For example, if a brightness value is used for a pattern "densely finely speckled" (AC-2) of the pattern type n=4, a brightness value determination can be carried out in such a way that the interphase cells of the segment class s=4, which are also in the Figure 12a are shown, with their brightness values at the corresponding points of image B2 from the Figure 10For example, for each interphase cell or each segment area with index s=4, the 0.65 quantile of the brightness values can be determined separately and then the resulting multiple quantile brightness values of the multiple interphase cells can be averaged again by averaging to obtain a brightness value h 4 for the fluorescence pattern with index n=4.
[0105] For so-called finely or coarsely speckled patterns of type AC-4, AC-5 with index n=5, for example, a 0.65 quantile of the brightness values of an interphase cell or a corresponding segment area of class s=4 can be used and then these brightness values can be used to determine a brightness value h 5 for the fluorescence pattern with index n=5.
[0106] For so-called homogeneous patterns (AC-1) of the type with index n=2, as in Figure 11Illustrated using the example pattern BM2, a 0.5 quantile can then be determined for each valid mitotic cell or each corresponding segment area of the class s=6 and then averaged over the quantile values of all valid mitotic cells to obtain a brightness value h 2 for the fluorescence pattern with index n=2.
[0107] Brightness values determined in this way h → = h 1 ⋮ h N Brightness information HI can then be provided.
[0108] The Figure 4 illustrates an exemplary procedure in which, based on the first image B1 and the second image B2, a partial convolutional neural network TCNN2, which is used in the Figure 3 is indexed, a vector P ‾ → = P ‾ 1 ⋮ P ‾ N with corresponding confidence measures for the corresponding n=1... n classes. This is then provided as confidence measure information or prediction information PI.
[0109] For this purpose, the partial convolutional neural network TCNN2 uses those partial images or partial image information TBI1, TBI2 that were determined based on the first image B1 using the segmentation step SC3 and the resulting segmentation information SI, as well as the second image B2. The partial convolutional neural network TCNN2 therefore incorporates those partial images that were selected from the first image B1 and the second image B2 by the segmentation step SC3 and the selection step SC4.
[0110] The segmentation information SI is then used in the step SH to determine the brightness values, which is already in the Figure 5 was explained, is used to determine and provide the brightness information HI based on the brightness values of the second image B2, which is also available as a vector h → = h 1 ⋮ h N can be described.
[0111] In a test step PS, the confidence measure information PI is then verified on the basis of the brightness information Hl.
[0112] The values of the confidence measure are then P ‾ → = P ‾ 1 ⋮ P ‾ N verified using one or more threshold values related to the confidence measure values in the form of threshold values T1,..., TN and preferably using brightness threshold values in H1,..., HN, which are based on the brightness values h → = h 1 ⋮ h N be applied.
[0113] This then enables the output of verified confidence measures PI*. The threshold values T1,..., TN are predefined. The threshold values H1,..., HN are predefined. Preferably, the threshold values T1,..., TN can be specified by a user to influence the evaluation. Preferably, the threshold values H1,..., HN can be specified by a user to influence the evaluation. The threshold values H1,..., HN are individual thresholds for each fluorescence pattern type. The determination of the verified confidence measures PI* as a vector P * ‾ → = P * ‾ 1 ⋮ P * ‾ N is then preferably carried out in accordance with P * ‾ n = { 1 fuer P ‾ n > T n ∩ h n > HS n 0 sonst .
[0114] The approach proposed here is advantageous because, although the confidence measures are initially determined based on the first image and the second image or the corresponding partial images, in which a convolutional neural network determines the confidence measures PI, brightness values HI are now also explicitly determined from the second image or the entire second image, in particular without reduction to partial images, in order to verify the confidence measure or the confidence measures PI. This enables even more precise detection of fluorescence pattern types. One advantage here is in particular that the entire second image B2 is examined with regard to brightness values in corresponding segment classes, so that not only the brightness of cells or segment areas of a single partial image is observed, but also segment areas outside of selected partial images are taken into account.This means that a possible brightness artifact on an individual cell or an individual segment area of a partial image has less of an impact on the determination of the brightness values and thus the determination of the verified confidence measures, since brightness values from segment areas outside the partial image are also applied to the confidence measures of a partial image and a brightness artifact of an individual segment area is weighted less heavily to determine the verified confidence measures.
[0115] The Figure 2shows preprocessing steps VS, VS1, VS2, which can be used to preprocess the first partial image TB11 and the second partial image TB12. In the first preprocessing step VS1, the first partial image TB11 and the second partial image TB12 are convolved as input feature maps, preferably with eight different convolution kernels. No striding is performed, and the dimensionality of the partial images TB11, TB12 is maintained at an exemplary dimensionality of 512x512 image pixels. The choice of the number of eight convolution kernels does not necessarily have to correspond to the number of N=8 fluorescence pattern types or fluorescence pattern classes.
[0116] In the further preprocessing step VS2, a so-called batch normalization is then carried out, whereby the number of feature maps, here for example eight feature maps, is retained.
[0117] The eight feature maps resulting from step VS2 are then processed in a sequence of P different layers L1,..., LP with index p=1...P, and here, for example, P=6. At the end of layer L1, a set of, for example, R feature maps FMA1,..., FMAR with index r=1...R results. In this example, the parameter R is preferably 11. The number R of feature maps at the end of the first layer L1 can differ from the number of fluorescence pattern types N.
[0118] In detail, the P layers L1,..., LP follow one another, with the Figure 25 an example structure of the first layer L1.
[0119] After the last layer (LP), for example, 56 feature maps with a dimension of 8x8 pixels are obtained. In a final processing step (AVS), convolution is performed with N convolution kernels corresponding to the number of N classes or N fluorescent pattern types to generate the N different feature maps (FM1,..., FMN).
[0120] The Figure 25 shows an example of the first layer L1 from the Figure 2 The preprocessing step VS2 is shown in dashed lines, since it is not part of this layer L1 but only precedes it.
[0121] The feature maps from the preprocessing step VS2 are forwarded to the processing steps PS1, PS2, and PS4, respectively. Each of these processing steps PS1, PS2, and PS4 processes all the feature maps received from the VS2 step independently.
[0122] In step PS1, a convolution with 8 kernels without striding is performed. In step PS2, a convolution with 8 kernels with a striding factor of 2 is performed. In step PS4, a convolution with 11 kernels with a striding factor of 2 is performed.
[0123] In step PS3, a so-called max pooling is performed with a striding factor of 2 for each of the feature maps.
[0124] In step PS5, the 8 feature maps from step PS3 are concatenated with the 8 feature maps from step PS2. In step PS6, the 16 incoming feature maps are convolved with 11 kernels without striding.
[0125] In step PS7, a so-called batch normalization is performed on each of the 11 feature maps.
[0126] In a step PS8, a so-called activation takes place, preferably in the form of a RELU activation.
[0127] In step PS9, a dropout with a dropout factor of 20% is preferably performed during the training phase. This dropout does not occur during the classification phase.
[0128] In step PS10, a convolution with 11 kernels without striding is performed.
[0129] The results of the feature maps, or rather the 11 feature maps from step PS10, are added element by element with the 11 feature maps from step PS4 in step PS11, so that step PS11 again generates 11 feature maps. No striding occurs here.
[0130] In step PS12, a so-called batch normalization takes place.
[0131] In a step PS13, a so-called activation takes place, preferably in the form of a RELU activation.
[0132] In a step PS14, a dropout preferably occurs during the training phase with a dropout factor of, for example, 20%.
[0133] Returning to the Figure 2It can be noted that preferably P=6 such layers follow one another, which are dimensioned such that the last layer LP with P=6 then generates feature maps having a size of 8x8 pixels. Preferably, the last layer LP generates 56 (fifty-six) different such feature maps, wherein then in a Figure 2 In the post-processing step AVS shown, the 56 feature maps are convolved with N=8 kernels to generate the N=8 feature maps FM1,..., FMN.
[0134] In the Figure 5The segmentation convolutional neural network SEG-CNN is shown, which segments the first image B1 or the image information BI1 using the layers LA1, LAQ. Preferably, the so-called downscaling takes place in an optional step DWS. Preferably, the first image in the form of the image data BI1 has a dimensionality of 2400×1700 pixels. This first image can then be reduced to a size of 800×800 pixels by so-called downsampling. The further processing steps from the Figure 26 are explained in detail for the exemplary case where a downscaling DWS of the first image to a size of 800×800 pixels was carried out. However, this is not mandatory, the Convolutional Neural Network CNNS from the Figure 26 can also be dimensioned so that it can process the first image with a size of 2400x1700 pixels without downscaling.
[0135] The Figure 26illustrates details of the Convolutional Neural Network CNNS from the Figure 5 . In Figure 26 In a first step PS21, a convolution with 32 kernels takes place, which can be read off the output layer based on the parameter 32.
[0136] Since the input size of step PS21 has a dimension of 800×800 pixels and the output size of step PS21 also has a dimension of 800×800 pixels, it can be deduced that no striding occurs. For each processing step PS21, ..., P37, the dimensionality of the input size is specified as "input" in parentheses using the two numbers after the "None" specification. For each processing step PS21, ..., P37, the dimensionality of the output size is specified as "output" in parentheses using the two numbers after the "None" specification.
[0137] Through the various steps PS21 to PS37, the first image or image data BI1 is processed into the feature maps M1,.., MS, as already described in the Figure 5 illustrated.
[0138] It should be noted that step PS29 performs a so-called deconvolution or transposed convolution.
[0139] The next steps either perform a folding, as already described in the course of the description of the Figure 25 or a max pooling or a concatenation, as already explained in the description of Figure 25 explained.
[0140] This was done before processing the image data BI1, which is also included in the Figure 5 are illustrated, a so-called downsampling of the first image or the image data BI1, a corresponding upsampling can take place after determining the segmentation map SM before the segmentation map SM is fed to the further image processing step BV.
[0141] The Figure 17a shows an image BX as a fluorescence image of a second color channel or related to a second fluorescent dye for a mixed pattern in which a nucleolar pattern (AC-8, 9,10) of class n=7 is present as well as a pattern of fine or coarse speckling as the fluorescence pattern type of class n=5 (AC-4, AC-5).
[0142] An example of this is the Figure 17bmagnified an interphase cell IZX as well as a mitotic cell MZX. It is clearly visible that the metaphase plate of the mitotic cell MZX is not significantly stained, or rather the mitotic cell with its chromosomes in the metaphase plate is not stained. This information makes it possible to distinguish whether a so-called finely or coarsely speckled pattern (AC-4, AC-5) is present here, or whether a homogeneous pattern (AC-1) is perhaps present. Since the mitotic cell MZX with its metaphase plate is not stained in the chromosome region, homogeneous staining of class n=2 can be ruled out, because with homogeneous staining the mitotic cell in the region of the metaphase plate would also have to be homogeneously stained, which is not the case here.Therefore, by including the mitotic cell in the fluorescence pattern type detection, it can be reliably detected that it is not a homogeneous fluorescence pattern type but a finely speckled fluorescence pattern type with n=5 (AC-4) or a coarsely speckled fluorescence pattern type with n=5 (AC-5).
[0143] The Figure 18 shows another example BY of an image in the second fluorescence channel or related to the second fluorescent dye, highlighting a mitotic cell MZY as well as an interphase cell IZY.
[0144] The Figure 19a shows an enlarged view of this interphase cell IZY and in the Figure 19b the corresponding mitotic cell MZY with its metaphase plate MP.
[0145] The picture BY is from the Figure 18This is an image with three present fluorescence pattern types. This fluorescence pattern image includes fluorescence pattern type n=7, also called nucleolar (AC-8, 9, 10), as well as fluorescence pattern type n=6, also called nuclear dots (AC-6, 7), and fluorescence pattern type n=5, also called finely speckled (AC-4), as a variant of fluorescence pattern type n=5 (AC-4, AC-5). Figure 19a This is indicated by the staining of the nucleolar pattern through the NU areas, the staining of the nucleolar dot pattern through the DO areas and also the staining of the finely speckled pattern through the wider FG area of the nucleus.
[0146] It is particularly advantageous here that, according to the method according to the invention, the mitotic cell MZY must also be present in a partial image, since the metaphase plate MP is not stained, or rather, its chromosomes are not stained. Thus, the area of fine speckling FG can be reliably detected as finely speckled (AC-4) with n=5 and not mistakenly considered homogeneously stained, as could be the case with the homogeneous pattern n=2 (AC-1), since in this homogeneous pattern the metaphase plate MP would have to be significantly stained. Likewise, a pattern staining of the type "densely finely speckled" with n=4 (AC-2) can be excluded, since in this fluorescence pattern type the metaphase plate MP of the mitotic cell MZY would have to be significantly stained.
[0147] This shows once again that the method according to the invention, in which partial images are selected depending on whether they have image segments with at least one mitotic cell of sufficient quality, enables a particularly high quality detection of different fluorescence pattern types.
[0148] The Figure 20 shows another example of a fluorescence image of BZ, in which a partial image area TBZ12 contains a mitotic cell MZZ. This fluorescence image of the second color channel, or rather, based on the second fluorescent dye, clearly shows a so-called dense, finely speckled pattern of the type n=4 (AC-2).
[0149] The Figure 21 shows the exemplary partial image TBZ11 as the first partial image of the first fluorescent dye and the second partial image TBZ12, which corresponds to the first partial image TBZ11, from the second fluorescent color channel. Furthermore, the Figure 21the corresponding segmented sub-image TBZS.
[0150] The Figure 22 shows corresponding feature maps FM1,..., FM8 for the different fluorescence pattern types or fluorescence pattern classes n=1...8 related to the partial images TBZ11, TBZ12 from Figure 21 .
[0151] Clearly visible here is the activation in the feature map FM4 of the corresponding class n=4 for the densely finely speckled pattern as fluorescence pattern type (AC-2), which clearly dominates here.
[0152] The Figure 23a once again the second partial image TBZ12, which shows the Fig.23c enlarged mitotic cell MZZ, in which Figure 23bThe feature map FM4 of the corresponding fluorescence pattern type n=4 is visible, as well as an overlay of this feature map FM4 over the second partial image TBZ12 as the overlaid image OLM4. It is clearly evident that the feature map FM4 focuses on those image areas of the second partial image TBZ12 that contain the interphase cells stained according to the fluorescence pattern type and also the mitotic cell stained according to the fluorescence pattern type.
[0153] Furthermore, the Figure 23c Another enlarged view of the mitotic cell MZZ, which is present in the partial image TBZ12. The clear staining of the metaphase plate of the mitotic cell MZZ allows the detection of the densely speckled fluorescence pattern type n=4.
[0154] The method according to the invention has been described here by, in a step SC2, Figure 1The first and second images are captured. Therefore, a method for detecting the respective potential presences of different cellular fluorescence pattern types on a biological cell substrate comprising human epithelioma cells using digital image processing is proposed, which begins with the step of capturing the first image and the second image.
[0155] As an alternative to step SC2 of capturing the first and second images, a method for digital image processing can also be carried out correspondingly and in an analogous manner, in which such a first image and such a second image are provided or received in a corresponding step and in which the further steps SC3, SC4, SC5 are then carried out.
[0156] The Figure 7Aa proposed computing unit R, which receives a corresponding first image or first image information BI1 as well as a corresponding second image or second image information BI2 in the form of a preferably data signal SIG via an interface DS2. The computing unit R can then output the determined confidence measures PI, the verified confidence measures PI* and / or the detection information DI, preferably via an output interface AS, in particular to a display unit AE.
[0157] This information can preferably also be output via a data interface DS3 in the form of a data network interface in the form of a signal SI3.
[0158] The Figure 7Bshows a proposed data network device, which receives the images or image information B11, BI2 via an interface S4 and a corresponding data signal SIG1. The data network device preferably further comprises an internal data bus IDB, which connects a previously described computing unit R to preferably a memory unit MEM.
[0159] The Figure 7C illustrates a proposed computer program product CPP, which includes instructions that, when executed by a computer, cause the computer to perform the proposed method for digital image processing. This computer program product can, for example, be provided or transmitted in the proposed manner via a data signal SIG2. This transmission can be made to a computer CO or to a data location DSX of the computer CO.
[0160] The Figure 6illustrates an embodiment of the device V1 according to the invention. The device V1 has a holding device HL for the substrate SU. Excitation light AL from an excitation light source LQ is pre-filtered via an optical filter F1 and then guided through an optics O to the substrate by means of a dichroic mirror SP1. Resulting fluorescence radiation or resulting fluorescence light FL then exits from the substrate SU back through the objective O, through the dichroic mirror SP1 and through an optical filter F2. The optical filter F2 filters out a wavelength of the excitation radiation or excitation light AL. The fluorescence light FL is then fed to at least one image capture unit in the form of a camera K1 and preferably to a further camera K2. The fluorescence light FL is split via a dichroic mirror SP2.Fluorescence light FR1 from a first color channel, preferably a red color channel, is filtered out via an optical filter FR and fed to the image acquisition unit K1. The image acquisition unit K1 acquires a first fluorescence image of the substrate SU in the first color channel. Fluorescence radiation FL2 from the second color channel, preferably the green color channel, is filtered out via an optical filter FG and fed to the image acquisition unit K2, which acquires a second fluorescence image of the substrate SU in the second color channel.
[0161] A computing unit R is configured to receive the first fluorescence image in the form of digital image data BI1. Furthermore, the computing unit R is configured to receive the second fluorescence image in the form of digital image data BI2. The computing unit R is further configured to detect, based on the first image, respective image segments which each represent at least one mitotic cell, to select partial images of the first image and corresponding partial images of the second image based on the detected image segments, and to detect, based on the selected partial images of the first image and the selected partial images of the second image, respective actual presences of respective cellular fluorescence pattern types using a convolutional neural network.
[0162] Via a data interface DS1, the device V1 can provide a detection result in the form of detection information DI and / or presence information or prediction information PI, PI*.
[0163] Although some aspects have been described in connection with a device, it is understood that these aspects also represent a description of the corresponding methods, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Similarly, aspects described in connection with or as a method step also represent a description of a corresponding block, detail, or feature of a corresponding device.
[0164] Depending on specific implementation requirements, embodiments of the invention can implement the computing unit R or the data network device DV in hardware and / or software. An implementation of a computing unit R mentioned here can be carried out as at least one computing unit or by several computing units in a network. The implementation can be carried out using a digital storage medium, for example a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, a hard disk or another magnetic or optical memory on which electronically readable control signals are stored, which can interact or interact with a programmable hardware component in such a way that the respective method is carried out.
[0165] A programmable hardware component can be formed as a computing unit by a processor, a computer processor (CPU = Central Processing Unit), a computer, a computer system, an application-specific integrated circuit (ASIC = Application-Specific Integrated Circuit), an integrated circuit (IC = Integrated Circuit), a single-chip system (SOC = System on Chip), a programmable logic element or a field-programmable gate array with a microprocessor (FPGA = Field Programmable Gate Array).
[0166] The digital storage medium can therefore be machine- or computer-readable. Some embodiments thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system or a programmable hardware component such that one of the methods described herein is performed.
[0167] In general, embodiments or parts of the embodiments of the present invention may be implemented as a program, firmware, computer program or computer program product with a program code or as data, wherein the program code or the data is or are effective to perform one of the methods or a part of a method when the program runs on a processor or a programmable hardware component.
[0168] To implement one or more embodiments of the convolutional neural networks proposed here, a person skilled in the art can use an open-source deep learning library called "Keras." Detailed information can be found at https: / / keras.io. Example 1
[0169] The specified system and the method according to the invention were tested on 161 different patient samples. The samples were diluted in a linear dilution series starting at 1:40. This means that the patient samples were diluted and incubated in a linearly increasing sequence starting at 1:40. In this case, samples were diluted in steps of 1:40, 1:80, 1:640, 1:1280, or 1:40, 1:80, 1:160, 1:320, 1:640, 1:1280, 1:2560, 1:5120, 1:10240. However, the system is in no way limited to specified dilution series.
[0170] A pattern was declared present by an expert for a specific sample if it was recognized as present in any dilution or fluorescence image for that specific sample. The pattern did not have to be recognized by the expert in all dilutions or all fluorescence images for that specific sample to be declared present.
[0171] A pattern was declared negative (showing no pattern) by an expert for a specific sample if it was identified as negative (showing no pattern) in all dilutions or in all fluorescence images for the specific sample.
[0172] A pattern was detected as generally present with respect to a specific sample by the method according to the invention if it was detected as present in any dilution or in any fluorescence image with respect to the specific sample. The pattern did not have to be detected by the method according to the invention in all dilutions or in all fluorescence images with respect to the specific sample in order to be generally detected as present.
[0173] A pattern was detected as negative (having no pattern) with respect to a specific sample by the method according to the invention if it was detected as negative (having no pattern) in all dilutions or all fluorescence images with respect to the specific sample.
[0174] For example, in the presence of two different actually present patterns, an expert declared both patterns to be present even if he recognized the two patterns in different dilutions or different fluorescence images of the same sample.
[0175] In the presence of, for example, two different actually present patterns, the two patterns were detected as both present by the method according to the invention even if the method detected the two patterns in different dilutions or different fluorescence images of a same sample.
[0176] For example, if the homogenous pattern was detected at a dilution of 1:40 and another pattern at a different dilution, for example 1:80, both patterns will be output for this sample.
[0177] The Figure 27 The detection rates listed in Table 1 refer to a complete patient sample of all dilutions. Table 1 shows the following parameters for different samples as well as for the case of a negative sample: TA: "True Acceptance" is the number of samples for which an expert has decided that this pattern is present at any dilution and for which the system has decided that this pattern is present at at least one dilution. FA: "False Acceptance" is the number of samples for which an expert has decided that this pattern is not present at all dilutions and for which the system has decided that this pattern is present at at least one dilution. TR: "True Rejection" is the number of samples for which an expert has decided that this pattern is not present at all dilutions and for which the system has decided that this pattern is not present at all dilutions. FR: "False Rejection"; this is the number of samples for which an expert has decided that this pattern is present at any dilution and for which the system has decided that this pattern is absent at at least all dilutions. Sens: Sensitivity of the system as an absolute number. FAR: "False Acceptance Rate" as an absolute number. Spec: Specificity as an absolute number. FRR: "False Rejection Rate" as an absolute number.
[0178] The results are obtained by applying the complete system, i.e. segmentation network, classification network, application of thresholds of the probabilities of the individual patterns and application of thresholds of the brightness of the individual patterns.
[0179] The convolutional neural network for pattern presence detection used a total of 5,949 sub-image tuples for training, of which 25% of the respective pattern types were used for separate validation. A total of 6,463 images were available for the convolutional neural network for segmentation, 4,847 of which were in the actual training set and 1,616 in a validation set.
Claims
1. Method for detecting respective potential presences of respective different cellular fluorescence pattern types on a biological cell substrate (SU) comprising human epithelioma cells, wherein the cellular fluorescence pattern types comprise a plurality of different antinuclear antibody fluorescence pattern types, the method comprising - incubating the cell substrate (SU) with a liquid patient sample which potentially comprises primary antibodies, further with a first fluorescent dye and further with secondary antibodies which are labelled with a second fluorescent dye, - acquiring a first image (B1) which represents staining of the cell substrate by the first fluorescent dye and acquiring a second image (B2) which represents staining of the cell substrate by the second fluorescent dye, - detecting, on the basis of the first image (B1), respective image segments (VMZ) which in each case represent at least one mitotic cell (MZX, MZY) of a sufficient quality, wherein a mitotic cell is of sufficient quality if it is in metaphase. - selecting, on the basis of the detected image segments (VMZ), subimages (TB11) of the first image (B1) and subimages (TB12) corresponding thereto of the second image (B2), which in each case represents at least one mitotic cell (MZX, MZY) of a sufficient quality and - detecting, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network (CNN2) characterized by - determining, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), respective confidence measures (PI, PI*) for the respective actual presences of the respective fluorescence pattern types by means of the convolutional neural network (CNN2). - wherein for a respective subimage tuple a respective subimage confidence measure (PI1) is determined for respective actual subimage presences of respective cellular fluorescence pattern types by means of the convolutional neural network (CNN2), wherein a subimage tuple comprises a subimage (TB11) of the first image (B1) and a corresponding subimage (TB12) of the second image (B2) and a subimage confidence measure corresponds to a vector of confidence measures, wherein a respective confidence measure corresponds to a respective fluorescence pattern type, - wherein the convolutional neural network simultaneously evaluates the subimage of the first image (B1) and the corresponding subimage of the second image (B2), - and determining respective confidence measures for the respective actual presences of the respective fluorescence pattern types on the basis of the subimage confidence measures.
2. Method according to Claim 1, wherein the convolutional neural network (CNN2) comprises an output layer (LP, AVS) which generates a respective feature map (FM1,...,FMN) for a respective cellular fluorescence pattern type, and wherein the convolutional neural network (CNN2) determines a respective confidence measure (P1,...,PN) on the basis of a respective feature map (FM1,...,FMN).
3. Method according to Claim 1, further comprising - segmenting the first image (B1) into image segments of different segment classes, - determining at least one lightness value (HI) for at least one fluorescence pattern type on the basis of one or more image segments of at least one specific segment class and - verifying the confidence measure (PI, PI*) for the actual presence of the at least one fluorescence pattern type on the basis of the lightness value (HI) of the at least one fluorescence pattern type.
4. Method according to Claim 3, wherein the verification of the confidence measure (PI, PI*) is carried out on the basis of the lightness value (HI) and as a function of a user-predeterminable threshold value (H1,...,HN).
5. Method according to Claim 1, further comprising - dividing the second image (B2) into a set of subimages in accordance with a predetermined division scheme, - selecting subimages (TB12) of the second image (B2) on the basis of the detected image segments and selecting subimages corresponding thereto (TB11) of the first image (B1) - and detecting, on the basis of the selected subimages (TB12) of the second image (B2) and on the basis of the selected subimages (TB11) of the first image (B1), respective actual presences of respective cellular fluorescence pattern types by means of the convolutional neural network (CNN2).
6. Computing unit (R) for detecting respective potential presences of respective different cellular fluorescence pattern types, wherein the cellular fluorescence pattern types comprise a plurality of different antinuclear antibody fluorescence pattern types, which, in the course of digital image processing, is configured - to receive a first image (B1) which represents staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, - to receive a second image (B2) which represents staining of the biological cell substrate by a second fluorescent dye, - to detect, on the basis of the first image (B1), respective image segments (VMZ) which in each case represent at least one mitotic cell of a sufficient quality, wherein a mitotic cell is of sufficient quality if it is in metaphase, - to select, on the basis of the detected image segments (VMZ), subimages (TB11) of the first image (B1) and subimages (TB12) corresponding thereto of the second image (B2), which in each case represent at least one mitotic cell (MZX, MZY) of a sufficient quality and - to detect, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network (CNN2) characterized in that the computing unit is further configured - to determine, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), respective confidence measures (PI, PI*) for the respective actual presences of the respective fluorescence pattern types by means of the convolutional neural network (CNN2). - for a respective subimage tuple a respective subimage confidence measure (PI1) is determined for respective actual subimage presences of respective cellular fluorescence pattern types by means of the convolutional neural network (CNN2), wherein a subimage tuple comprises a subimage (TB11) of the first image (B1) and a corresponding subimage (TB12) of the second image (B2) and a subimage confidence measure corresponds to a vector of confidence measures, wherein a respective confidence measure corresponds to a respective fluorescence pattern type, - wherein the convolutional neural network simultaneously evaluates the subimage of the first image (B1) and the corresponding subimage of the second image (B2), - and determining respective confidence measures for the respective actual presences of the respective fluorescence pattern types on the basis of the subimage confidence measures.
7. Data network apparatus (DV) comprising at least one data interface (DS4) for receiving a first image (B1) which represents staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, and for receiving a second image (B2) which represents staining of the biological cell substrate by a second fluorescent dye, characterized by a computing unit (R) according to claim 6.
8. Computer program product (CPP) comprising commands which, on execution of the program by a computer (CO), cause the latter to carry out the method for digital image processing for detecting respective potential presences of respective different cellular fluorescence pattern types, wherein the cellular fluorescence pattern types comprise a plurality of different antinuclear antibody fluorescence pattern types, comprising - receiving a first image (B1) which represents staining of a biological cell substrate by a first fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, - receiving a second image (B2) which represents staining of the biological cell substrate by a second fluorescent dye, - detection, on the basis of the first image (B1), respective image segments (VMZ) which in each case represent at least one mitotic cell (MZX, MZY) of a sufficient quality, wherein a mitotic cell is of sufficient quality if it is in metaphase, - selecting, on the basis of the detected image segments (VMZ), subimages (TB11) of the first image (B1) and subimages (TB12) corresponding thereto of the second image (B2), which in each case represent at least one mitotic cell (MZX, MZY) of a sufficient quality and - detecting, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), of respective actual presences of respective cellular fluorescence pattern types by means of a convolutional neural network (CNN2) characterized by - determining, on the basis of the selected subimages (TB11) of the first image (B1) and of the selected subimages (TB12) of the second image (B2), of respective confidence measures (PI, PI*) for the respective actual presences of the respective fluorescence pattern types by means of the convolutional neural network (CNN2). - for a respective subimage tuple a respective subimage confidence measure (PI1) is determined for respective actual subimage presences of respective cellular fluorescence pattern types by means of the convolutional neural network (CNN2), wherein a subimage tuple comprises a subimage (TB11) of the first image (B1) and a corresponding subimage (TB12) of the second image (B2) and a subimage confidence measure corresponds to a vector of confidence measures, wherein a respective confidence measure corresponds to a respective fluorescence pattern type, - wherein the convolutional neural network simultaneously evaluates the subimage of the first image (B1) and the corresponding subimage of the second image (B2), - and determining respective confidence measures for the respective actual presences of the respective fluorescence pattern types on the basis of the subimage confidence measures.
9. Data carrier signal (SIG2) which transmits the computer program product (CPP) according to claim 8.
