Methods and apparatus for detecting the presence of different antibody fluorescent pattern types

Abstract

The invention relates to a method and a device for detecting the respective potential presence of respective different cell fluorescence pattern types on a biological cell substrate with human epithelioma cells (HEp cells), said fluorescence pattern types comprising different antinuclear antibody fluorescence pattern types. The invention also relates to a method for detecting the potential presence of different cell fluorescence pattern types on a biological cell substrate with human epithelioma cells by digital image processing, as well as a computing unit, a data network device, a computer program product and a data carrier signal therefor.

Description

Technical Field

[0001] The invention relates to a method and a device for detecting the corresponding potential presence of different cell fluorescence pattern types on a biological cell substrate with human epithelioma cells (HEp cells), wherein the fluorescence pattern types include different antinuclear antibody fluorescence pattern types. The invention also relates to a method for detecting the potential presence of different cell fluorescence pattern types on a biological cell substrate with human epithelioma cells by digital image processing, as well as a computing unit, a data network device, a computer program product and a data carrier signal therefor. Background Art

[0002] For the purpose of medical diagnosis, in particular for the detection of autoantibodies in liquid patient samples (such as blood or serum), a known method is to incubate a biological substrate containing human epithelial tumor cells with a liquid patient sample or a diluted liquid patient. Sample. Depending on the presence of certain primary autoantibodies in the patient sample, these primary antibodies will bind to different areas of the mentioned cells. The biological cell substrate is then also incubated with secondary antibodies labeled with so-called fluorescent dyes. Such secondary antibodies can in turn bind to the primary antibodies bound to the cell substrate. After irradiation of the substrate with excitation radiation, the mentioned fluorescent dyes will fluoresce, so that in the corresponding fluorescent image, the primary antibodies of the patient sample are bound to the cells of the cell substrate, and the secondary antibodies are bound to the secondary antibodies and the fluorescent dyes as a localizable fluorescent pattern. Depending on the patient's disease or the presence of specific primary antibodies in the patient sample, different specific fluorescent patterns will be generated. By detecting primary antibodies against so-called autoantibodies to the cell nucleus or nuclear antigens, so-called antinuclear antibody fluorescent patterns are generated. These are also called so-called ANA fluorescent patterns or ANA patterns.

[0003] Such autoantigens are named either based on biochemical characteristics (DNA, histones, ribonucleoproteins: RNPs) or on the diseases to which they are associated.

[0004] During indirect immunofluorescence (IIFT) on human epithelial cells or human epithelioma cells, the task is to identify and classify the different types of fluorescence patterns produced. A single pattern does not necessarily have to appear alone in the fluorescence image, but multiple patterns can also appear simultaneously.

[0005] Antibodies against nuclear antigens are directed against various nuclear components (biochemical substances in the cell nucleus). These include nucleic acids, nuclear proteins, and ribonucleoproteins.

[0006] Different types of antinuclear antibody fluorescence patterns produced are distinguished into different pattern types. Methods for this can be found on the website www.anapatterns.org, where different nuclear patterns are listed and distinguished. An exemplary classification according to this scheme is given inFigure 1 Shown in. Summary of the invention

[0007] The object of the present invention is to provide a method by digital image processing which automatically detects the corresponding potential presence of different antinuclear antibody fluorescence pattern types in a fluorescence image.

[0008] The purpose of the present invention is achieved by a method for detecting the corresponding potential existence of different types of cell fluorescence patterns on a biological cell substrate having human epithelial cells, a device for detecting the corresponding potential existence of different types of cell fluorescence patterns on a biological cell substrate having human epithelial cells by digital image processing, a method for detecting the corresponding potential existence of different types of cell fluorescence patterns on a biological cell substrate having human epithelial cells by digital image processing, a method for digital image processing, a computing unit, a data network device and a computer program product.

[0009] A method for detecting the corresponding potential presence of different cell fluorescence pattern types on a biological cell substrate with human epithelial tumor cells is proposed, wherein the cell fluorescence pattern types include a plurality of different antinuclear antibody fluorescence pattern types. The method comprises different steps: the cell substrate is incubated with a liquid patient sample that potentially contains primary antibodies. The liquid patient sample is preferably diluted patient blood, particularly preferably diluted patient serum. The cell substrate is incubated with secondary antibodies labeled with a fluorescent dye. The cell substrate is also preferably irradiated with excitation radiation. In addition, an overall image is acquired, which represents the staining of the cell substrate with the fluorescent dye.

[0010] Furthermore, a segmented image is determined by segmenting the total image, preferably by a separate and pre-trained convolutional neural network for the segmentation. In particular, the segmented image has at least one segment class, which represents a valid mitotic cell, preferably a mitotic cell in the metaphase stage, and particularly preferably a metaphase plate of a mitotic cell in the metaphase stage.

[0011] In the method according to the invention, corresponding image segments are also detected in the segmented image, each representing a mitotic cell. Such image segments preferably indicate mitotic cells. Such image segments particularly preferably indicate mitotic cells in the so-called metaphase phase. Such image segments particularly preferably indicate the metaphase plate of mitotic cells in the so-called metaphase phase. Since such image segments particularly preferably indicate the metaphase plate, mitotic cells can be detected particularly well in the metaphase phase.

[0012] Then, also based on the detected image segments, sub-images of the overall image each having at least one mitotic cell and corresponding sub-images of the segmented image are selected. Based on the selected sub-images of the overall image and the selected sub-images of the segmented image, the corresponding actual presence of the corresponding cell fluorescence pattern type is detected by a convolutional neural network.

[0013] In particular, the convolutional neural network simultaneously processes a respective tuple of sub-images, such a tuple having at least one selected sub-image of the overall image and a corresponding selected sub-image of the segmented image.

[0014] The overall image represents in particular the staining of the cell substrate by a fluorescent dye in the form of a green fluorescent dye.

[0015] In order to explain one or more possible advantages of the present invention, a more detailed description will now be given.

[0016] Secondary antibodies can bind to those primary antibodies from liquid patient samples and bind to specific areas of the cell base or nucleus, so that according to the presence of different types of primary antibodies, there are corresponding different antibody fluorescence patterns in the total image, which can then be identified. In order to detect different ANA patterns, in principle, the entire fluorescence image or total image with most human epithelial tumor cells (HEp cells) can be analyzed by a convolutional neural network, and such a whole fluorescence image is supplied to the convolutional neural network in order to identify different ANA patterns, so as to detect the corresponding presence of the corresponding fluorescence pattern type. Then, the convolutional neural network (CNN) must analyze all HEp cells in the evaluation image together or at once to identify different antinuclear antibody fluorescence pattern types. For accurate detection, it is advantageous to consider as many HEp cells as possible for statistical reasons. However, there are also various disadvantages here. On the one hand, the computational workload for processing the entire fluorescence image with most HEp cells will be very high. In addition, processing such a large total image with a large number of HEp cells will constitute a large degree of freedom in terms of abstract image information. If the convolutional neural network is trained with such a large entire fluorescence image during the training phase, the amount and variance of the abstract image information may be too large, so that the convolutional neural network cannot fully converge to a state that can achieve reliable detection of different fluorescence pattern types or fluorescence pattern categories during the training phase.

[0017] According to the invention, it is therefore proposed that the overall image is first segmented to obtain a segmented image, since such a segmented image represents or indicates the general position of cell structures or general cell areas in the overall image. On the basis of the segmented image, specific image segments can then first be detected, which represent or indicate mitotic cells, preferably mitotic cells in a specific mitotic phase. Such image segments particularly preferably indicate mitotic cells in the so-called metaphase phase. Such image segments particularly preferably indicate the metaphase plate of mitotic cells in the so-called metaphase phase.

[0018] Sub-images of the segmented image and the overall image can then be selected which each have at least one mitotic cell, in particular at least one mitotic cell in metaphase.In particular, the image segments are smaller than the sub-images in terms of their spatial extension or spatial range.

[0019] This results in the following advantage: the convolutional neural network only has to process the sub-images of the total image stained with the antibody-specific fluorescent dye and the corresponding sub-images of the segmented image, instead of processing the total image at once. The convolutional neural network can therefore be trained for one size of such sub-images and only has to evaluate such sub-images with a limited number of HEp cells, and therefore only has to evaluate a limited amount of abstract image information.

[0020] In the method according to the invention, the convolutional neural network processes jointly or simultaneously a selected sub-image of the total image, which has a patient-specific staining with a fluorescent dye, and a corresponding sub-image of the segmented image, which represents the cell substrate for highlighting the general cell structure. Thus, the convolutional neural network can use the localization information about the cell areas or cell structures indicated in the segmented sub-image with the help of the sub-image of the segmented image, while also detecting the actual patient-specific fluorescence pattern in the sub-image of the total image. In other words: the sub-image of the segmented image helps the convolutional neural network to use the corresponding areas of cells or cell structures that are characteristic for the corresponding pattern type when analyzing the sub-image of the total image.

[0021] It is known from the prior art methods that the cell substrate can be stained with an additional fluorescent dye, preferably with the aid of propidium iodide as a fluorescent dye or indicator dye for staining in the red channel. Such indicator dyes bind non-specifically to cell regions for the presence of primary antibodies in the patient sample, thus allowing a basic identification or indication of different cell regions in the fluorescence image of the red channel. In other words, this optional and additional staining of the cell substrate in the red channel can make more coarse or general cell regions or cell structures visible, so that such structures can be identified or detected in the red channel image.

[0022] However, the method proposed here is sufficient without this optional additional staining of the substrate by an indicator dye, while a fluorescence total image requires only one fluorescent dye, preferably a green fluorescent dye, as input information in order to detect the corresponding actual presence of the corresponding cell fluorescence pattern type. Particularly advantageously, in a particularly advantageous manner, the proposed method uses other image information instead of the indicator dye image or the red channel image, namely a segmented image derived from the patient-specific stained total image, and then uses the corresponding sub-images of the total image and the segmented image to detect the corresponding actual presence of the corresponding cell fluorescence pattern type by means of a convolutional neural network.

[0023] The task of providing, in addition to the image information of the overall image, further image information of cell structures having a general staining or indication is solved in the prior art by the necessity of using indicator dyes, whereas now, according to the invention, this necessity is dispensed with and a segmented image derived from the overall image, in particular in green, is used instead.

[0024] Preferably, the total image is segmented with the aid of a network separate from the aforementioned convolutional neural network, in particular an additional, separate and pre-trained convolutional neural network, to determine the segmented image. The inventors have recognized that, instead of classical image processing methods, such as Otsu's threshold method, the use of a separate convolutional neural network for segmentation can lead to significantly more reliable segmentation results. For so-called positive patients, the total image of pattern-specific staining sometimes has two separate staining pattern types, for example, up to n=8 different pattern types can be considered, but this is unknown before segmentation, so that it is uncertain which of the n=8 potential pattern types is present in the total image, so the segmentation algorithm must also recognize general cell structures independently of the patient-specific pattern types present. For example, without loss of generality, assuming that there are n=8 different pattern types, and that k=2 pattern types are always present in the positive total image, then for combinatorial reasons we get

[0025]

[0026] Different combinations, in this case n=8 and k=2, i.e. 28 different patient-specific pattern stainings, which the segmentation algorithm must capture in order to reliably find the general cell structure. This is only difficult to achieve with classical image processing algorithms with usual feature extraction, since the features to be extracted must be individually delineated or "designed", so 28 pattern combinations must be captured here. Since a separate convolutional neural network is used for segmentation according to the invention, the separate convolutional neural network can be trained in advance with different total images of different pattern type combinations for the task of segmenting the total image.

[0027] Advantageous embodiments of the invention are explained in more detail in the following description, partly with reference to the drawings.

[0028] Preferably, image segments are detected which in the segmented image each represent a valid mitotic cell in that the respective image segment indicates a respective metaphase plate of a respective mitotic cell in the metaphase phase.

[0029] Preferably, the method further comprises: detecting corresponding image segments representing mitotic cells of sufficient quality in the total image based on the segmented image, and selecting a sub-image of the total image and a corresponding sub-image of the segmented image based on the detected image segments representing at least one mitotic cell of sufficient quality.

[0030] Preferably, the method further comprises: determining a corresponding confidence measure for the actual presence of a corresponding fluorescence pattern type by means of a convolutional neural network based on the selected sub-image of the overall image and the selected sub-image of the segmented image.

[0031] The convolutional neural network preferably has an output layer, which generates a corresponding feature map for a corresponding cell fluorescence pattern type, and the convolutional neural network determines a corresponding confidence measure based on the corresponding feature map.

[0032] Preferably, the method also includes: segmenting the total image into image segments of different segment categories, determining at least one brightness value for at least one fluorescent pattern type based on one or more image segments of at least one specific segment category, and verifying the confidence measure of the at least one fluorescent pattern type based on the brightness value of the at least one fluorescent pattern type.

[0033] Preferably, the confidence measure is validated based on brightness values ​​and according to a threshold that can be preset by a user.

[0034] Preferably, the method also includes: for a corresponding sub-image tuple having a sub-image of the total image and a corresponding sub-image of the segmented image, determining a corresponding sub-image confidence measure for the corresponding actual sub-image existence of the corresponding cell fluorescence pattern type by a convolutional neural network, and determining a corresponding confidence measure for the corresponding actual existence of the corresponding fluorescence pattern type based on the sub-image confidence measure.

[0035] Preferably, the method further comprises: dividing the total image into a set of sub-images according to a preset division scheme, selecting a sub-image of the total image based on the detected image fragments, and selecting a corresponding sub-image of the segmented image, and based on the selected sub-image of the total image and the selected sub-image of the segmented image, detecting the corresponding actual existence of the corresponding cell fluorescence pattern type through a convolutional neural network.

[0036] Furthermore, a device according to the invention is proposed for detecting the respective potential presence of respective different types of cell fluorescence patterns on a biological cell substrate with human epithelioma cells by digital image processing. The device comprises a holding device for a biological substrate, which has been incubated with a liquid patient sample potentially containing primary antibodies and furthermore with secondary antibodies labeled with fluorescent dyes. The device further comprises at least one image acquisition unit for acquiring a total image, which represents the staining of the cell substrate with fluorescent dyes. The device is characterized in that at least one computing unit is provided, which is designed to determine segmented images by segmenting the total image, to detect respective image segments in the segmented images, which respectively represent mitotic cells, to select sub-images of the total image, which respectively have at least one mitotic cell, and to select corresponding sub-images of the segmented images, based on the detected image segments, and to detect the respective actual presence of the respective cell fluorescence pattern types based on the selected sub-images of the total image and the selected sub-images of the segmented images by means of a convolutional neural network.

[0037] Furthermore, a method for detecting the respective potential presence of respective different cell fluorescence pattern types on a biological cell substrate with human epithelioma cells by digital image processing is proposed. The method comprises: acquiring a total image, which represents the staining of the cell substrate by a fluorescent dye, determining a segmented image by segmenting the total image, detecting respective image segments in the segmented image, which respectively represent mitotic cells, selecting sub-images of the total image, each having at least one mitotic cell, and selecting corresponding sub-images of the segmented image based on the detected image segments, and detecting the respective actual presence of the respective cell fluorescence pattern type based on the selected sub-images of the total image and the selected sub-images of the segmented image by means of a convolutional neural network.

[0038] A method for digital image processing is also proposed. The method comprises: receiving a total image, the total image representing the staining of a biological cell substrate by a fluorescent dye, wherein the biological cell substrate has human epithelial tumor cells, determining a segmented image by segmenting the total image, detecting corresponding image segments representing mitotic cells in the segmented image, selecting sub-images of the total image each having at least one mitotic cell and selecting corresponding sub-images of the segmented image based on the detected image segments, and detecting the corresponding actual presence of corresponding cell fluorescence pattern types based on the selected sub-images of the total image and the selected sub-images of the segmented image by a convolutional neural network.

[0039] A computing unit is also proposed, which is constructed to receive a total image during digital image processing, the total image representing the staining of a biological cell substrate by fluorescent dyes, wherein the biological cell substrate has human epithelial tumor cells, determine a segmented image by segmenting the total image, detect corresponding image segments representing mitotic cells in the segmented image, select sub-images of the total image each having at least one mitotic cell based on the detected image segments and select corresponding sub-images of the segmented image, and detect the corresponding actual existence of corresponding cell fluorescence pattern types based on the selected sub-images of the total image and the selected sub-images of the segmented image through a convolutional neural network.

[0040] A data network device is also proposed, comprising: at least one data interface for receiving a total image, the total image representing the staining of a biological cell substrate by a fluorescent dye, wherein the biological cell substrate has human epithelial tumor cells; and the above-mentioned computing unit according to the present invention.

[0041] A computer program product is also proposed, comprising instructions which, when the program is executed by a computer, cause the computer to perform the method for digital image processing according to the invention.

[0042] A data carrier signal is also proposed, which transmits the computer program product. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be described in more detail below based on specific embodiments and with reference to the accompanying drawings without limiting the general concept of the present invention.

[0044] Figure 1 The steps of the method according to the invention are shown according to a preferred embodiment.

[0045] Figure 2 The processing of the sub-image by the convolutional neural network according to the preferred embodiment is shown,

[0046] Figure 3 The processing of multiple sub-images by a convolutional neural network according to a preferred embodiment is shown.

[0047] Figure 4 showing steps for verifying a confidence measure based on a determined brightness value according to a preferred embodiment,

[0048] Figure 5 Detailed steps of the method for segmenting the total image according to the preferred embodiment are shown,

[0049] Figure 6 shows the proposed device according to a preferred embodiment,

[0050] Figure 7a shows the proposed computing unit according to a preferred embodiment,

[0051] Figure 7b shows the proposed data network device according to a preferred embodiment,

[0052] Figure 7c shows the proposed computer program product and data carrier signal according to a preferred embodiment,

[0053] Figure 8 shows the total image or the entire fluorescence image,

[0054] Fig. 9 Different exemplary antinuclear antibody fluorescence pattern types are shown,

[0055] Fig.10a Shows the segmented total image,

[0056] Fig.10b Show Fig.10a Illustration of

[0057] Fig.11 shows the division of the total image into sub-images and the highlighting of different mitotic cells,

[0058] Fig.12 shows a sub-image of the overall image and a corresponding sub-image of the segmented image,

[0059] Fig.13 Shows the Fig.12 Different fluorescent pattern types or different feature maps of fluorescent pattern categories of sub-images,

[0060] Fig.14a Show Fig.12 A sub-image of the total image,

[0061] Fig.14b Characteristic graphs showing specific fluorescent pattern types and Fig.14a The associated superimposed image of the sub-image,

[0062] Fig.14c Characteristic diagrams showing additional specific fluorescent pattern types and for Fig.14a The associated superimposed image of the sub-image,

[0063] Fig.15a Another first fluorescent image is shown,

[0064] Fig.15b Show Fig.15a Fluorescence images of highlighted interphase cells and highlighted mitotic cells,

[0065] Fig.16Another second fluorescent pattern image is shown,

[0066] Fig.17a Show Fig.16 The fluorescent pattern image highlights the interphase cells.

[0067] Fig.17b Show Fig.16 The fluorescent pattern image highlights the mitotic cells.

[0068] Fig.18 A classification scheme showing different fluorescent pattern types,

[0069] FIG. 19 shows the layers of a convolutional neural network for identifying the type of fluorescent pattern according to a preferred embodiment,

[0070] FIG. 20 shows a convolutional neural network for image segmentation according to a preferred embodiment,

[0071] Fig.21a shows a fluorescent image of a cell with active mitosis,

[0072] Figure 21b Show Fig.21a The segmentation result of the fluorescence image is

[0073] Fig. 22 An example of experimental results is shown. DETAILED DESCRIPTION

[0074] Figure 8 An overall image or fluorescence pattern image B is shown with a grey staining which represents the staining of the cell substrate by the fluorescent dye in the so-called green channel.

[0075] Due to the incubation of the cell substrate shown here with diluted patient-specific serum and the binding of the primary autoantibodies produced thereby, staining in the green channel occurs due to the further incubation of the cell substrate with secondary antibodies labeled with a fluorescent dye. The fluorescent dye is preferably fluorescein isothiocyanate (FITC).

[0076] The patterned staining of the cell regions in image B allows for the presence of differentially identified antinuclear fluorescent pattern types. Figure 8 In the image B, there does not necessarily have to be only a single staining pattern or a single antinuclear antibody fluorescence pattern of a specific pattern type; a plurality of such antinuclear antibody fluorescence pattern types may also appear simultaneously.

[0077] Image B can be obtained by Figure 6 The image acquisition unit in the form of a camera K of the device V1 acquires.

[0078] Fig. 9Different antinuclear antibody fluorescence pattern types BM1, ..., BM8 are shown. If no other antinuclear antibody fluorescence pattern types are to be detected, pattern type BM1 represents so-called negative staining. This corresponds to Fig.18 The pattern AC-0 in FIG. 1 has the pattern index n=1 used here by way of example. Fig.18 The exemplary pattern classification in comes from the following source: www.anapatterns.org / trees-full.php.

[0079] Pattern BM2 is a so-called uniform dyeing or pattern type: uniform, which is Fig.18 It is denoted as AC-1 in FIG. 1 , with the pattern index n=2 used here as an example.

[0080] The pattern type of pattern BM3 is the so-called centromeric staining, which is Fig.18 It is denoted as AC-3 in FIG. 1 , with the pattern index n=3 used here as an example.

[0081] Pattern type BM4 is a so-called dense fine spot pattern. Fig.18 It is represented as pattern type AC-2 in FIG. 1 , with pattern index n=4 used here as an example.

[0082] Pattern type BM5 is a so-called fine or coarse speckle pattern, which Fig.18 In the figure, it is indicated as AC4 or AC5, with the pattern index n=5 used as an example here.

[0083] Pattern type BM6 as a pattern type is a so-called kernel point, which is Fig.18 In the figure, pattern types AC-6, 7 are indicated, with the pattern index n=6 used here as an example.

[0084] Pattern type BM7 is a so-called nucleolar pattern type, which Fig.18 In the figure, they are represented as AC-8, 9, 10, with the pattern index n=7 used as an example here.

[0085] Pattern type BM8 is the so-called nuclear envelope or nuclear membrane pattern type, which is Fig.18 In FIG. 1 , pattern types AC-11, 12 are indicated, with the pattern index n=8 used here as an example.

[0086] Thus, different antinuclear antibody fluorescence pattern types include uniform pattern (AC-1), speckled pattern (AC-2, 4, 5), centromere pattern (AC-3), nuclear dot pattern (AC-6, 7), nucleolar pattern (AC-8, 9, 10), and fluorescence pattern type of nuclear envelope (AC-11, 12). In case the patient sample has no specific primary antinuclear antibodies, another fluorescence pattern type category is the so-called negative category (AC-0).

[0087] Figure 8 Image B is an exemplary selected fluorescence image, which has both uniform pattern (AC-1) and fine or coarse speckle pattern (AC-4, AC-5) patterns. Now it is also discussed below how the convolutional neural network proposed here can detect these two pattern types.

[0088] Figure 1 The essential steps of method V according to the invention are shown. In a first step SC1 , a cell substrate is incubated.

[0089] In step SC2 , an overall image is acquired, thereby providing image information BI.

[0090] In step SC2A, the segmented image is determined by segmenting the overall image, thereby obtaining so-called segmentation information SI.

[0091] This segmentation information SI is used as Fig.10a The segmented image SB in is shown. Fig.10b The different segment classes s=1...S for different gray value levels S=5 are shown, which indicate here Fig.12 Different fragment categories in an image SB. The first fragment category s=1 is the so-called background category. The second category s=2 indicates a poor mitotic cell or a mitotic cell of insufficient quality, alternatively also referred to as an invalid mitotic cell. For the purposes of the present application, a mitotic cell is of insufficient quality or invalid if it is in an incorrect or invalid mitotic phase. Another category s=3 indicates an interphase cell. Another category s=4 indicates an area with a nucleolus. Another category s=5 indicates a so-called good mitotic cell or a mitotic cell of sufficient quality, also referred to as a valid mitotic cell. Such an image fragment of category s=5 preferably indicates a mitotic cell in the so-called metaphase stage. Such an image fragment particularly preferably indicates a metaphase plate of a mitotic cell in the so-called metaphase stage. For the purposes of the present application, a mitotic cell is of sufficient quality or is valid if it is in a correct or valid mitotic phase. A correct or valid mitotic phase is a so-called metaphase. Therefore, a mitotic cell that is not in metaphase is a false or invalid mitotic cell. In particular, the segmented image thus has at least one segment class which represents or indicates a metaphase plate of a mitotic cell in the metaphase stage. Fig.10a The image in the middle is pure white.

[0092] In the next step SC3, corresponding image segments are detected in the segmented image, which image segments respectively represent at least one mitotic cell, in particular at least one valid mitotic cell. For the purposes of the present application, a mitotic cell is of sufficient quality or is valid if it is in a correct or valid mitotic phase. A correct or valid mitotic phase is the so-called metaphase. Therefore, a mitotic cell that is not in metaphase is a false or invalid mitotic cell. It is therefore preferred to detect image segments that, in the segmented image, respectively represent valid mitotic cells in such a way that these corresponding image segments indicate the corresponding metaphase plates of the corresponding mitotic cells that are in the metaphase phase. These Fig.10a The image in the middle is pure white.

[0093] according to Figure 1 Then, in step SC4, based on the detected image segments, sub-images of the total image are selected, each of which preferably has at least one mitotic cell of sufficient quality, and corresponding sub-images of the segmented image are selected. Thus, preferably, sub-images are selected, each of which has at least one image segment, which in turn indicates the metaphase plate of a mitotic cell in the metaphase stage.

[0094] to this end, Fig.11 Image B is shown again, which is divided into sub-images TBA, TBB, TBC, TBD and further sub-images. Valid and preferably mitotic cell VMZs of sufficient quality or in metaphase are indicated by white squares. White circles indicate invalid mitotic cell IMZs of insufficient quality. Sub-image TBB is not selected in this example because it does not have valid mitotic cell VMZs.

[0095] The division of the image B into corresponding sub-images can preferably be carried out in such a way that Fig.10a The position information of the mitotic cells in the image SB is used to infer whether such valid mitotic cells are present in the sub-image of the image B. For this purpose, Fig.11 The adopted division of image B into sub-images and the corresponding division of detected image segments are shown, which have valid mitotic cells VM of sufficient quality.

[0096] according to Figure 1 In step SC4, sub-images of the total image each having at least one mitotic cell and corresponding sub-images of the segmented image are selected based on the detected image segment VMZ.

[0097] Then, in step SC5 , the corresponding actual presence of the corresponding cell fluorescence pattern type is detected by a convolutional neural network based on the selected sub-image of the total image and the selected sub-image of the segmented image.

[0098] By preferably using image segments with mitotic cells of sufficient quality and also selecting sub-images based on those detected image segments, which respectively indicate at least one mitotic cell of sufficient quality, it is ensured that the mitotic cells to be observed or the mitotic cells present in the sub-images are in the correct phase to serve as valid and reliable information for the detection of different types of antinuclear antibody fluorescence patterns as effective mitotic cell representatives with regard to metaphase plate staining of mitotic cells.

[0099] Figure 2 The basic steps of a convolutional neural network CNN1 in one embodiment are shown for determining detection information DI1 about the presence of an antinuclear antibody fluorescence pattern type for a sub-image TB11 of the total image by simultaneously processing or analyzing a sub-image TB11 of the total image and a sub-image TB12 of the segmented image corresponding thereto. Therefore, the sub-images TB11 and TB12 constitute a sub-image tuple, which is preferably normalized to a value range of 0 to 1 or a grayscale value range.

[0100] Fig.12 An exemplary partial image TBA is shown as a partial image TB11 of the overall image B and the corresponding partial images TBS, TB12 from the segmented image SB.

[0101] It can be clearly seen that the sub-image TB11 has valid mitotic cells or mitotic cells of sufficient quality, which are indicated as VMZ in the segmented sub-images TBS, TB12. The segmented image section VMZ shown in white represents or indicates in particular the metaphase plate of a mitotic cell in the metaphase phase.

[0102] Such sub-image tuples TB11, TB12 are then based on Figure 2 The convolutional neural network CNN1 processes the sub-images TB12 of the segmented image SB simultaneously or jointly. Thus, the convolutional neural network can utilize the localization information related to the cell area or cell structure indicated in the segmented sub-image TB12 by means of the sub-image TB12 of the segmented image SB, while also detecting the actual patient-specific fluorescence pattern in the sub-image TB11 of the total image. In other words: the sub-image TB12 of the segmented image SB helps the convolutional neural network to utilize the corresponding characteristic areas of the cells or cell structures for the corresponding pattern types when analyzing the sub-image TB11 of the total image B.

[0103] The convolutional neural network CNN1 then determines the detection information DI1 in order to output a value for each of N different classes, in this case by way of example with the indices n=1 . . . N and N=8, which is within the value range D n ∈{0,1} as a Boolean value to detect or indicate the presence of a pattern. The detection information DI1 for all N pattern types or pattern categories with values ​​D11, ..., D1N then corresponds to a vector:

[0104]

[0105] according to Figure 2 , and then processed on multiple layers L1, ..., LP of the convolutional neural network CNN1, each layer L1, ..., LP has at least one convolutional layer.

[0106] The convolutional neural network CNN1 then determines a corresponding feature map FM1, ..., FMN for each of the N=8 classes or N=8 fluorescence pattern types. Based on the corresponding feature map FM1, ..., FMN, a corresponding confidence measure P1, ..., PN or a corresponding prediction value P1, ..., PN is then determined regarding the corresponding actual presence of the corresponding fluorescence pattern type with the indicator n=1 ... N.

[0107] This is preferably achieved by reducing the feature map FM1 to a single scalar value or average value (which is preferably also referred to as Logit) as value LO1 by means of a pooling layer PL or a pooling layer function PL (which preferably performs a so-called global average pooling).

[0108] The logit values ​​LO1, ..., LON of the N classes or fluorescence pattern types are then individually subjected to a so-called sigmoid function SF in order to determine, for the respective class n, a respective prediction value or a respective confidence level Pn with the indicator n based on the respective feature map FMn with the indicator n. The advantage resulting from this structure of the convolutional neural network is that for each individual fluorescence pattern type there is a separate detection channel and not only a single pattern or a single pattern type can be detected as present, but also multiple pattern types can be detected simultaneously.

[0109] Determining the corresponding confidence measure Pn as the prediction information or the existence information PI1 enables the prediction value or the confidence measure to be output in the form of a vector:

[0110]

[0111] Among them, the value range is 0≤P n ≤1.

[0112] It is advantageous if such a confidence measure Pn is determined and preferably also output, since the user can use the respective confidence measure in a later final diagnosis as an indication for the presence of the respective pattern.

[0113] The subnetwork TCNN1 of the convolutional neural network CNN1 (which is used to determine the confidence measure Pn based on the sub-images TB11, TB12) can be applied separately to the corresponding sub-image tuples TB11, TB12 (the sub-image tuple consists of the sub-image TB11 of the total image and the corresponding sub-image TB12 of the segmented image), as described in detail later.

[0114] Figure 2 Further steps are also shown, by which detection results or detection values ​​D11 , . . . , D1N can be determined based on confidence measures or prediction values ​​(which can be summarized as prediction information PI1 ), which in turn can be summarized as detection information DI1 .

[0115] The confidence measure Pn can then be evaluated by a threshold function TS1, applying a threshold value T1 which can preferably be preset by the user or preset in another way, in order to determine the corresponding detection information DI1. If the confidence measure Pn exceeds the threshold value T1 of 0.5, the detection value D11 can preferably assume the value 1, that is, indicate the presence of a pattern with the indicator n=1.

[0116] Using the corresponding individual threshold values ​​T1 , . . . , TN, preferably for the corresponding patterns, the application of further threshold value operations TS1 , . . . , TSN then enables the determination of the detection values ​​D11 , . . . , D1N.

[0117] Figure 2 The convolutional neural network CNN1 in has a sub-convolutional neural network TCNN1, which is then Figure 3 Used in.

[0118] Fig.13 Exemplary feature maps FM1, ..., FM8 are shown for detecting Fig.12 Different fluorescent pattern types or example patterns of the sub-image TB11 in FIG. Such patterns are exemplarily shown in FIG. Fig. 9 is shown in and is for Fig.18 Examples of corresponding fluorescent pattern types are shown with corresponding reference numerals in FIG. Fig.13 The feature map FM2 in the figure shows that Fig.12The brightness level in the feature map FM2 therefore indicates the possible presence or the probability of the presence of a uniform fluorescence pattern type in the sub-image TB11. Furthermore, the feature map FM5 shows the possible presence or the probability of the presence of a fine speckle or coarse speckle pattern in the corresponding sub-image TB11. For the other feature maps FM1, FM3, FM4, FM6, FM7, FM8, no activation or a relatively small activation occurs. Fig.13 The feature maps FM1, ..., FM8 in are initially determined with a resolution of 8×8 pixels and then interpolated to a resolution of 512×512 pixels of the sub-image TB11 by interpolation.

[0119] Fig.14a Show again Fig.12 The sub-image TB11 together with Fig.14b The feature map FM5 in FIG. 1 and the superposition of the feature map FM5 on the sub-image TB11 are used as the superimposed feature map OLM5. It can be clearly seen here that the feature map FM5 has been successfully generated with the help of the proposed convolutional neural network, which highlights or identifies the cell nucleus with fine or thick spots.

[0120] Fig.14c Show Fig.13 The feature map FM2 for uniform patterns and the superposition of Fig.14a Here, it can also be seen that the region representing the cell nucleus with uniform staining or uniform pattern type has been successfully highlighted or detected from the sub-image TB12 with the help of the proposed convolutional neural network.

[0121] If you observe Fig.12 From the segmented sub-images TBS and TB12, it can be seen that the VMZ of valid mitotic cells with metaphase plates are exactly those that are detected according to the uniform pattern. Fig.14c Cells detected in the superimposed feature map OLM2. Although in principle Fig.14b There may be uniform coloring on the interphase cells selected in the superimposed feature map OLM5, and the uniform coloring is also covered by the spot coloring of these interphase cells, so that the detection of the uniform pattern is only in Fig.14b These interphase units in cannot be reliably achieved in the superimposed feature map OLM5. However, by ensuring Fig.12 The selected sub-image TB11 in any case has valid mitotic cells VMZ, and it is advantageously possible to identify a uniform pattern staining or a uniform fluorescence pattern type precisely in these valid mitotic cells, for example by jointly observing Fig.14c The superimposed feature map of OLM2 and Fig.14aThe sub-image TB11 of FIG. 14 and the segmented sub-image TB12 of FIG. 14 , which is compared thereto, can be seen.

[0122] Figure 3 A further advantageous embodiment of the method according to the invention is shown. The convolutional neural network CNN2 shown here uses Figure 2 The sub-convolutional neural network TCNN1 is used to determine the sub-image confidence measures PI1, ..., PIJ for the corresponding sub-image tuples TB11, TB12, ..., TBJ1, TBJ2 with tuple indicators j = 1 ... J, as shown above Figure 2 In order to subsequently determine the respective actual presence of the respective fluorescence pattern type and the corresponding confidence measure associated therewith as presence information or prediction information PI based on the sub-image confidence measure, as shown and explained in detail in FIG. Here, the information PI corresponds to a vector:

[0123]

[0124] Among them, the corresponding confidence measure of index n For the corresponding fluorescent pattern type.

[0125] The obtained index is the sub-image confidence measure for j=1…J Then, in the averaging step MS, for all selected j=1...J sub-images or sub-image tuples, the following is determined:

[0126]

[0127] The confidence measure

[0128]

[0129] It can then be output as prediction information or confidence measure information PI.

[0130] This embodiment of the method according to the invention is advantageous because the convolutional neural network CNN2 does not have to process all information of the entire fluorescence image B and the entire segmentation image SB simultaneously, but can evaluate the sub-image tuples separately in separate, mutually separate processing paths.

[0131] The confidence measure PI or

[0132]

[0133] In the respective threshold value evaluation steps TS1, ..., TSN, using preferably predeterminable or otherwise predeterminable threshold values ​​T1, ..., TN, the respective detection information DI with the respective detection values ​​D1, ..., DN with the index n=1 ... N is evaluated as the following output:

[0134]

[0135] Figure 5 Show Figure 1 The details of step SC3 are provided for detecting image segments representing at least one mitotic cell, preferably of sufficient quality, by analyzing the total image or its image information BI. This is achieved with the aid of a segmentation convolutional neural network SEG-CNN. The segmentation convolutional neural network SEG-CNN has steps or layers LA1, ..., LAQ, which can also be understood as convolutional neural networks. The segmentation is performed by a further convolutional neural network SEG-CNN, which is different from the convolutional neural network CNN2, which is used to detect the corresponding actual presence of corresponding cell fluorescence pattern types by means of a convolutional neural network based on selected sub-images of the total image and selected sub-images of the segmented image. It is advantageous to divide the segmentation task and the detection task onto two different convolutional neural networks, because the convolutional neural network used for detection does not have to be trained for segmentation, and thus the convolutional neural network used for segmentation can be trained in a particularly targeted manner.

[0136] according to Figure 5 , the total image or the corresponding image information BI is supplied to the convolutional neural network CNNS for segmentation.

[0137] exist Figure 5 A segmentation convolutional neural network SEG-CNN is shown, which segments an input image with the aid of a plurality of layers LA1, LAQ. In this case, the image data BI are preferably standardized to a value range from 0 to 1. The so-called downscaling also preferably occurs in the optional DWS step. The image of the image data BI preferably has a dimension of 2400×1700 pixels. The image can then be downscaled to a size of 800×800 pixels by so-called downsampling. The further processing steps are explained in detail here for the exemplary case of downscaling DWS of the total image to a size of 800×800 pixels. However, this is not absolutely necessary, and the segmentation convolutional neural network SEG-CNN can also be designed in terms of its dimensions so that it can process a total image of 2400x1700 pixels without downscaling.

[0138] The convolutional neural network CNNS has a plurality of layers LA1, ..., LAQ, each of which has at least one convolutional layer. Then, the convolutional neural network CNNS generates a corresponding activation map M1, ..., MS for each segment category s=1...S, where, for example, S=5 categories, such as Fig.10b The activation maps M1, ..., MS each preferably have the same size or resolution of preferably 800x800 pixels as the total image or image data BI after downscaling DWS.

[0139] The activation map M1 for the first segment class s=1 indicates in its first pixel PI1 with the value present there to what extent the corresponding pixel of the image B or the image information BI belongs to the first segment class. The corresponding individual pixels PI1, ...PIS of the activation maps M1, ...MS respectively indicate to what extent the pixels of the image B or the image information BI located at the same position belong to the corresponding class. By associating the values ​​of the activation maps M1, M2, ..., MS at the same pixel position or the corresponding values ​​of these corresponding pixel positions PI1, PI2, ..., PIS by means of the so-called Softmax function SMX, a value can be generated for the probability map PM1 at the corresponding pixel position PIX1, which indicates with which probability in the value range of 0 to 1 the corresponding pixel of the image B belongs to the first class s=1. At the same time, the corresponding probabilities are determined for the probability maps PM1, ..., PMS at the corresponding or identical pixel positions PIX1, ..., PIX2 by the Softmax function, which respectively indicate with which probability in the value range of 0 to 1 the corresponding pixel of the image B belongs to the corresponding class of the indicator s=1...S. The axis extensions x and y shown here indicate corresponding indices for corresponding pixel positions within the probability maps PM1, ..., PMS. The probability maps PM1, ..., PMS preferably have the same size or resolution of preferably 800×800 pixels as the image B or image data BI after downscaling DWS and as the activation maps M1, M2, ..., MS,

[0140] A segmentation map SM is then determined, which has a value from the value range s∈{1, ..., S} at the corresponding pixel position PIX1 and indicates to which segmentation class s=1...S the corresponding pixel at the position PIX1 belongs. To this end, the corresponding pixel values ​​of the corresponding pixel position PIX1 are associated by all probability maps PM1, ..., PMS with the aid of the Argmax function. The pixel of the segmentation map SM with the index x, y then contains the index value of the probability map PM1, ..., PMS for which the corresponding probability value at the pixel position is the highest. This can be determined according to the following:

[0141]

[0142] The segmentation map SM then contains for each individual pixel a value from the value range s=1 . . . S which assigns the respective pixel to the respective segment class.

[0143] The segmentation map SM preferably has the same size or resolution of preferably 800x800 pixels as the image B or the image data BI after downscaling DWS and the activation maps M1 , . . . , MS and the probability maps PM1 , . . . , PMS.

[0144] If a so-called downscaling of the image B or the image information B took place in the optional step DWS before the segmentation, then a so-called upscaling of the segmentation map SM can preferably be carried out in the preferably performed step UP back to the original image resolution of the image B, preferably 800x800 pixels, for example 2400x1700 pixels.

[0145] The segmentation map SM can then be further processed by means of digital image processing in an image processing step BV by conventional image processing measures (e.g. erosion, dilation and / or contour matching) to merge sub-surfaces or segmented surfaces, so as to then arrive at a modified segmentation map SM' which can be output as segmentation information SI. This information with the segmentation SI or the segmentation image SB is then Fig.10a Shown in.

[0146] Figure 5 It shows how the overall image can be segmented into image segments of different segment classes, which can be Fig.10b At least one of the plurality of segment categories preferably indicates a metaphase stage of a mitotic cell. In other words: at least one of the plurality of segment categories preferably indicates a metaphase plate of a mitotic cell in a metaphase stage.

[0147] Segmentation information SI, for example Fig.10a The segmented image SB shown in FIG. 1 can then be used to determine at least one brightness value for at least one fluorescent pattern type based on one or more image segments of the at least one fluorescent pattern type, and to verify at least one confidence measure for a fluorescent pattern type based on at least one such brightness value.

[0148] to this end, Figure 5 The possibility of extracting at least one brightness value of at least one type of fluorescence pattern from the image B or the image information BI based on the image B or the image information BI and based on the segment information SI is shown. This takes place in a step SH for determining the brightness.

[0149] according to Figure 5, determining a brightness value of at least one fluorescent pattern type based on one or more image segments of at least one specific segment class, and verifying a confidence measure of the fluorescent pattern type based on the brightness value of the fluorescent pattern type.

[0150] For different n=1...N categories, the corresponding brightness values ​​h with indices n=1..N n Then it can be used as a vector:

[0151]

[0152] Determined and can be output as brightness information HI.

[0153] For example, if the brightness value of the pattern "dense fine spots" (AC-2) with pattern type n=4 is used, the brightness value can be determined as follows, that is, using the following interphase cells with segment class s=3 (which are also Fig.10a ), the interphase cells have their Figure 8 The brightness value at the corresponding point in image B. For example, the 0.65 quantile of the brightness value can be determined for each interphase cell with index s=3 or each segment surface, and then the multiple quantile brightness values ​​of multiple interphase cells obtained are averaged again by averaging to determine the brightness value h4 of the fluorescent pattern with index n=4.

[0154] For the so-called fine or coarse spotted patterns of the AC-4, AC-5 types with index n=5, the 0.65 quantile of the brightness values ​​of the interphase cells of category s=3 or the corresponding segmented surfaces can then be used, and then these brightness values ​​are averaged to determine the brightness value h5 for the fluorescent pattern with index n=5.

[0155] For the so-called uniform pattern (AC-1) of index n=2, Fig. 9 As shown in the example pattern BM2, the quantile of 0.5 can be first determined for each valid mitotic cell of category s=5 or each corresponding segment surface, and then the quantiles of all valid mitotic cells are averaged to determine the brightness value h2 for the fluorescence pattern with index n=2.

[0156] The brightness value determined in this way

[0157]

[0158] It can then be provided as brightness information HI.

[0159] Figure 4 For this purpose, an exemplary procedure is shown, in which, based on the image B or its image information BI, Figure 3 The sub-convolutional neural network TCNN2 represented in determines the vector:

[0160]

[0161] It carries with it a corresponding confidence measure for the corresponding class n=1 ... N. The confidence measure is then provided as confidence measure information or prediction information PI.

[0162] For this purpose, the sub-convolutional neural network TCNN2 uses sub-images or sub-image information TBI1, TBI2, which are determined based on the image B by a segmentation step SC2A, a detection step SC3 and a selection step SC4.

[0163] The previously obtained segmentation information SI is then used in the step SH for determining the brightness value (said step having been Figure 5 ), so as to determine and provide brightness information HI based on the brightness value of image B or image information BI, and the brightness information can be described as the following vector.

[0164]

[0165] Then in a verification step PS the confidence measure information PI is verified based on the brightness information HI.

[0166] And then here, the value of the confidence metric

[0167]

[0168] The verification is performed according to one or more thresholds for the confidence measure in the form of thresholds T1, ..., TN, and preferably according to brightness thresholds H1, ..., HN, which apply the brightness value:

[0169]

[0170] From this, then the data of the verified confidence measure PI* is achieved. The thresholds T1, ..., TN are preset. The thresholds H1, ..., HN are preset. The thresholds T1, ..., TN can preferably be preset by the user to influence the analysis evaluation. The thresholds H1, ..., HN can preferably be preset by the user to influence the analysis evaluation. The thresholds H1, ..., HN are individual thresholds for each type of fluorescence pattern. Determination of the verified confidence measure PI* as a vector:

[0171]

[0172] Preferably, the following is performed

[0173]

[0174] The approach proposed here is advantageous because the confidence measure is first determined based on the total image B or the selected sub-image, wherein the convolutional neural network determines the confidence measure PI, but the brightness value HI is also explicitly determined from the total image B in order to verify the confidence measure or the confidence measures PI; this is preferably done without reducing the selected sub-image. This makes it possible to detect the type of fluorescence pattern more accurately. In this case, an advantage is particularly that the total image B is taken into account in the corresponding segment class with respect to the brightness value, so that not only the brightness of the cells or segment surfaces of the selected sub-image is taken into account, but also the segment surfaces outside the selected sub-image. This has the effect that brightness artifacts that may occur on individual cells or individual segment surfaces of the sub-image have a smaller influence on the determination of the brightness value and the determination of the verified confidence measure, because the brightness values ​​of the segment surfaces outside the sub-image are also used for the confidence measure of the sub-image, and the brightness artifacts of the individual segment surfaces are not weighted much to determine the verified confidence measure.

[0175] Figure 2 Preprocessing steps VS, VS1, VS2 are shown, which can be used to preprocess a sub-image TB11 of the total image and a sub-image TB12 of the segmented image. In a first preprocessing step VS1, the sub-images TB11 and TB12 are convolved into an input feature map, preferably with eight different convolution kernels, wherein no striding is performed and the dimensions of the sub-images TB11, TB12 with the exemplary dimensions of 512x512 image pixels are retained. The selection of the number of 8 convolution kernels does not necessarily have to correspond to the number of N=8 fluorescence pattern types or fluorescence pattern classes.

[0176] In a further preprocessing step VS2, a so-called batch normalization is then performed, wherein the number of feature maps is retained, here, for example, eight feature maps.

[0177] The eight feature maps generated by step VS2 are then processed in a sequence of P different layers L1, ..., LP, where the index p=1 ... P and here, for example, P=6. At the end of the L1 layer, a set of, for example, R feature maps FMA1, ..., FMAR is generated, where the index r=1 ... R. In this example, the parameter R=11 is preferred. The number R of feature maps at the end of the first layer L1 can be independent of the number N of fluorescent pattern types.

[0178] In detail, the P layers L1, . . . , LP are arranged one after another, wherein FIG. 19 shows an exemplary structure of the first layer L1.

[0179] After the last layer LP, for example, 56 feature maps are obtained with a dimension of 8x8 pixels. In the final processing step AVS, convolution is then performed with N convolution kernels corresponding to the number of N categories or N fluorescent pattern types to generate N different feature maps FM1, ..., FMN.

[0180] FIG. 19 shows Figure 2 The pre-processing step VS2 is shown here as a dashed line, since it is not a technical solution of this layer L1, but is only arranged upstream of this layer.

[0181] The two-dimensional image information from the pre-processing step VS2 is forwarded to the processing steps PS1, PS2, PS4, respectively. Each of these processing steps PS1, PS2, PS4 processes all feature maps received by step VS2.

[0182] In step PS1, convolution with no stride is performed with 8 kernels. In step PS2, convolution with stride factor 2 is performed with 8 kernels. In step PS4, convolution with stride factor 2 is performed with 11 kernels.

[0183] In step PS3, a so-called max pooling is performed on each feature map with a stride factor of 2.

[0184] In step PS5, the 8 feature maps from step PS3 are joined with the 8 feature maps from step PS2. In step PS6, the 16 feature maps are convolved with 11 kernels without stride.

[0185] In step PS7, so-called batch normalization is performed on each of the 11 feature maps.

[0186] In step PS8, a so-called activation is performed, preferably in the form of a RELU activation.

[0187] In step PS9, a so-called dropout preferably takes place during the training phase with a dropout factor of 20%. This dropout does not take place during the classification phase.

[0188] In step PS10, a convolution without stride is performed with 11 kernels.

[0189] In step PS11 , the result of the characteristic map or the 11 characteristic maps from step PS10 is element-wise added to the 11 characteristic maps from step PS4 , so that step PS11 again generates 11 characteristic maps. No striding occurs in this case.

[0190] In step PS12, a so-called batch normalization is performed.

[0191] In step PS13, a so-called activation is performed, preferably in the form of a RELU activation.

[0192] In step PS14, dropout preferably occurs during the training phase with a dropout factor of 20%.

[0193] return Figure 2 , it can be noted that preferably, P=6 such layers follow one another, and these layers are designed so that the last layer LP with P=6 then generates a feature map with a size of 8×8 pixels. The last layer LP preferably generates 56 (fifty-six) different such feature maps, and then Figure 2 In the post-processing step AVS shown in , 56 feature maps are convolved with N = 8 kernels to generate N = 8 feature maps FM1, ..., FMN.

[0194] exist Figure 5 A segmentation convolutional neural network SEG-CNN is shown in , which segments the image B or image information BI with the help of the LA1, LAQ layers. The so-called downscaling preferably occurs in the optional DWS step. The image of the image data BI preferably has a dimension of 2400×1700 pixels. The image can then be downscaled to a size of 800×800 pixels by so-called downsampling. Further processing steps (which are shown in detail later in Figure 20) are explained for the exemplary case of downscaling DWS of the total image to a size of 800×800 pixels. However, this is not absolutely necessary, and the dimensions of the convolutional neural network CNNS in Figure 20 can also be designed so that it can process a total image of 2400×1700 pixels without downscaling.

[0195] FIG. 20 shows Figure 5 Details of the convolutional neural network CNNS in FIG20 . In FIG20 , in the first step PS21 , convolution is performed with 32 kernels, which can be read based on the parameters 32 at the output layer.

[0196] From the fact that the dimension of the input variable of step PS21 is 800×800 pixels and the dimension of the output variable of step PS21 is also 800×800 pixels, it can be inferred that no striding occurs. For processing steps PS21, ..., P37, the dimension of the input variable is given as "input" in brackets according to the two numbers after "None". In addition, for processing steps PS21, ..., P37, the dimension of the output variable is also given as "output" in brackets according to the two numbers after "None". Through different steps PS21 to PS37, the image or image data BI is processed until it becomes a feature map M1, ..., MS, such as Figure 5 Already explained in .

[0197] It should be noted here that step PS29 performs a so-called deconvolution or transposed convolution.

[0198] Further steps either perform convolutions, as already explained in the description of FIG. 19 , or perform maximum pooling or union, as also already explained in the description of FIG. 19 .

[0199] If in the processing of the image data BI (which is also Figure 5 If so-called downsampling of the total image B or of the image data BI takes place before the segmentation map SM is determined, a corresponding upsampling can take place after the segmentation map SM has been determined before the segmentation map SM is supplied to a further image processing step BV.

[0200] Fig.15a Image BX is shown as a fluorescence image for a mixed pattern in which there are nucleolar patterns (AC-8, 9, 10) of class n=7 and patterns of fine or coarse spots (AC-4, AC-5) as fluorescence pattern types of class n=5.

[0201] For example, Fig.15b An enlarged interphase cell IZX and an effective mitotic cell MZX are shown. It can be clearly seen that the metaphase plate of the effective mitotic cell MZX is not significantly stained or that the mitotic cell is not stained with the chromosomes of the metaphase plate. This information allows the distinction between whether there is a so-called fine or coarse speckled pattern (AC-4, AC-5) or whether there is a uniform pattern (AC-1). Since the effective mitotic cell MZX with a metaphase plate has no staining in the chromosome region, uniform staining of the category n=2 can be excluded, because in the case of uniform staining, the mitotic cell must also be uniformly stained in the metaphase plate region, but this is not the case here. Therefore, by including mitotic cells in the fluorescence pattern type detection, it is possible to reliably detect a fine speckled fluorescence pattern type that does not involve a uniform fluorescence pattern type but involves a fine speckled fluorescence pattern type of n=5 (AC-4) or a coarse speckled fluorescence pattern type of n=5 (AC-5).

[0202] Fig.16 Another example of a fluorescence image BY is shown, in which active mitotic cells MZY and interphase cells IZY are highlighted.

[0203] Fig.17a The interphase cell IZY is shown in an enlarged manner, and Fig.17b The corresponding active mitotic cell MZY is shown together with its metaphase plate MP.

[0204] Fig.16The image BY is an image with three existing fluorescent pattern types. In this fluorescent pattern image, there is a fluorescent pattern type n=7, also called nucleoli (AC-8, 9, 10), and there is also a fluorescent pattern type n=6, also called nuclear dots (AC-6, 7), and a fluorescent pattern type n=5, also called fine spots (AC-4), as one of the variants of the fluorescent pattern type n=5 (AC-4, AC-5). For this reason, Fig.17a Staining of a nucleolar pattern by the NU region, staining of a nucleolar dot pattern by the DO region, and staining of a fine speckled pattern by another region FG of the cell nucleus are shown.

[0205] It is particularly advantageous here that, according to the method according to the invention, the mitotic cell MZY must also be present in the sub-image, since the metaphase plate MP is not stained or its chromosomes are not stained, so that here the fine spot FG region can be reliably detected as a fine spot of n=5 (AC-4) and will not be mistakenly regarded as a uniform staining, as is the case with a uniform pattern of n=2 (AC-1), since in this uniform pattern the pattern of the metaphase plate MP must be significantly stained. Here, a pattern staining of the “dense fine spot” type with n=4 (AC-2) can also be excluded, since in this type of fluorescent pattern the metaphase plate MP of the mitotic cell MZY must be significantly stained.

[0206] It has thus been shown here again that the method according to the invention allows a particularly high-quality detection of different fluorescence pattern types, in which the sub-images are selected depending on whether they contain an image segment with at least one valid mitotic cell in the metaphase phase.

[0207] As described above, the overall image is preferably segmented by a network separate from the aforementioned convolutional neural network, in particular a further, separately separate and pre-trained convolutional neural network, to determine the segmented image. Classical image processing methods, such as Otsu's threshold method, have only limited performance.

[0208] For illustration purposes, Fig.21a A fluorescence image FBX is shown, which has two valid mitotic cells VMZ in the subregion FBXT, which are in particular in a valid mitotic phase and show a metaphase plate. As described above, such mitotic cells must be reliably identified in the segmentation step. Since the metaphase plate is stained only slightly differently than other regions, such as the cytoplasm, it is difficult to reliably detect these cell VMZs using thresholding methods. Figure 21bThe segmentation result FBS achieved by the convolutional neural network according to the method proposed here is shown, which already detects valid mitotic cells or their metaphase plates well as white areas. It is therefore clear that the use of convolutional neural networks for segmentation tasks has clear advantages over classical image processing methods.

[0209] The method according to the invention has been already described here by Figure 1 Therefore, a method for detecting the corresponding potential presence of corresponding different types of cell fluorescence patterns on a biological cell substrate with human epithelioma cells by digital image processing is proposed, which starts with the step of acquiring a total image.

[0210] As an alternative to the step SC2 of acquiring the overall image, a method for digital image processing can also be performed in a corresponding and similar manner, wherein such an overall image is provided or received in the form of data in a corresponding step. Further steps SC2A, SC3, SC4, SC5 are then also performed in the method for digital image processing.

[0211] to this end, Figure 7a The proposed calculation unit R is shown, which receives the corresponding image B or image information BI in the form of a data signal SIG via an interface DS2. The calculation unit R can then output the determined confidence measure PI, the verified confidence measure PI* and / or the detection information DI, preferably via an output interface AS, in particular to a display unit AE.

[0212] This information can also be output preferably in the form of a signal SI3 via a data interface DS3 in the form of a data network interface.

[0213] to this end, Figure 7b A proposed data network device is shown, which receives images or image information BI via an interface DS4 and corresponding data signals SIG1. The data network device also preferably has an internal data bus IDB, which connects the previously described calculation unit R with preferably a memory unit MEM.

[0214] Figure 7c The proposed computer program product CPP is shown, which comprises instructions which, when executed by a computer, cause the computer to carry out the proposed method for digital image processing. For example, the computer program product can be provided or transmitted in the proposed manner via a data signal SIG2. The transmission can take place to a computer CO or a data location DSX of the computer CO.

[0215] Figure 6An exemplary embodiment of a device V1 according to the invention is shown. The device V1 has a holding device HL for a substrate SU. The excitation light AL of the excitation light source LQ is pre-filtered via an optical filter F1 and then directed to the substrate via an optical device O by means of a dichroic mirror SP1. The generated fluorescence radiation or the generated fluorescence light FL then passes from the substrate through the objective O, through the dichroic mirror SP1 and back through the optical filter F2. The optical filter F2 filters out the wavelength of the excitation radiation or the excitation light AL. The fluorescence light FL is then supplied to an image acquisition unit in the form of a camera K. The fluorescence light FL is supplied to the image acquisition unit K via an optical filter FG, which allows the fluorescence radiation to be transmitted in a preferably green channel. The image acquisition unit K acquires a fluorescence image or a total image.

[0216] The computing unit R is designed to receive the fluorescence image in the form of digital image data BI. The computing unit R is also designed to determine a segmented image by segmenting the overall image, to detect corresponding image segments in the segmented image that respectively represent mitotic cells, to select sub-images of the overall image that respectively have at least one mitotic cell and to select corresponding sub-images of the segmented image based on the detected image segments, and to detect the corresponding actual presence of the corresponding cell fluorescence pattern type based on the selected sub-images of the overall image and the selected sub-images of the segmented image by means of a convolutional neural network.

[0217] The device V1 can provide the detection result via the data interface DS1 in the form of detection information DI and / or presence information or prediction information PI, PI*.

[0218] Although some aspects have been described in conjunction with devices, it goes without saying that these aspects are also descriptions of corresponding methods, so that blocks or structural elements of devices should also be understood as corresponding method steps or features of method steps. Similarly, aspects that have been described in conjunction with method steps or as method steps are also descriptions of corresponding blocks or details or features of corresponding devices.

[0219] According to specific implementation requirements, in an exemplary embodiment of the present invention, the computing unit R or the data network device can be implemented in hardware and / or software. The computing unit R mentioned here can be implemented as at least one computing unit or can also be implemented by a plurality of connected computing units. The implementation can be implemented using a digital storage medium, such as a floppy disk, DVD, Blu-ray disc, CD, ROM, PROM, EPROM, EEPROM or flash memory, hard disk drive or other magnetic or optical storage device, which stores electronically readable control signals, which interact or can interact with the programmable hardware component so that the corresponding method is executed.

[0220] As a computing unit, a programmable hardware component can be formed by a processor, a central processing unit (CPU), a computer, a computer system, an application specific integrated circuit (ASIC), an integrated circuit (IC), a system on a chip (SOC), a programmable logic element or a field programmable gate array (FPGA) with a microprocessor.

[0221] Thus, the digital storage medium may be machine-readable or computer-readable.Some embodiments thus comprise a data carrier having electronically readable control signals, which are capable of interacting with a programmable computer system or a programmable hardware component in such a way that one of the methods described herein is performed.

[0222] Generally speaking, embodiments of the present invention or parts of embodiments can be implemented as a program, firmware, computer program or computer program product with program code or data, wherein the program code or data can be executed when the program runs on a processor or programmable hardware component to perform one of the methods or a part of a method.

[0223] For the implementation of one or more exemplary embodiments of the convolutional neural network presented here, a person skilled in the art may resort to the so-called open source deep learning library called "Keras". A person skilled in the art may find detailed information under https: / / keras.io.

[0224] Example 1

[0225] The system as defined or the method according to the invention was tested on 196 different patient samples, which were diluted in a linear dilution series starting at 1:100. This means that the patient samples were diluted and incubated in increasing order starting at 1:100. In this case, the samples were diluted in different steps of the following series: 1:100, 1:320, 1:1000, 1:3200, 1:10000, 1:32000, 1:100000, 1:320000, 1:1000000. However, the system is by no means limited to the given dilution series.

[0226] If a pattern is identified as present in any dilution or any fluorescence image for a particular sample, then the expert declares the pattern as present for a particular sample. The expert does not have to identify the pattern in all dilutions or all fluorescence images for a particular sample to declare it present.

[0227] If a pattern is identified as negative (no pattern) in all dilutions or all fluorescence images for a particular sample, then the expert declares the pattern as negative (no pattern) for the particular sample.

[0228] If a pattern is detected as present in any dilution or any fluorescence image with respect to a particular sample, then the pattern is detected as ubiquitous by the method according to the invention with respect to a particular sample. The pattern need not be detected in all dilutions or all fluorescence images with respect to a particular sample by the method according to the invention in order to be detected as ubiquitous.

[0229] If the detection is negative (no pattern) in all dilutions or all fluorescent images for a specific sample, then the pattern is detected as negative (no pattern) according to the method of the present invention for the specific sample.

[0230] For example, if there are two different patterns that actually exist, the expert will declare both patterns to be present even if the expert identifies the two patterns in different dilutions or different fluorescence images of the same sample.

[0231] For example, if there are two different actually existing patterns, both patterns are detected as being present according to the invention when the method detects the two patterns in different dilutions of the same sample or in different fluorescence images.

[0232] For example, if a uniform pattern is detected at a dilution of 1:100, and another pattern is detected at a different dilution (eg, 1:320), both patterns are output for that sample.

[0233] Fig. 22 The recognition rates in the table relate to the entire patient sample of all dilutions. Table 1 shows the following characteristic values ​​for different samples and negative sample cases:

[0234] - TA: "True Acceptances", this is the number of samples in which the expert called present for the pattern in any dilution and in which the system called present for the pattern in at least one dilution.

[0235] - FA: "False Accepts", this is the number of samples in which the expert judged the pattern as absent in all dilutions and in which the system judged the pattern as present in at least one dilution.

[0236] - TR: "True Rejections", this is the number of samples in which the expert judged as absent for the pattern in all dilutions and in which the system judged as absent for the pattern in all dilutions.

[0237] - FR: "False Rejects", this is the number of samples in which the expert called present for the pattern in any dilution and in which the system called absent for the pattern in all dilutions.

[0238] - Sensitivity: The sensitivity of the system as an absolute number.

[0239] - Specificity: Specificity as an absolute number.

[0240] - Overall: Summary consistency as absolute numbers

[0241] The results are obtained by the entire system, namely the use of the segmentation network, the classification network, the use of probability thresholds for the individual patterns and the use of brightness thresholds for the individual patterns.

[0242] The CNN used to detect the presence of patterns was trained using a total of 95,661 sub-image tuples, of which 71,649 were used for actual training and 24,012 were used for separate validation. For the CNN used for segmentation, a total of 5,509 images were used for training, of which 4,131 were in the actual training set and 1,378 were in the validation set.

[0243] The substrates used for the incubation were substrates of type 50% FA 1520-0110 and 50% FA 1522-0110, available from EUROIMMUN Medical Laboratory Diagnostics. Fluorescence images of the substrates were taken using the "EUROPattern MicroscopeLive" device.

Claims

1. A method for detecting the corresponding potential presence of corresponding different types of cell fluorescence patterns on a biological cell substrate (SU) with human epithelial tumor cells, in, The cell fluorescence pattern types include several different antinuclear antibody fluorescence pattern types, The method comprises: - incubating the cell substrate (SU) with a liquid patient sample potentially with primary antibodies and further with secondary antibodies labeled with fluorescent dyes, - acquiring a total image (B) representing the staining of the cell base (SU) by fluorescent dyes, - determining a segmented image (SB) by segmenting the overall image (B), - detecting in the segmented image (SB) a corresponding image segment, the corresponding image segment representing a corresponding metaphase plate of a mitotic cell in the metaphase phase, characterized in that - based on the detected image segments (VMZ), selecting sub-images (TB11) of the overall image (B) each having at least one mitotic cell (VMZ) and selecting corresponding sub-images (TB12) of the segmented image (SB), and - based on the selected sub-image (TB11) of the overall image (B) and the selected sub-image (TB12) of the segmented image (SB), detecting the respective actual presence of the respective cell fluorescence pattern type by means of a convolutional neural network (CNN2), The convolutional neural network (CNN2) simultaneously processes a corresponding tuple of sub-images, which tuple has at least one selected sub-image of the overall image and a corresponding selected sub-image of the segmented image.

2. The method according to claim 1, further comprising: - detecting in the overall image (B) corresponding image segments (VMZ) based on the segmented image (SB) which each represent a mitotic cell of sufficient quality, wherein the mitotic cell (VMZ) is of sufficient quality if the mitotic cell is in an effective mitotic phase, i.e., metaphase, - based on the detected image segments (VMZ) each representing at least one mitotic cell (VMZ) of sufficient quality, selecting a sub-image (TB11) of the overall image (B) and a corresponding sub-image (TB12) of the segmented image (SB).

3. The method according to claim 1, further comprising: - Based on the selected sub-images (TB11) of the overall image (B) and the selected sub-images (TB12) of the segmented image (SB), a corresponding confidence measure (PI, PI*) for the actual presence of the corresponding fluorescence pattern type is determined by means of a convolutional neural network (CNN2).

4. The method according to claim 3, wherein: The convolutional neural network (CNN2) has an output layer (LP, AVS) that generates corresponding feature maps (FM1, ..., FMN) for corresponding cell fluorescence pattern types, And, the convolutional neural network (CNN2) determines corresponding confidence measures (P1, ..., PN) based on the corresponding feature maps (FM1, ..., FMN).

5. The method according to claim 3, further comprising: - segment the total image (B) into image segments of different segment categories, - determining at least one brightness value (HI) for at least one fluorescent pattern type based on one or more image segments of at least one specific segment class, and - verifying a confidence measure (PI, PI*) of the at least one fluorescent pattern type based on a brightness value (HI) of the at least one fluorescent pattern type.

6. The method according to claim 5, wherein: The confidence measures (PI, PI*) are verified based on the brightness values ​​(HI) and according to thresholds (H1, ..., HN) that can be preset by the user.

7. The method according to claim 3, further comprising: For a corresponding sub-image tuple (TB11, TB12) having a sub-image (TB11) of the overall image (B) and a corresponding sub-image (TB12) of the segmented image (SB), - determining, by means of a convolutional neural network (CNN2), a respective sub-image confidence measure (PI1, . . . , PI3) for the presence of a respective actual sub-image of a respective cell fluorescence pattern type, - and, determining a respective confidence measure for the respective actual presence of the respective fluorescence pattern type based on the sub-image confidence measures.

8. The method according to claim 1, further comprising: - dividing the total image (B) into a set of sub-images (TBA, ..., TBD) according to a preset division scheme, - based on the detected image segments, selecting a sub-image (TBA, TBC) of the overall image (B) and selecting a corresponding sub-image of the segmented image (SB), - and, based on the selected sub-image of the total image and the selected sub-image of the segmented image, detecting the corresponding actual presence of the corresponding cell fluorescence pattern type by means of a convolutional neural network (CNN2).

9. Device (V1) for detecting the corresponding potential presence of corresponding different types of cell fluorescence patterns on a biological cell substrate with human epithelioma cells by digital image processing, the device comprising: a holding device (HL) for a biological substrate (SU) which has been incubated with a liquid patient sample potentially containing primary antibodies and furthermore with secondary antibodies labeled with fluorescent dyes, - furthermore at least one image acquisition unit (K) for acquiring an overall image (B) representing the staining of the cell substrate by means of a fluorescent dye, At least one computing unit (R) is provided, which is designed to: - determining a segmented image (SB) by segmenting the total image (B), detecting corresponding image segments (VMZ) in the segmented image (SB), the corresponding image segments representing corresponding metaphase plates of mitotic cells in the metaphase stage, and characterized in that the computing unit is also constructed to - based on the detected image segments, select sub-images (TB11) of the total image (B) each having at least one mitotic cell and select corresponding sub-images (TB12) of the segmented image (SB), and based on the selected sub-images (TB11) of the total image (B) and the selected sub-images (TB12) of the segmented image (SB), detect the corresponding actual presence of corresponding cell fluorescence pattern types by means of a convolutional neural network (CNN2), the convolutional neural network (CNN2) simultaneously processing a corresponding tuple of sub-images, which has at least one selected sub-image of the total image and a corresponding selected sub-image of the segmented image.

10. A method for detecting the corresponding potential presence of corresponding different types of cellular fluorescence patterns on a biological cell substrate having human epithelioma cells by digital image processing, the method comprising: - acquiring a total image (B) representing the staining of the cell substrate by the fluorescent dye, - determining a segmented image (SB) by segmenting the overall image (B), - detecting a corresponding image segment (VMZ) in the segmented image (SB), the corresponding image segment representing a corresponding metaphase plate of a mitotic cell in the metaphase phase, characterized in that - based on the detected image segments (VMZ), selecting sub-images (TB11) of the overall image (B) each having at least one mitotic cell and selecting sub-images of the segmented image corresponding thereto, and Based on the selected sub-images of the total image and the selected sub-images of the segmented image, the corresponding actual presence of the corresponding cell fluorescence pattern type is detected by a convolutional neural network (CNN2), which simultaneously processes a corresponding tuple of sub-images, which has at least one selected sub-image of the total image and a corresponding selected sub-image of the segmented image.

11. A method for digital image processing, comprising: - receiving a total image (B), said total image representing the staining of a biological cell substrate with a fluorescent dye, wherein said biological cell substrate comprises human epithelioma cells, - determining a segmented image (SB) by segmenting the overall image (B), - detecting a corresponding image segment (VMZ) in the segmented image (SB), the corresponding image segment representing a corresponding metaphase plate of a mitotic cell in the metaphase phase, characterized in that - based on the detected image segments (VMZ), selecting sub-images (TB11) of the overall image (B) each having at least one mitotic cell and selecting sub-images (TB12) of the segmented image (SB) corresponding thereto, and Based on the selected sub-images of the total image and the selected sub-images of the segmented image, the corresponding actual presence of the corresponding cell fluorescence pattern type is detected by a convolutional neural network (CNN2), which simultaneously processes a corresponding tuple of sub-images, which has at least one selected sub-image of the total image and a corresponding selected sub-image of the segmented image.

12. A computing unit (R), the computing unit being configured to: - receiving a total image (B) representing the staining of a biological cell substrate by a fluorescent dye, wherein The biological cell substrate comprises human epithelial tumor cells, - determining a segmented image (SB) by segmenting the overall image (B), - detecting a corresponding image segment (VMZ) in the segmented image (SB), the corresponding image segment representing a corresponding metaphase plate of a mitotic cell in the metaphase phase, characterized in that the computing unit is further designed to: - based on the detected image segments (VMZ), selecting sub-images (TB11) of the overall image (B) each having at least one mitotic cell and selecting sub-images (TB12) of the segmented image (SB) corresponding thereto, and Based on the selected sub-images (TB11) of the overall image (B) and the selected sub-images (TB12) of the segmented image (SB), the corresponding actual presence of the corresponding cell fluorescence pattern type is detected by a convolutional neural network (CNN2), which simultaneously processes a corresponding tuple of sub-images, which has at least one selected sub-image of the overall image and a corresponding selected sub-image of the segmented image.

13. Data network device (DV), including: at least one data interface (DS4) for receiving an overall image (B), the overall image representing the staining of a biological cell substrate with a fluorescent dye, wherein the biological cell substrate comprises human epithelioma cells, Characterized in that the data network device comprises a calculation unit (R) according to claim 12.

14. Computer program product (CPP) comprising instructions which, when the program is executed by a computer (CO), cause the computer to carry out the method for digital image processing according to claim 11.

Images