Method and device for detecting the presence of a fluorescent pattern on an immunofluorescence image of a biological cell substrate

Description

[0001] The invention relates to a digital image processing method and a device for detecting the presence of a fluorescence pattern on an immunofluorescence image of a biological cell substrate. The invention further relates to a computer program product, a data carrier signal, and a data network device.

[0002] Immunofluorescence images of biological cell substrates can be acquired using image acquisition devices such as microscopes. To support a diagnostic question, information regarding the presence of an expected fluorescence pattern in a fluorescence image can be advantageous.

[0003] Immunofluorescence microscopy, or indirect immunofluorescence microscopy, is an in vitro test for determining the presence of human antibodies against specific antigens in order to answer or assess a diagnostic question. Such antigens are found, for example, in specific areas of biological cell substrates. The substrate is a cell substrate that is incubated with a patient sample in the form of blood or diluted blood, or blood serum or diluted blood serum. The patient sample therefore potentially contains certain primary antibodies, which may indicate the presence of a disease in the patient. Such primary or specific antibodies can then bind to antigens in the substrate.Such bound primary antibodies can then be labeled by allowing secondary antibodies, preferably anti-human antibodies, to bind to the bound primary antibodies in a further incubation step. These secondary antibodies can then be visualized later by labeling the secondary antibodies with a fluorescent dye. Such a fluorescent dye is preferably a green fluorescent dye, in particular the fluorescent dye FITC. Such binding of a primary antibody together with a fluorescently labeled secondary antibody can then be visualized later by irradiating the substrate with excitation light of a specific wavelength, thus exciting the bound fluorescent dyes to emit fluorescent radiation.

[0004] Depending on the diagnostic question, the presence of one or more specific types of fluorescence patterns on specific substrate areas can be targeted. The task therefore arises of detecting the presence of a potential fluorescence pattern in a fluorescence image obtained by immunofluorescence microscopy using digital image processing during immunofluorescence microscopy on a cell substrate incubated in the prescribed manner.

[0005] A biological cell substrate can in particular be a cell smear from cell lines, a colony of cell lines and / or biological tissue with biological tissue cells, in particular from a tissue section.

[0006] In the indirect immunofluorescence test (IIFT), freeze-fixed tissue sections of the primate pancreas are preferably used to detect autoantibodies against pancreatic islets (islet cell antibodies = ICA). The pancreas can be divided into an exocrine and an endocrine region. Antibodies against islet cells react only with the endocrine part of the organ, the islets of Langerhans, also called "islets," and are directed against the insulin-producing beta cells. The antigens identified include the 65 kDa isoform of the enzyme glutamate decarboxylase (GAD65), the tyrosine phosphatase homologs IA-2 proteins (IA-2α and IA-2β), the cation transporter ZnT8, insulin, and the insulin precursor proinsulin.

[0007] The resulting destruction of beta cells leads to insulin deficiency and thus to type 1 diabetes mellitus. Islet cell antibodies can be detected in 70 to 80% of type 1 diabetics at the time of clinical manifestation. They are therefore an important distinguishing feature between type 1 diabetes mellitus and type 2 diabetes mellitus (adult-onset diabetes).

[0008] In a particular form of type 1 diabetes mellitus, latent autoimmune diabetes in adults (LADA), islet cell antibodies also serve as a criterion for insulin dependency. LADA is present in 3 to 12% of patients with phenotypic type 2 diabetes mellitus, which is characterized by insulin resistance and impaired insulin secretion by β-cells.

[0009] In 90% of patients, islet cell antibodies are detectable years before the onset of diabetes. They are considered markers of the so-called prediabetic phase. This allows for early identification of individuals at increased risk of developing the disease. Appropriate interventions may prevent the onset of the disease.

[0010] The titer for islet cell antibodies decreases again as the disease progresses. A very high concentration of autoantibodies against islet cells (GAD65) can be an indicator of stiff-person syndrome, cerebellar degeneration, and limbic encephalitis.

[0011] In the indirect immunofluorescence test, the islets of Langerhans in the pancreatic tissue show a finely spotted, smooth to granular fluorescence in a positive result. In the indirect immunofluorescence test (IIFT), a substrate can also preferably be used in which several Crithidia luciliaeCells are fixed to the substrate. Such a substrate is, in particular, a cell smear. Such fixation can be carried out, for example, using ethanol. The substrate can then be used to detect the binding of autoantibodies from a patient sample to double-stranded deoxyribonucleic acid (DNA). The detection of autoantibodies against deoxyribonucleic acids (DNA) is, for example, important for the diagnosis of SLE (synonym: Disseminated lupus erythematosus) is crucial. A fundamental distinction must be made between two types: antibodies against dsDNA and antibodies against single-stranded, denatured DNA (ssDNA). Antibodies against dsDNA react with epitopes located in the deoxyribose phosphate backbone of DNA. In contrast, antibodies against ssDNA primarily bind to epitopes from the purine or pyrimidine base region. However, they can also recognize epitopes of the deoxyribose phosphate backbone. Anti-dsDNA antibodies are found almost exclusively in SLE. Their prevalence is 20% to 90%, depending on the detection method and disease activity. Anti-dsDNA antibodies are also occasionally detected in patients with other autoimmune diseases and infections, and in rare cases in clinically healthy individuals. In the latter case, 85% of cases develop SLE within 5 years of the initial anti-dsDNA detection.However, SLE cannot be completely ruled out if antibodies against dsDNA are not present. SLE is a systemic autoimmune disease belonging to the group of collagen vascular diseases. Diagnosis is based on the 11 criteria of the American College of Rheumatology (ACR), modified in 1997. If four of the 11 criteria are present, a diagnosis of SLE can be made with 80% to 90% certainty. Indirect immunofluorescence can therefore be used as an in vitro test for the determination of human antibodies against dsDNA. For example, so-called BIOCHIPs can serve as substrates, which are labeled with... Crithidia luciliae Smears are coated. These are incubated, for example, with diluted patient samples. In the case of positive reactions, specific antibodies bind to the antigens. Bound antibodies (IgG) are stained in a second incubation step, for example, with fluorescein-labeled anti-human antibodies, and visualized under a fluorescence microscope.

[0012] The document EP 3971827 A1 discloses a method for detecting binding of antibodies of a patient sample to a biological substrate comprising Crithidia luciliae Cells and fluorescence microscopy, whereby only initial partial confidence measures are determined for partial images.

[0013] The document EP 4030340 A1 discloses a method for detecting fluorescence patterns in partial areas of an immunofluorescence image of a cell substrate by means of a neural network that processes partial images.

[0014] The document CN 108364006 A refers to an evaluation of medical images of diseased tissue instead of fluorescence images.

[0015] The object of the present invention is to provide an automated image processing method for analyzing an immunofluorescence image of a biological cell substrate, which can detect a presence of a fluorescence pattern on the cell substrate.

[0016] The object of the invention is achieved by the inventive method according to patent claim 1.

[0017] According to the invention, the method comprises various steps.

[0018] The Figure 1A shows an example fluorescence image FB1, in this example of a pancreatic tissue section. In the example fluorescence image FB1, an expected fluorescence pattern is not present. Figure 1B shows another fluorescence image FB2, also in this case of a pancreas, where an expected fluorescence pattern is present.

[0019] For the example of a substrate in the form of a pancreatic section, a specific or relevant sub-area must be considered, which are the so-called islets.

[0020] The Figure 2AFor the fluorescence image FB1, the TBA region is shown as a partial image region that essentially contains an islet. No expected fluorescence pattern is present in the TBA region. Figure 2B shows the partial image areas TBB1, TBB2, TBB3 for the fluorescence image FB2 with the corresponding islets or islands on which an expected fluorescence pattern is present. These partial image areas TBB1, ..., TBB3 are shown again in the Figure 2C shown separately and enlarged.

[0021] The Figure 3A shows the example of a cell substrate in the form of a cell smear of Crithidia luciliae Cells a fluorescence image FB11, in which essentially no expected fluorescence pattern is present. Figure 3B shows an example of a fluorescence image with a cell substrate in the form of a cell smear with Crithidia luciliaeCells in which an expected fluorescence pattern is present as image FB12. Relevant partial areas or partial image areas are indicated by rectangles in the fluorescence image FB12. The relevant partial areas in particular show a kinetoplast of a Crithidia of Lucilia cell.

[0022] Figure 3C shows exemplary partial image areas TBB11, TBB12, TBB13 from the fluorescence image FB12, which are shown in the relevant areas in the form of Crithidia luciliae Cells exhibit an expected fluorescence pattern. This is particularly the glow of the kinetoplast.

[0023] According to the invention, after the fluorescence image has been provided, respective localization information is determined, which indicates the respective locations of respective relevant subregions of the cell substrate in the fluorescence image. For the example of the pancreas, the relevant subregions are the so-called islets, since an expected fluorescence pattern must be detected on these, or it must be determined whether the expected fluorescence pattern is not present. Such localization information can, for example, be in the form of a rectangle, as in the Figure 2B to be seen, for each relevant sub-area.

[0024] Furthermore, according to the invention, respective first partial confidence measures of the respective presence of the fluorescence pattern on the respective partial areas are determined based on the entire fluorescence image. Such partial areas correspond, for example, to the partial image areas TBB1, TBB2, TBB3 from the Figure 2B .

[0025] The determination of the localization information and the determination of the respective first partial confidence measures are carried out by a first neural network based on the entire fluorescence image. In particular, this determination of the localization information and the first partial confidence measures is carried out simultaneously by the first neural network, so that the first neural network is, in particular, a combined neural network for both tasks of this determination.

[0026] According to the invention, the respective partial image regions corresponding to the respective partial regions of the cell substrate are extracted from the fluorescence image. This is done on the basis of the previously obtained localization information. Such partial image regions are Figure 2C shown as partial image areas TBB1, TBB2, TBB3 as examples.

[0027] Furthermore, according to the invention, on the basis of the respective partial image areas, respective second partial confidence measures are determined with regard to the respective presence of the fluorescence pattern on the respective partial areas by means of a second neural network. Thus, for example, the partial image areas TBB1, TBB2, TBB3 are selected from the Figure 2Canalyzed by means of a second neural network. In particular, the determination of the respective second partial confidence measures by means of the second neural network is carried out in such a way that each of the respective partial image regions is analyzed or processed separately by the second neural network in order to determine a respective second partial confidence measure.

[0028] Finally, according to the invention, a confidence measure of the presence of the fluorescence pattern in the fluorescence image is determined on the basis of the first partial confidence measures and the second partial confidence measures.

[0029] Possible advantages of the invention will now be explained in more detail.

[0030] The fundamental task of detecting the presence of a fluorescence pattern in a large, entire fluorescence image could, in principle, be accomplished by a single neural network. Such large images, with corresponding ground truth information regarding the presence of the fluorescence pattern, are analyzed by such a single neural network in a training phase, in order to then determine such class membership in the "positive" or "negative" classes. With such an approach, this single neural network would have to incorporate and consider a large amount of image information from an entire fluorescence image. This would require a high level of network complexity and thus a large number of computational operations, so the corresponding computational effort can be challenging.

[0031] Furthermore, it must be noted that neural networks for the analysis of large overall images can extract image features and possibly classify them as relevant which are present in the overall image but may be irrelevant for the actual classification task of the neural network. In order to specifically direct the attention and the image features to be learned by the neural network to only those image structures that are actually relevant, the method proposed here is suitable. In a first step, a first neural network directly identifys and classifies the relevant structures of the entire fluorescence image with regard to their location and then, in a second step, additionally classifies the corresponding partial image areas of the relevant structures individually using a second neural network. In this case, the determination of the classification results orThe combination of the first partial confidence measures from the first stage with the determination of the classification results or the second partial confidence measures from the second stage and the merging of these results into the final confidence measure of the presence of the fluorescence pattern in the entire fluorescence image ensures a high degree of validity of the final confidence measure. A further advantage arises from the fact that the partial confidence measures of the partial areas or the partial image areas allow the classification results from the two stages or the two neural networks to be incorporated into the final confidence measure in a controlled manner. This results in good model interpretability with regard to the function of the two neural networks in the process.

[0032] By determining the relevant sub-areas and their corresponding sub-image areas as individual elements, the number of partial results available in the method, e.g. in the form of partial confidence measures, is automatically determined. In an overall image approach, such information or number information is not available, since there is then only one overall output of a single confidence measure related to the overall image, preferably as a statement "positive" or "negative".

[0033] In other words, the method according to the invention explicitly deviates from an approach for analyzing an entire fluorescence image using only one neural network. The first neural network first determines localization information that indicates relevant subregions of the cell substrate. The second neural network then has to process each such subregion separately in order to detect a possible presence of the fluorescence pattern in the respective subregion and determine a corresponding partial confidence measure.

[0034] The second neural network can therefore be significantly reduced in complexity compared to a solution in which a single neural network has to determine the confidence measure of the presence of the fluorescence pattern for the entire fluorescence image.

[0035] The second neural network can thus be trained on significantly smaller partial image areas and only needs to analyze those types of partial images that actually represent relevant sub-areas of a cell substrate.

[0036] Therefore, the second neural network can determine significantly more valid partial confidence measures regarding the presence of the fluorescence pattern in the respective partial images or partial areas. Therefore, the second neural network can also determine a more valid confidence measure regarding the presence of the fluorescence pattern in the fluorescence image.

[0037] The method according to the invention is further advantageous because the first neural network determines both the relevant subregions using the localization information and also determines the first partial confidence measures, in particular simultaneously. The first neural network is therefore in particular one that has been trained to simultaneously determine the first partial confidence measures and the localization information. Such networks can be particularly efficient compared to other networks in terms of computational complexity and required memory. Such networks can be implemented, in particular, as so-called "one-shot detectors."

[0038] Furthermore, the method according to the invention is advantageous because the confidence measure is determined taking into account or on the basis of the first partial confidence measures and the second partial confidence measures, so that such information regarding the partial confidence measures can be incorporated from both neural networks in order to determine a particularly reliable confidence measure of the presence of the fluorescence pattern in the fluorescence image.

[0039] Preferably, the confidence measure is output.

[0040] Preferably, weighted partial confidence measures are determined by weighting the first partial confidence measures and the second partial confidence measures, and the confidence measure is determined on the basis of the weighted partial confidence measures.

[0041] Preferably, the confidence measure is determined by applying a threshold value to the weighted partial confidence measures.

[0042] Preferably, the method further comprises the steps of: determining respective presence confidence measures, which indicate the respective locations of the respective relevant sub-regions, to what extent relevant sub-regions of the cell substrate are actually present at the respective locations, by means of the first neural network on the basis of the entire fluorescence image, determining whether the localization information indicates a set of several overlapping sub-image regions, retaining the localization information of a specific sub-image region from the set of overlapping sub-image regions on the basis of the presence confidence measures of the overlapping sub-image regions and discarding the localization information of the other sub-image regions from the set of overlapping sub-image regions, and extracting respective sub-image regions from the fluorescence image on the basis of the remaining localization information.

[0043] Furthermore, the method comprises in particular the steps of: determining a set of partial image regions which have a presence of the fluorescence pattern, and determining a brightness value of the presence of the fluorescence pattern on the immunofluorescence image on the basis of the set of the set of partial image regions which have a presence of the fluorescence pattern.

[0044] Also proposed is a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method for digital image processing.

[0045] A data carrier signal which transmits the computer program product is also proposed.

[0046] Furthermore, a device for detecting a fluorescence pattern on an immunofluorescence image of a biological cell substrate is proposed, comprising a holding device for a slide with the cell substrate, which has been incubated with a patient sample containing the autoantibodies, as well as with secondary antibodies, each of which is labeled with a fluorescent substance. The device further comprises at least one image acquisition unit for acquiring a fluorescence image of the cell substrate. Furthermore, the device comprises at least one computing unit configured to perform the steps of: determining respective localization information indicating respective locations of respective relevant subregions of the cell substrate in the fluorescence image,and determining respective first partial confidence measures of respective presences of the fluorescence pattern on the respective partial regions by means of a first neural network based on the entire fluorescence image, extracting respective partial image regions corresponding to the respective partial regions of the cell substrate based on the localization information, determining respective second partial confidence measures of respective presences of the fluorescence pattern on the respective partial regions by means of a second neural network based on the respective partial image regions, and determining a confidence measure of the presence of the fluorescence pattern in the fluorescence image based on the first partial confidence measures and the second partial confidence measures.

[0047] Furthermore, a computing unit is disclosed which is designed to carry out the following steps during digital image processing: receiving an immunofluorescence image which represents a staining of a biological cell substrate by a fluorescent dye, determining respective localization information which indicates respective locations of respective relevant partial regions of the cell substrate in the fluorescence image, and determining respective first partial confidence measures of respective presences of the fluorescence pattern on the respective partial regions by means of a first neural network based on the entire fluorescence image, extracting respective partial image regions which correspond to the respective partial regions of the cell substrate, based on the localization information,Determining respective second partial confidence measures of respective presences of the fluorescence pattern on the respective partial areas by means of a second neural network based on the respective partial image areas, and determining a confidence measure of the presence of the fluorescence pattern in the fluorescence image based on the first partial confidence measures and the second partial confidence measures.

[0048] Furthermore, a data network device is proposed, comprising at least one data interface for receiving a fluorescence image representing a staining of a cell substrate by a fluorescent dye. The data network device further comprises at least one computing unit configured to perform the following steps during digital image processing: determining respective localization information indicating respective locations of respective relevant subregions of the cell substrate in the fluorescence image, and determining respective first partial confidence measures of respective presences of the fluorescence pattern on the respective subregions by means of a first neural network based on the entire fluorescence image; extracting respective partial image regions corresponding to the respective subregions of the cell substrate based on the localization information;Determining respective second partial confidence measures of respective presences of the fluorescence pattern on the respective partial areas by means of a second neural network based on the respective partial image areas, and determining a confidence measure of the presence of the fluorescence pattern in the fluorescence image based on the first partial confidence measures and the second partial confidence measures.

[0049] The invention will be explained in more detail below using specific embodiments without limiting the general inventive concept and the figures. In the figures: Figures 1 and 2 show immunofluorescence images and partial images of a first biological cell substrate. Figure 3 shows immunofluorescence images and partial images of a second biological cell substrate. Figure 4 shows steps of a preferred embodiment of the method according to the invention. Figure 5 shows preferred steps for determining the confidence measure when weighting partial confidence measures. Figure 6 shows steps of the method according to the invention with respect to a further preferred embodiment. Figure 7 shows steps for determining a brightness measure of the presence of the fluorescence pattern. Figure 8 shows a preferred embodiment of a second neural network. Figure 9 shows a preferred embodiment of a device according to the invention. Figure 10 shows a preferred embodiment of a disclosed computing unit. Figure 11 shows a preferred embodiment of a data network device according to the invention. Figure 12 shows an illustration of a computer program product and a data carrier signal.Figure 13 a table of experimental results.

[0050] The Figure 4 shows steps of a preferred embodiment of the method according to the invention. In step S1, the cell substrate is incubated with a liquid patient sample potentially containing primary antibodies, as well as with secondary antibodies labeled with a fluorescent dye. Furthermore, in step S1, the cell substrate is irradiated with excitation radiation and the immunofluorescence image is acquired.

[0051] The immunofluorescence image can be provided as a data element FB by step S1.

[0052] Such an immunofluorescence image is shown as an example image FB1 in the Figure 1A In this case, the relevant cell areas, here the islets or islands, of the cell substrate, here in the form of a pancreatic section, are not present. In the immunofluorescence image FB2 of the Figure 1B Such immunofluorescence is present in the relevant subareas.

[0053] The Figure 2A shows the immunofluorescence image FB1 again with an indicated partial area or a partial image area TBA, within which a relevant partial area of the cell substrate lies. In comparison to partial areas or partial image areas TBB1, TBB2, TBB3 of the fluorescence image FB2 from the Figure 2B It is evident that in the immunofluorescence image FB1, the presence of an expected fluorescence pattern is not present, but this presence of the fluorescence pattern is present in the partial image areas of the immunofluorescence image FB2. The fluorescence patterns of the respective partial image areas TBB1, TBB2, TBB3 are shown again in an enlarged view in Figure 2C to see.

[0054] For the purposes of this application, a partial image may also be referred to as a partial image area.

[0055] According to the Figure 4In step S2, respective localization information is determined, which indicates the respective locations of respective relevant subregions of the cell substrate in the fluorescence image. This is done using a neural network NN1, which can be referred to as a first neural network.

[0056] Furthermore, in step S2, respective first partial confidence measures of the respective presences of the fluorescence pattern in the respective partial regions or respective relevant partial regions are determined, in particular simultaneously, using the neural network NN1 based on the entire fluorescence image. This is thus carried out, in particular, in a combined or simultaneous processing.

[0057] The determined localization information can be provided as a combined data element LI. The first partial confidence measures can be provided as an ETKM data element.

[0058] Localization information preferably indicates a position of a partial region in a fluorescence image, as well as width information and height information of the partial region. The localization information can therefore preferably indicate a rectangle at a specific position in the fluorescence image, as well as the height and width of the rectangle.

[0059] Such localization information for a partial area or a partial image area can be given as a vector n, n → = n 1 n 2 n 3 n 4 where the entry n1 can indicate an X position and the entry n2 a Y position of a rectangle in the fluorescence image, and furthermore the entry n3 a width of the rectangle and the entry n4 a height of the rectangle.

[0060] A first partial confidence measure ETKM for a sub-area, for example a sub-area TBB1 from the Figure 2C , can be represented, for example, by a vector o o → = o 1 o 2 be given, where the entry o1 indicates a probability that the partial image area belongs to a class of positive fluorescence of the pattern and where the entry o2 indicates a probability that the partial area belongs to a class of negative fluorescence of the pattern or an absence of the fluorescence pattern.

[0061] The localization information can therefore be a respective vector n for each partial image area and the first partial confidence measures can therefore be a respective vector o for each partial image area.

[0062] The first neural network is preferably a so-called object detection network. Particularly preferred is an object detection network of the so-called one-shot detector type. In particular, the first neural network is a so-called YOLO network.

[0063] Such a network can be trained, for example, by presenting entire fluorescence images to an input layer of the neural network, wherein localization information of respective sub-areas of the cell substrate is also provided as ground truth information, as well as ground truth information regarding a class affiliation of a corresponding partial image area of the sub-area for the classes "positive fluorescence" or "negative fluorescence".

[0064] Preferably, the first neural network NN1 further determines, on the basis of the entire fluorescence image, a further information OB, also shown in Figure 6This information OB represents a presence confidence measure for each location or piece of localization information. This measure indicates, for each location of a respective relevant sub-area, the degree to which a relevant sub-area of the cell substrate is actually present at that location. This information OB can be output for each partial image area using a respective scalar value c.

[0065] The first neural network therefore preferably outputs a tuple for each detected relevant sub-image area with index k=1... K {c, o , n} The training of the neural network can therefore be carried out by providing complete fluorescence images as well as ground truth information in the form of corresponding tuples {c, o, n} for each sub-area.

[0066] Detailed structures of such a first neural network for the particular embodiment of a Yolo network can be found in J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788, doi: 10.1109 / CVPR.2016.91.

[0067] The confidence measure c is referred to there as "confidence." The vector n is given by the values of image position x, y, as well as width b and height h. The class probabilities o1 and o2 are given there as so-called conditional class probabilities "C."

[0068] In a step S3, the respective partial image regions corresponding to the respective partial regions of the cell substrate are extracted from the fluorescence image based on the localization information. These are, for example, the partial image regions from the Figure 2CThese partial image regions can be provided as a combined data element TB. Preferably, an extracted partial image region is identical to the corresponding partial region, which is indicated by corresponding localization information. In particular, a partial image region can exhibit a certain deviation from the partial region, particularly in the edge region.

[0069] In a step S4, the respective second partial confidence measures of respective presences of the fluorescence pattern on the respective partial areas are further determined by means of a second neural network NN2 on the basis of the respective corresponding partial image areas.

[0070] Step S4 can then provide these second partial confidence measures as a data element ZTKM.

[0071] The second neural network preferably outputs a vector d with a class probability for each sub-image area for the respective classes "fluorescence positive" or "fluorescence negative". In particular, the vector d can be a vector d → = d 1 d 2 , where entry d1 can be a class probability "fluorescence positive" and entry d2 a class probability "fluorescence negative." The second partial confidence measure can therefore be such a vector d for a partial image area.

[0072] In step S5, the confidence measure KM of the presence of the fluorescence pattern is then determined based on the second partial confidence measures ZTKM and the first partial confidence measures ETKM, which are input into step S5. Preferably, the confidence measure KM is output in a step S6. Such output can preferably be in the form of an output of a data element via a data interface or, more preferably, by means of an optical output on a display of a computing unit, such as a computer monitor.

[0073] Preferably, the confidence measure KM can be determined by averaging the respective first partial confidence measure and the respective second partial confidence measure for a respective partial image or a respective partial image region with index k=1...K, resulting in respective averaged partial confidence measures for respective partial image regions. These averaged partial confidence measures can then be averaged again across all partial image regions or all indices k=1...K to determine the confidence measure of the presence of the fluorescence pattern.

[0074] According to the Figure 5As an alternative to step S5, a modified step S51 with sub-steps S511 and S512 can be performed. In step S511, the first partial confidence measures ETKM and the second partial confidence measures ZTKM are input. In the first step, weighted partial confidence measures are determined by weighting the first partial confidence measures ETKM and the second partial confidence measures ZTKM. The weighted partial confidence measures can be provided as a data element GTKM. The confidence measure KM is then determined in step S512 based on the weighted partial confidence measures GTKM.

[0075] According to a preferred embodiment, the first partial confidence measure and the second partial confidence measure are weighted for each partial image region. This can therefore be referred to as determining respective weighted partial confidence measures by weighting the respective first partial confidence measures and the respective second partial confidence measures.

[0076] In a special embodiment, a weighting with weighting factors w1 and w2 is carried out for each partial image area according to r → = w 1 ∗ o → + w 2 ∗ d → = r 1 r 2 to determine a vector r k → each sub-image area k=1... K.

[0077] These weighted partial confidence measures in the form of vectors r k Each partial image area with index k=1...K can then be used to determine the confidence measure KM. In particular, the values r1 for each partial image area, which represent a class probability for the class "fluorescence positive," can be considered. These values R1 can then each be subjected to the application of a threshold value. Thus, the confidence measure is determined by applying a threshold value to the weighted partial confidence measures.

[0078] Preferably, a partial image area is assigned to the class "fluorescence positive" if the corresponding value r1 is greater than the threshold. For example, the value R1 can be in the range [0...1] and a threshold of 0.6 can be applied.

[0079] Partial image areas for which the value r1 does not exceed the threshold are then classified as belonging to the class "fluorescence negative".

[0080] A so-called majority voting can then preferably be used. In this majority voting, the confidence measure can then be determined, in particular, as a "yes" / "no" statement, with the majority of partial image regions being decisive. Thus, if there are more partial image regions for which the fluorescence pattern was detected as present or for which the class "fluorescence positive" was recognized, the confidence measure is "yes." The reverse applies to the possibility of a "no" confidence measure.

[0081] The advantage of applying a threshold value to the weighted partial confidence measures is that the method allows for setting the confidence measure value at which the entire fluorescence image is considered positive or the expected fluorescence pattern is considered present. This can be adjusted, in particular, depending on the individual weighted partial confidence measures of the partial image regions.

[0082] Due to a resulting grid structure in connection with a so-called striding of a neural network such as the neural network NN1, it is possible for such a first neural network NN1 to determine several localization information items that indicate respective sub-areas that overlap. A spatial separation of sub-areas or partial image areas, as in the partial image areas TBB1, TBB2, TBB3 from Figure 2C, does not necessarily have to be given. It can also happen that partial image regions are determined which at least partially overlap in their area. In particular, an overlap of several partial regions can result which indicate an identical relevant partial region, e.g. an identical islet, on the substrate. Preferably, two partial regions are regarded as overlapping if their degree of areal overlap exceeds a predetermined value. For example, a degree of overlap can be determined in a value range of [0...1] and then an overlap is only determined if the degree of overlap exceeds a threshold value, e.g. the value 0.2.

[0083] Such overlapping partial image regions can, in principle, indicate the same or similar relevant partial image region. Therefore, it is advantageous to select only one of such overlapping partial image regions, since a specific relevant partial region of a cell substrate should only be included once in the subsequent evaluation to determine the confidence measure, and not multiple times.

[0084] Figure 6 shows a preferred embodiment of the method according to the invention.

[0085] The order of steps S1 and S2 is the same as in the embodiment of the Figure 4In step S2, the localization information LI, the first partial confidence measures ETKM, and here also the presence confidence measures OB are determined, in particular for each relevant partial area. Step S2 is then followed by a step S21, in which it is determined whether the localization information, given by the data element LI, indicates a set of multiple overlapping partial areas. In particular, this set is determined in step S21. This set of overlapping partial image areas and their localization information can be provided as a set of localization information MLI as a data element.

[0086] From the set of overlapping partial image areas, the partial image area or localization information which has the highest value of the presence confidence measure OB is retained.

[0087] In a step S22, localization information for a specific partial image region is then determined from this set of overlapping partial regions and retained. The localization information for the other partial image regions from this set of overlapping partial image regions is discarded. Localization information that does not indicate any overlapping partial image regions is also retained. The retained localization information then forms the basis for further processing. The retained localization information is represented as a data element (VLI). Based on the retained localization information (VLI), it can be ensured that, in the case of overlapping partial image regions that indicate the same relevant area of the substrate, only a specific partial region is included in the final evaluation or processing.This ensures that a relevant cell substrate area is included only once as a partial image area in the subsequent evaluation and does not distort the result.

[0088] In step S3, the respective partial image areas are then extracted from the fluorescence image on the basis of the remaining or retained localization information VLI.

[0089] The Figure 7 shows preferred steps for determining a brightness measure of the presence of the fluorescent pattern. In a step S100, a set of partial image regions is determined that exhibit a presence of the fluorescent pattern. For example, based on the weighted partial image confidence measures GTKM, a set of partial image regions can then be determined whose partial image confidence measure GTKM or whose value R1 exceeds a threshold value.

[0090] This set of sub-image areas or remaining sub-image areas can be represented as a data element VTB.

[0091] In a step S101, the brightness measure HM of the presence of the fluorescence pattern on the immunofluorescence image is then determined on the basis of the set of partial image areas VTB which have a presence of the fluorescence pattern.

[0092] In particular, only partial image areas are considered as the partial image areas VTB, which show a class decision "positive fluorescence".

[0093] For these partial image areas, an analysis is carried out using statistics of the pixel values in these partial image areas.

[0094] For each partial image area, a histogram of the pixel brightness values is created, and then the brightness value is determined at which a 0.8 quantile, or at which 80% of the pixels have an intensity value greater than the 0.8 quantile, is found. This 0.8 quantile value is then used as the partial brightness value of a partial image area. For all partial image areas under consideration, the respective partial image brightness values are then preferably averaged. Thus, these brightness quantile values of the respective partial image areas are preferably averaged. Alternatively, a median value can be determined.

[0095] This brightness quantile value, determined using the median or mean, can then be output as the brightness measure HM.

[0096] Particularly preferably, the resulting brightness quantile value is further quantized into four quantization ranges. For example, the brightness quantile value can be in a value range from [0...255], so that quantization is carried out into four quantization ranges with step values from [1...5].

[0097] Such a level value can then be output to the user.

[0098] The Figure 8 shows a preferred structure of the second neural network NN2. The neural network NN2 analyzes a respective partial image TB and determines a respective second partial confidence measure ZTKM for each respective partial image TB.

[0099] For this purpose, the neural network NN2 can access a sequence of respective convolutional modules (CM). Each convolutional module (CM) preferably comprises a sequence of a convolutional step (CS), a batch normalization step (BS), another convolutional step (CS), and another batch normalization step (BS), followed by a max pooling step (MAS).

[0100] Several such convolutional modules CM can follow one another.

[0101] The final convolutional module (CM) can then preferably be further processed using multiple dense layer (DL), also called fully connected layers. The second partial confidence measure (ZTKM) can then be determined, preferably using a softmax layer (SML).

[0102] Figure 9shows a device V1 by means of which the method according to the invention can preferably be carried out. The device V1 can be referred to as a fluorescence microscope. The device V1 has a holder H for a substrate S or a slide with such a substrate S, which has been incubated in the manner described above. Excitation light AL from an excitation light source LQ is guided to the substrate S via an optics O. The resulting fluorescence radiation FL is then retransmitted through the optics O and passes through the dichroic mirror SP1 and an optional optical filter F2. The fluorescence radiation FL preferably passes through an optical filter FG, which filters out a green channel. A camera K1 is preferably a monochrome camera, which then records the fluorescence radiation FL in a green channel if an optical filter FG is present.According to an alternative embodiment, the camera K1 is a color camera that does not require the optical filter FG and captures the fluorescence image in the corresponding color channel as a green channel using a Bayer matrix. The camera K1 provides the image information BI or the fluorescence image to a computing unit R, which processes this image information BI. Preferably, the computing unit R can output or provide data ED, such as a fluorescence image and / or confidence measures, via a data interface DS1.

[0103] Figure 10shows a disclosed computing unit which, preferably according to a preferred embodiment, receives a fluorescence image FB as a data signal SI via a data interface DS2. The computing unit R can then determine the previously described information and provide it as a data signal SI3 via a data interface DS3. This can preferably be done via a wired or wireless data network. Particularly preferably, the computing unit R has an output interface AS for outputting the information via an output unit AE. The output unit AE is preferably a display unit for visually displaying the aforementioned information.

[0104] Figure 11shows a data network device DV according to the invention according to a preferred embodiment. The data network device DV receives the fluorescence image FB as a data signal SI1 via a data interface DS4. The data network device DV has a previously described computing unit R and a memory unit MEM. The computing unit R, a memory unit MEM, and the data interface DS4 are preferably connected to one another via an internal data bus IDB.

[0105] The Figure 12 shows an embodiment of a proposed computer program product CPP. The computer program product CPP can have its data signal SI2 received by a computer CO via a data interface DSX.

[0106] Figure 13 shows a table of results for the biological cell substrate in the form of a cell smear from Crithidia luciliae.

[0107] 4923 fluorescence images which, according to the ground truth information, show no fluorescence or are to be classified as "negative" were also predicted as negative by the method according to the invention.

[0108] 23 negative fluorescence images were classified as positive by the method of the invention.

[0109] 2532 fluorescence images with the fluorescence class "positive" were also classified as "positive" by the method according to the invention.

[0110] 195 images of the fluorescence class "positive" were classified as fluorescence class "negative" by the method according to the invention.

[0111] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray disc, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, a hard disk, or other magnetic or optical storage device storing electronically readable control signals that can interact or interact with a programmable hardware component to perform the respective method.

[0112] A programmable hardware component, such as in particular a computing unit, can be formed by a processor, a computer processor (CPU = Central Processing Unit), a graphics processor (GPU = Graphics Processing Unit), a computer, a computer system, an application-specific integrated circuit (ASIC = Application-Specific Integrated Circuit), an integrated circuit (IC = Integrated Circuit), a single-chip system (SOC = System on Chip), a programmable logic element or a field-programmable gate array with a microprocessor (FPGA = Field Programmable Gate Array).

[0113] The digital storage medium can therefore be machine- or computer-readable. Some embodiments thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system or a programmable hardware component such that one of the methods described herein is performed. One embodiment is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the program for performing one of the methods described herein is recorded.

[0114] In general, embodiments of the present invention can be implemented as a program, firmware, computer program, or computer program product with program code or data, wherein the program code or data is effective to perform one of the methods when the program runs on a processor or a programmable hardware component. The program code or data can also be stored, for example, on a machine-readable medium or data carrier. The program code or data can be present, among other things, as source code, machine code, or bytecode, as well as other intermediate code.

[0115] A further embodiment is a data stream, a signal sequence, or a sequence of signals that represents the program for performing one of the methods described herein. The data stream, the signal sequence, or the sequence of signals can be configured, for example, to be transferred via a data communication connection, for example, via the Internet or another network. Thus, embodiments also include signal sequences representing data that are suitable for transmission via a network or data communication connection, wherein the data represents the program.

[0116] A program according to one embodiment can implement one of the methods during its execution, for example, by reading memory locations or writing one or more pieces of data into them, which may trigger switching operations or other processes in transistor structures, amplifier structures, or other electrical, optical, magnetic, or other components operating according to a different functional principle. Accordingly, by reading a memory location, data, values, sensor values, or other information can be acquired, determined, or measured by a program. A program can therefore acquire, determine, or measure quantities, values, measured quantities, and other information by reading one or more memory locations.

Claims

1. Method for digital image processing, the method comprising - providing an immunofluorescence image (FB), which represents a staining of a biological cell substrate (S) by a fluorescence stain, - determining respective items of location information (LI), which indicate respective locations of respective relevant subsections of the cell substrate in the fluorescence image, - extracting respective image subsections (TB), which correspond to the respective subsections of the cell substrate, from the fluorescence image on the basis of the items of location information (LI), - determining respective second partial confidence measures (ZTKM) of respective presences of the fluorescence pattern on the respective subsections using a second neural network (NN2) on the basis of the respective image subsections (TB), characterized in that - determining respective first partial confidence measures (ETKM) of respective presences of the fluorescence pattern on the respective subsections using a first neural network (NN1) on the basis of the overall fluorescence image - determining a confidence measure (KM) of the presence of the fluorescence pattern in the fluorescence image (FB) on the basis of the first partial confidence measures (ETKM) and the second partial confidence measures (ZTKM).

2. Method according to Claim 1, furthermore comprising outputting the confidence measure (KM).

3. Method according to Claim 1, furthermore comprising - determining weighted partial confidence measures (GTKM) via weighting of the first partial confidence measures (ETKM) and the second partial confidence measures (ZTKM), - determining the confidence measure on the basis of the weighted partial confidence measures.

4. Method according to Claim 3, furthermore comprising determining the confidence measure (KM) via application of a threshold value to the weighted partial confidence measures (GTKM).

5. Method according to Claim 1, furthermore comprising - determining respective presence confidence measures (OB), which indicate for the respective locations of the respective relevant subsections to which degrees at the respective locations actually relevant subsections of the cell substrate are present, using the first neural network on the basis of the overall fluorescence image, - determining whether the items of location information (LI) indicate a set (MLI) of multiple overlapping image subsections, - retaining the item of location information of a specific image subsection from the set of overlapping image subsections on the basis of the presence confidence measures (OB) of the overlapping image subsections and discarding the items of location information of the other image subsections from the set of overlapping image subsections, - extracting respective image subsections from the fluorescence image (FB) on the basis of the remaining items of location information (VLI).

6. Method according to Claim 1, furthermore comprising - determining a set of image subsections which comprise a presence of the fluorescence pattern, - ascertaining a brightness value (HM) of the presence of the fluorescence pattern on the immunofluorescence image on the basis of the set of image subsections which comprise a presence of the fluorescence pattern.

7. Computer program product (CPP), comprising commands which, upon the execution of the program by a computer, prompt it to carry out the method for digital image processing according to Claim 1.

8. Data carrier signal (SI2), which transmits the computer program product (CPP) according to Claim 7.

9. Device for detecting at least one fluorescence pattern on an immunofluorescence image (FB) of a biological cell substrate, comprising - a holding device (H) for an object carrier having the cell substrate (S), which was incubated with a patient sample, including autoantibodies, and furthermore with secondary antibodies, which are each marked using a fluorescence stain, - at least one image capture unit (K1) for capturing a fluorescence image (SG) of the cell substrate (S) and furthermore comprising at least one computing unit (R), which is designed to execute the following steps - determining respective items of location information (LI), which indicate respective locations of respective relevant subsections of the cell substrate in the fluorescence image (FB), - extracting respective image subsections, which correspond to the respective subsections of the cell substrate, on the basis of the items of location information (LI), - determining respective second partial confidence measures (ZTKM) of respective presences of the fluorescence pattern on the respective subsections using a second neural network (NN2) on the basis of the respective image subsections, characterized by the steps: - determining respective first partial confidence measures (ETKM) of respective presences of the fluorescence pattern on the respective subsections using a first neural network (NN1) on the basis of the overall fluorescence image, and - determining a confidence measure (KM) of the presence of the fluorescence pattern in the fluorescence image (FB) on the basis of the first partial confidence measures (ETKM) and the second partial confidence measures (ZTKM).

10. Data network device (DV), comprising at least one data interface (DS4) for accepting a fluorescence image (FB), which represents a staining of a cell substrate by a fluorescence stain, and furthermore comprising at least one computing unit (R), which is designed to execute the following steps in the course of digital image processing - determining respective items of location information (LI), which indicate respective locations of respective relevant subsections of the cell substrate (S) in the fluorescence image (FB), - extracting respective image subsections (TB), which correspond to the respective subsections of the cell substrate, on the basis of the items of location information (LI), - determining respective second partial confidence measures (ZTKM) of respective presences of the fluorescence pattern on the respective subsections using a second neural network (NN2) on the basis of the respective image subsections, characterized by the steps: - determining respective first partial confidence measures (ETKM) of respective presences of the fluorescence pattern on the respective subsections using a first neural network (NN1) on the basis of the overall fluorescence image (FB), and - determining a confidence measure (KM) of the presence of the fluorescence pattern in the fluorescence image on the basis of the first partial confidence measures (ETKM) and the second partial confidence measures.