Method and apparatus for detecting presence of fluorescent patterns on immunofluorescence images

Abstract

The invention proposes a method for detecting the presence of a fluorescent pattern on an immunofluorescence image of a biological cell substrate, the method having the steps of incubating a cell substrate with a liquid patient sample, the patient sample potentially having a primary antibody and a secondary antibody labeled with a fluorescent dye, illuminating the cell substrate and detecting the immunofluorescence image; determining localization information indicating a localization of a respective partial area of the cell substrate in the fluorescence image and determining, based on the entire fluorescence image, a first partial confidence of the presence of the fluorescent pattern on the respective partial area by means of a first neural network; extracting, based on the localization information, a partial image area corresponding to the respective partial area of the cell substrate from the fluorescence image; determining, based on the respective partial image area, a second partial confidence of the presence of the fluorescent pattern on the respective partial area by means of a second neural network; determining a confidence of the presence of the fluorescent pattern in the fluorescence image based on the first and second partial confidences.

Description

Technical Field

[0001] This invention relates to methods and apparatus for detecting fluorescent patterns on immunofluorescence images of biological cell matrices. The invention also relates to digital image processing methods, computer program products, data carrier signals, computing units, and data network devices. Background Technology

[0002] Immunofluorescence images of biological cell matrix can be detected using an image acquisition unit such as a microscope. For diagnostic purposes, information about the presence of the expected fluorescent pattern on the fluorescence image may be helpful.

[0003] Immunofluorescence microscopy, or indirect immunofluorescence microscopy, is an in vitro assay used to determine the presence of human antibodies against a specific antigen, in order to answer or assess diagnostic questions. Such antigens are, for example, located in a defined region of a biological cellular matrix. The cellular matrix thus functions as a substrate, incubated with a patient sample in the form of blood or diluted blood, or serum or diluted serum. The patient sample thus potentially contains a defined primary antibody that could indicate the presence of the patient's disease. Such a primary antibody, or specific antibody, can then bind to the antigen in the matrix. This bound primary antibody can then be labeled such that, in another incubation step, a secondary antibody, preferably an anti-human antibody, can bind to the bound primary antibody and subsequently become visible by labeling the secondary antibody with a fluorescent dye. Such a fluorescent dye is preferably a green fluorescent dye, particularly the fluorescent dye FITC. This binding of the primary antibody and the fluorescently labeled secondary antibody can then be seen such that, by irradiating the matrix with excitation light of a defined wavelength and thus stimulating the bound fluorescent dye to emit fluorescent radiation.

[0004] Based on diagnostic questioning, the presence of one or a completely defined type of fluorescent pattern on a specific matrix region can be considered. Therefore, the task is to detect potential fluorescent patterns in immunofluorescence microscopy images within a cell matrix cultured in a prescribed manner, using digital image processing during the immunofluorescence microscopy process.

[0005] Biological cell matrix here may in particular be cell smears of cell lines, colonies of cell lines, and / or biological tissues including biological tissue cells, particularly from tissue sections.

[0006] In indirect immunofluorescence assays (IIFT), islet cell antibodies (ICA) are preferably detected using frozen-fixed tissue sections of primate pancreatic matrix. The pancreas can be divided into exocrine and endocrine regions. Anti-islet cell antibodies react only with the endocrine portion of the organ, also known as the "islet," specifically the Langerhans islets, and target the insulin-producing β-cells. Antigens identified include the 65 kDa isoform of glutamate decarboxylase (GAD65), tyrosine phosphatase homologs IA-2 proteins (IA-2α and IA-2β), the cation transporter ZnT8, insulin, and the insulin precursor proinsulin.

[0007] The destruction of beta cells leads to insulin deficiency, which in turn causes type 1 diabetes. In 70 to 80% of patients with type 1 diabetes, islet cell antibodies can be detected at the time of clinical presentation. These islet cell antibodies are therefore an important distinguishing feature between type 1 diabetes and type 2 diabetes (adolescent diabetes).

[0008] Regarding the specific development of type 1 diabetes, latent autoimmune diabetes in adults (LADA), islet cell antibodies also serve as a criterion for determining insulin dependence in patients. LADA is present in 3 to 12% of patients with phenotype 2 diabetes, characterized by insulin resistance and impaired β-cell insulin secretion.

[0009] In 90% of patients, islet cell antibodies can be detected several years before the onset of diabetes. These islet cell antibodies serve as markers for the so-called prediabetes stage. Therefore, early identification of individuals at increased risk of disease can be ensured. With appropriate intervention, it may be possible to prevent the onset of the disease.

[0010] The titration concentration of antibodies against islet cells decreases as the disease progresses. Very high concentrations of autoantibodies against islet cells (GAD65) can be indicative of myotonic syndrome, cerebellar degeneration, and limbic encephalitis.

[0011] In indirect immunofluorescence assays, Langerhans islets of pancreatic tissue show microparticle-like, smooth to granular fluorescence in positive results.

[0012] In addition, a matrix may be preferably used in the indirect immunofluorescence assay (IIFT) in which multiple Crithidia luciliaeCells are fixed on the matrix. Such a matrix is ​​particularly a cell smear. For example, fixation can be performed using ethanol. The matrix can then be used to detect the binding of autoantibodies to double-stranded deoxyribonucleic acid (DNS) in patient samples. The detection of autoantibodies against DNS is, for example, for the diagnosis of SLE (synonyms: Lupus erythematodes disseminatus This is of decisive significance. Here, it is essential to distinguish, in principle, between two types: antibodies against dsDNS and antibodies against single-stranded denatured DNS (ssDNS). Anti-dsDNS antibodies react with epitopes located on the DNS, while antibodies against ssDNS primarily bind to epitopes originating from purine or pyrimidine base regions. However, these antibodies can also recognize epitopes on the deoxyribonucleophosphate backbone. This is almost exclusively the case in the deoxyribonucleophosphate backbone. Anti-dsDNS antibodies are found in SLE. Their prevalence ranges from 20% to 90%, depending on the assay method and disease activity. Anti-dsDNS antibodies are sometimes also detected in patients with other autoimmune diseases and infections, and in rare cases in clinically healthy individuals. The latter develop SLE in 85% of cases within 5 years of the first anti-dsDNS detection. However, the absence of anti-dsDNS antibodies does not completely rule out SLE. SLE is a systemic autoimmune disease originating in the connective tissue disease group. The diagnosis is based on 11 criteria as revised in the 1997 edition of the American College of Rheumatology (ACR). When four of these 11 criteria are met, a diagnosis of SLE can be made with 80 to 90% certainty. Therefore, indirect immunofluorescence can be an in vitro test for identifying human antibodies against dsDNS. For example, so-called biochips can function as a matrix, wherein the matrix is... Crithidia luciliae Smear preparation. These smears are cultured, for example, with diluted patient samples. In a positive reaction, specific antibodies bind to the antigen. The bound antibody (IgG) is stained with anti-human antibodies labeled with fluorescein in a second culture step and is visible under a fluorescence microscope. Summary of the Invention

[0013] The objective of this invention is to provide an automated method for analyzing immunofluorescence images of biological cell matrix, the method being able to detect the presence of fluorescent patterns on the cell matrix.

[0014] The task according to the present invention is solved by the method according to the present invention.

[0015] According to the present invention, the method has different steps.

[0016] Figure 1A illustrates an exemplary fluorescence image FB1, a fluorescence image of a tissue section of the pancreas in this example. In the example of fluorescence image FB1, the expected fluorescence pattern is not present. Figure 1 B shows another fluorescence image FB2, also a fluorescence image of the pancreas in this case, where the expected fluorescence pattern is present.

[0017] In the example of a matrix in the form of pancreatic slices, the so-called islets are those adapted to a specific or related partial region.

[0018] Figure 2 For this purpose, region TBA is shown as a partial image region in the fluorescence image FB1, which mainly contains islets, or pancreatic islets. The expected fluorescence pattern is not present in region TBA. Figure 2 For this purpose, the fluorescence image FB2 shows partial image regions TBB1, TBB2, and TBB3, which include corresponding islets or pancreatic islets where the desired fluorescence pattern is present. These partial image regions TBB1, ..., TBB3 are again shown in... Figure 2 C is shown separately and in magnification.

[0019] Figure 3 A for Crithidia luciliae An example of a cell matrix in the form of a cell smear shows a fluorescent image FB11 in which the expected fluorescent pattern is substantially absent. Figure 3 B shows that Crithidia luciliae An example of a fluorescence image of the cell matrix in the form of a cell smear, in which the expected fluorescence pattern exists as image FB12. Here, relevant partial regions or partial image areas are indicated by rectangles in fluorescence image FB12. The relevant partial regions here specifically contain the kinetoplasten of Crithidia Luciliae cells.

[0020] Figure 3 C illustrates exemplary partial image regions TBB11, TBB12, and TBB13 in the fluorescence image FB12, wherein the partial image regions are in the range of... Crithidia luciliae A fluorescent pattern is expected to be present in the relevant region of the cell. This fluorescent pattern is specifically the luminescence of the kinetoplast.

[0021] According to the invention, after detecting a fluorescence image, corresponding localization information is determined, indicating the location of a relevant portion of the cellular matrix in the fluorescence image. In the example of the pancreas, the relevant portion is precisely the so-called islet or islet, because the expected fluorescence pattern must be detected in these portions, or it must be confirmed whether the expected fluorescence pattern is absent. For example, in... Figure 2 As can be seen in B, for example, such positioning information can be determined for the corresponding relevant partial areas in the form of rectangles.

[0022] Furthermore, according to the present invention, a first partial confidence level is determined based on the presence of a fluorescent pattern in a corresponding partial region of the entire fluorescence image. Such a partial region, for example, corresponds to... Figure 2 Partial image regions TBB1, TBB2, and TBB3 in B.

[0023] The location information and the corresponding first confidence level are determined based on the entire fluorescence image using a first neural network. Specifically, the determination of both location information and the first confidence level is performed simultaneously using the first neural network, making the first neural network a combination of the two tasks described above.

[0024] According to this invention, corresponding partial image regions corresponding to specific regions of the cell matrix are extracted from fluorescence images. This is based on previously obtained localization information. Figure 2 In C, such partial image regions TBB1, TBB2, and TBB3 are shown as examples.

[0025] Furthermore, according to the present invention, a second partial confidence level regarding the presence of a fluorescent pattern in a corresponding partial image region is determined using a second neural network. Therefore, for example, analysis is performed using a second neural network. Figure 2 The image regions TBB1, TBB2, and TBB3 in C are used. In particular, a second neural network is used to determine the corresponding second part confidence in the following manner: each of the corresponding partial image regions is analyzed or processed separately by the second neural network in order to determine the corresponding second part confidence.

[0026] According to the present invention, the confidence level of the presence of the fluorescent pattern in the fluorescent image is finally determined based on the first part of the confidence level and the second part of the confidence level.

[0027] The possible advantages of the present invention will now be described in more detail.

[0028] It is possible, in principle, to perform the task of detecting the presence of fluorescent patterns across a large, comprehensive fluorescence image using a single neural network. This would involve analyzing the large image using the corresponding factual information of "positive" or "negative" regarding the presence of fluorescent patterns during the training phase, thereby determining the class classification for "positive" or "negative." In this approach, the single neural network must be able to take into account a large amount of image information from the entire fluorescence image. This would require high network complexity and therefore a significant amount of computational operations, making the computational cost a challenge.

[0029] Furthermore, it must be noted that neural networks used to analyze large total images can extract image features and perhaps classify them as relevant, even though these features exist in the total image, they may be irrelevant to the actual classification task of the neural network. To specifically direct the attention of the neural network and the image features to be learned onto only actually relevant image structures, the method proposed here can be used as follows: First, in the first level, a first neural network directly identifies and classifies relevant structures in the entire fluorescence image in terms of their localization, and then in the second level, a second neural network additionally classifies the corresponding partial image regions of the relevant structures. Here, by combining the classification results or the determination of the first partial confidence from the first level with the classification results or the determination of the second partial confidence from the second level, and by aggregating these results into a final confidence score indicating the presence of the fluorescence pattern in the entire fluorescence image, a high validity of the final confidence score is generated. This results in another advantage: by using partial confidence scores for partial regions or partial image regions, it is possible to controllably incorporate the classification results from two levels or two neural networks into the final confidence score. This leads to good model interpretability in terms of the functionality of the two neural networks in the method. By identifying relevant partial regions and corresponding partial image regions as individual elements, the number of partial results available in the method, for example in the form of partial confidence, is automatically generated, where such information or quantity information does not exist in the total image scheme, since only the total output of each confidence level for the total image is given at this time, preferably as a report of "positive" or "negative".

[0030] In other words, the method according to the present invention deviates from the approach used to analyze the entire fluorescence image using only a single neural network, in which a first neural network first determines localization information indicating relevant partial regions of the cellular matrix. A second neural network then must process each such partial image region individually to detect the possible presence of a fluorescence pattern in the corresponding partial image region and determine the corresponding partial confidence level.

[0031] That is, the second neural network can significantly reduce its complexity compared to the solution in which a single neural network must determine the confidence level of the presence of a fluorescence pattern for the entire fluorescence image.

[0032] The second neural network can therefore be trained on significantly smaller partial image regions and only needs to analyze partial images of this type, which represent the actual relevant partial regions of the cell matrix.

[0033] Therefore, the second neural network can thus determine a significantly more effective partial confidence level regarding the presence of a fluorescent pattern in a corresponding portion of the image or region. Consequently, the second neural network can also determine a more effective confidence level regarding the presence of a fluorescent pattern in a fluorescent image.

[0034] The method according to the invention is further advantageous because the first neural network not only determines the relevant partial region using the localization information but also, in particular, simultaneously determines the first partial confidence level. Therefore, the first neural network is specifically trained to simultaneously determine the first partial confidence level and the localization information. Such a network can be particularly efficient relative to other networks in terms of computational complexity and required computational memory. Such a network can be implemented, in particular, as a so-called "One-Shot detector."

[0035] Furthermore, the method according to the invention is advantageous because the confidence level is determined in consideration of or based on the first and second partial confidence levels, thereby allowing information about the partial confidence levels from the two neural networks to be accessed in order to determine a particularly reliable confidence level of the presence of the fluorescent pattern in the fluorescent image.

[0036] Preferably, the confidence level is output.

[0037] Preferably, the determination of the weighted partial confidence is performed by using the weights of the first and second partial confidence, and the determination of the confidence is based on the weighted partial confidence.

[0038] The confidence level is preferably determined by applying a threshold to a weighted partial confidence level.

[0039] Preferably, the method further comprises the steps of: determining a corresponding presence confidence level, the presence confidence level indicating the corresponding location of a relevant partial region: to what extent the actual relevant partial region of the cell matrix exists at the corresponding location; confirming, based on the entire fluorescence image using a first neural network, whether the location information indicates a certain number of overlapping partial image regions; retaining the location information of the determined partial image region from the number of overlapping partial image regions based on the presence confidence level of the overlapping partial image regions, and discarding the location information of other partial image regions from the number of overlapping partial image regions; and extracting the corresponding partial image region from the fluorescence image based on the remaining location information.

[0040] Furthermore, the method specifically includes the steps of: determining a certain amount of partial image regions where fluorescent patterns are present; and determining the brightness value of the presence of fluorescent patterns on the immunofluorescence image based on the amount of partial image regions where fluorescent patterns are present.

[0041] Furthermore, a method for digital image processing is proposed, comprising the steps of: providing an immunofluorescence image representing biological cell matrix stained with a fluorescent dye; determining corresponding localization information, the localization information indicating the corresponding location of a relevant partial region of the cell matrix in the fluorescence image, and determining a corresponding first partial confidence level of the presence of a fluorescent pattern in the corresponding partial region based on the entire fluorescence image using a first neural network; extracting a corresponding partial image region based on the localization information, the partial image region corresponding to a corresponding partial region of the cell matrix; determining a corresponding second partial confidence level of the presence of a fluorescent pattern in the corresponding partial region based on the corresponding partial image region using a second neural network, and determining the confidence level of the presence of the fluorescent pattern in the fluorescence image based on the first partial confidence level and the second partial confidence level.

[0042] Furthermore, a computer program product is proposed, the computer program product including instructions that, when the program is executed by a computer, cause the computer to implement the method for digital image processing.

[0043] Furthermore, a data carrier signal is proposed, which transmits the computer program product.

[0044] Furthermore, an apparatus for detecting at least one fluorescent pattern on an immunofluorescence image of a biological cell matrix is ​​proposed. The apparatus includes: a holding device for a slide comprising a cell matrix cultured with a patient sample containing an autoantibody and a secondary antibody, the autoantibody and the secondary antibody being labeled with a fluorescent material. The apparatus also includes at least one image acquisition unit for detecting the fluorescence image of the cell matrix. The apparatus further includes at least one computing unit configured to perform the following steps: determining corresponding location information indicating the location of a corresponding relevant partial region of the cell matrix in the fluorescence image; determining a first partial confidence level based on the entire fluorescence image using a first neural network to indicate the presence of the fluorescent pattern in the corresponding partial region; extracting a corresponding partial image region based on the location information, the partial image region corresponding to a corresponding partial region of the cell matrix; determining a second partial confidence level based on the corresponding partial image region using a second neural network to indicate the presence of the fluorescent pattern in the corresponding partial region; and determining a confidence level of the presence of the fluorescent pattern in the fluorescence image based on the first and second partial confidence levels.

[0045] Furthermore, a computing unit is proposed, configured to perform the following steps during digital image processing: receiving an immunofluorescence image representing biological cell matrix stained with a fluorescent dye; determining corresponding localization information indicating the location of a corresponding relevant partial region of the cell matrix in the fluorescence image, and determining a first partial confidence level based on the entire fluorescence image using a first neural network to indicate the presence of a fluorescent pattern in the corresponding partial region; extracting a corresponding partial image region based on the localization information, the partial image region corresponding to a corresponding partial region of the cell matrix; determining a second partial confidence level based on the corresponding partial image region using a second neural network to indicate the presence of a fluorescent pattern in the corresponding partial region; and determining a confidence level of the presence of a fluorescent pattern in the fluorescence image based on the first and second partial confidence levels.

[0046] Furthermore, a data network device is proposed, having at least one data interface for receiving fluorescence images representing the staining of cell matrix with fluorescent dyes. The data network device also includes at least one computing unit configured to perform the following steps during digital image processing: determining corresponding location information indicating the location of a corresponding relevant portion of the cell matrix in the fluorescence image; determining a first partial confidence level based on the entire fluorescence image using a first neural network to indicate the presence of a fluorescence pattern in the corresponding partial region; extracting a corresponding partial image region based on the location information, the partial image region corresponding to a corresponding portion of the cell matrix; determining a second partial confidence level based on the corresponding partial image region using a second neural network to indicate the presence of a fluorescence pattern in the corresponding partial region; and determining a confidence level of the presence of a fluorescence pattern in the fluorescence image based on the first and second partial confidence levels. Attached Figure Description

[0047] The invention will then be further illustrated with reference to the accompanying drawings, using a particular embodiment without limiting the general inventive concept. Herein lies: Figure 1 and 2 Immunofluorescence images and partial images of the first biological cell matrix are shown; Figure 3 Immunofluorescence images and partial images of the second biological cell matrix are shown; Figure 4 The steps of a preferred embodiment of the method according to the invention are shown; Figure 5 The preferred steps for determining the confidence level with partial confidence weighting are shown; Figure 6 The steps of the method according to the invention are shown with respect to another preferred embodiment; Figure 7 The steps for determining the brightness level of a fluorescent pattern are shown; Figure 8 This illustrates a preferred design of the second neural network; Figure 9 A preferred embodiment of the apparatus according to the invention is shown; Figure 10 A preferred embodiment of the computing unit according to the invention is shown; Figure 11 A preferred embodiment of the data network apparatus according to the present invention is shown; Figure 12 An illustration showing a computer program product and data carrier signals; Figure 13 The results panel displays the experimental results. Detailed Implementation

[0048] Figure 4 The steps of a preferred embodiment of the method according to the invention are shown. In step S1, a cell matrix is ​​cultured in a liquid patient sample, which potentially contains a primary antibody and also includes a secondary antibody labeled with a fluorescent dye. Furthermore, in step S1, the cell matrix is ​​irradiated with excitation radiation and detected with an immunofluorescence image.

[0049] The immunofluorescence image can be provided as a data item FB through step S1.

[0050] Such immunofluorescence images are exemplified as image FB1 in Figure 1 As shown in A. In this case, in the relevant cellular region, the islets, or pancreatic cellular matrix, are absent, and the cellular matrix is ​​here in the form of pancreatic slices. Figure 1 In the immunofluorescence image FB2 of B, such immunofluorescence is present in the relevant partial regions.

[0051] Figure 2 A again shows the immunofluorescence image FB1 with a specified partial area or partial image region TBA, where the relevant partial area of ​​the cell matrix is ​​located within the partial area. Compared to Figure 2 In immunofluorescence image FB2, partial regions or partial image regions TBB1, TBB2, and TBB3 show that the expected fluorescent pattern is not present in immunofluorescence image FB1, but is present in partial image regions of immunofluorescence image FB2. The corresponding fluorescent patterns in partial image regions TBB1, TBB2, and TBB3 are again observed. Figure 2 This can be seen in the enlarged diagram in C.

[0052] In the context of this application, a portion of an image may also be referred to as a portion of an image region.

[0053] according to Figure 4 In step S2, corresponding positioning information is determined, which indicates the location of a relevant portion of the cell matrix in the fluorescence image. This is performed using a neural network NN1, which can be referred to as the first neural network.

[0054] Furthermore, in step S2, based on the entire fluorescence image, the corresponding first partial confidence level of the fluorescence pattern in the corresponding partial region or the corresponding related partial region is determined simultaneously using a neural network NN1. Therefore, this is performed in particular as a combined or simultaneous process.

[0055] The determined location information can be provided as the combined data item LI. The first part of the confidence level can be provided as the data item ETKM.

[0056] The positioning information preferably indicates the location of a portion of the region in the fluorescence image, and further indicates the width and height of the region. Therefore, the positioning information preferably indicates a rectangle at a defined location in the fluorescence image, along with the height and width of the rectangle.

[0057] Such location information for a partial region or a partial image region can be given as a vector n.

[0058] Entry n1 can indicate the X position of the rectangle in the fluorescence image, and entry n2 can indicate its Y position. In addition, entry n3 can indicate the width of the rectangle, and entry n4 can indicate the height of the rectangle.

[0059] For Figure 2 A portion of C, such as the first part of the confidence score ETKM in the region TBB1, can be represented by vector o.

[0060] Given, where entry o1 gives the probability that a portion of the image region belongs to the positive fluorescence class of the pattern, and entry o2 indicates the probability that a portion of the region belongs to the negative fluorescence class of the pattern, or the missing fluorescence pattern.

[0061] Therefore, the positioning information can be a corresponding vector n for each partial image region, and thus the first confidence level can be a corresponding vector o for each partial image region.

[0062] The first neural network is preferably a so-called object detection network. Particularly preferred is a so-called one-shot detector type object detection network. Specifically, the first neural network is a so-called YOLO network.

[0063] For example, such a network can be trained by presenting the entire fluorescence image to the input layer of the neural network, whereby the localization information of the corresponding partial regions of the cell matrix is ​​provided as ground truth information, and the class attribution of the corresponding partial image regions for the class of "positive fluorescence" or "negative fluorescence" is also provided.

[0064] Preferably, the first neural network NN1 further determines another piece of information OB based on the entire fluorescence image, which also... Figure 6 The image shows information OB, which indicates the confidence level of a given location or location information. This confidence level indicates the degree to which the relevant partial region of the cellular matrix actually exists at the corresponding location. This information OB can be output for the corresponding partial image region using a corresponding scalar value c.

[0065] Therefore, the first neural network preferably outputs tuples for the relevant partial image regions of each detector with indices k=1…K. Therefore, training the neural network can be particularly effective when providing the complete fluorescence image and the corresponding tuples. The information is presented in the form of factual truth information for each of the regions.

[0066] The detailed structure of such a first neural network for the particular implementation of the YOLO network is given in J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only LookOnce: Unified, Real-Time Object Detection," 2016 IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2016, pp. 779-788, doi: 10.1109 / CVPR.2016.91.

[0067] The confidence level c is referred to there as the so-called "confidence". The vector n is given there by the values ​​of the image position x, y and the width b and height h. The class probabilities o1 or o2 are given there as the so-called conditional class probabilities "C".

[0068] In step S3, based on the positioning information, corresponding partial image regions are extracted from the fluorescence image, and these partial image regions correspond to corresponding partial regions of the cell matrix. These partial regions are, for example, Figure 2 A portion of the image region in C. These portion of the image region can be provided as a combined data item TB. Preferably, the extracted portion of the image region is the same as the corresponding portion region indicated by the corresponding positioning information. In particular, the portion of the image region may deviate to the portion region, especially the portion region in the edge region.

[0069] In step S4, the confidence level of the fluorescence pattern in the corresponding partial image region is determined by the second neural network NN2 based on the corresponding partial image region.

[0070] Step S4 can then provide the second part of the confidence level as data item ZTKM.

[0071] The second neural network preferably outputs a vector d for each partial image region, which has a class probability for the corresponding class "fluorescence positive" or "fluorescence negative". Therefore, the vector d can be a vector

[0072] Here, entry d1 can be the class probability of "fluorescence positive", and entry d2 can be the class probability of "fluorescence negative". Therefore, the second part of the confidence can be a vector d for a partial image region.

[0073] In step S5, the confidence level KM for the presence of the fluorescent pattern is determined based on the second confidence level ZTKM and the first confidence level ETKM obtained in step S5. Preferably, the confidence level KM is output in step S6. Such output can preferably be performed via a data interface in the form of data items, or particularly preferably by means of optical output on a display device of the computing unit, such as a computer monitor.

[0074] Preferably, the confidence level KM can be determined by averaging the corresponding first and second partial confidence levels for each partial image or region having indices k=1…K, thereby generating an average partial confidence level for each partial image region. These averaged partial confidence levels can then be averaged again over all partial image regions or all indices k=1…K to determine the confidence level for the presence of the fluorescent pattern.

[0075] according to Figure 5 Alternatively, step S51 can be modified by implementing sub-steps S511 and S512 in step S5. In step S511, the first partial confidence level ETKM and the second partial confidence level ZTKM are input. In the first step, a weighted partial confidence level is determined using the weights of the first partial confidence level ETKM and the second partial confidence level ZTKM. The weighted partial confidence level can be provided as a data item GTKM. Therefore, in step S512, the confidence level KM is determined based on the weighted partial confidence level GTKM.

[0076] Therefore, according to a preferred embodiment, a weighted average is performed on one portion of the image region for each of the first and second part confidence scores. This can thus be described as determining the corresponding weighted partial confidence score using the weights of the corresponding first and second part confidence scores.

[0077] Therefore, in a particular implementation, each partial image region is weighted using weight factors w1 and w2, i.e., according to...

[0078] The vector is determined for each partial image region k=1…K. .

[0079] Therefore, for each partial image region with indices k=1…K, we can consider vector representation. The weighted partial confidence scores are used to determine the confidence level KM. In particular, it is preferable to consider values ​​r1 for each partial image region, representing the class probability of "fluorescent positive". These values ​​R1 can then be subjected to thresholding. Therefore, the confidence level is determined by applying thresholds to the weighted partial confidence scores.

[0080] Preferably, when the corresponding value r1 is greater than the threshold, a portion of the image region is assigned to the class "fluorescent positive". For example, the value R1 can be in the range of [0…1] and a threshold of 0.6 can be applied.

[0081] The following image regions are then classified as belonging to the "fluorescence negative" category, and for the aforementioned image regions, the value r1 does not exceed the threshold.

[0082] Therefore, a so-called majority vote can be preferred. In this majority vote, the confidence level can be determined specifically as a report of "yes" / "no," where the majority of a portion of the image region is decisive. Thus, if there are more portions of the image region identified as having a fluorescent pattern or identifying a "fluorescent positive" pattern, the confidence level is "yes." Conversely, the probability of a "no" confidence level applies.

[0083] The advantage of applying a threshold to a weighted partial confidence level is that the method allows adjustment of the starting confidence level from which the entire fluorescence image itself is evaluated as positive or the expected fluorescence pattern is evaluated as present. This can be adjusted in particular based on the individual weighted partial confidence levels for different parts of the image.

[0084] Based on the association between the resulting grid structure and the so-called striding of a neural network, such as a neural network NN1, it is possible that such a first neural network NN1 determines multiple localization information, the localization indicating corresponding overlapping partial regions. It is not necessary to... Figure 2 In image regions TBB1, TBB2, and TBB3 in C, the spatial separation of partial regions or partial image regions must be given. It is also possible to identify partial image regions that at least partially overlap in their surfaces. In particular, multiple overlapping partial regions can be generated, these partial regions indicating the same related partial regions on the substrate, such as the same islet. Preferably, two partial regions are considered to overlap when the planar overlap of two partial regions exceeds a predetermined value. For example, the overlap can be determined in the value range of [0…1] and thus overlap is only confirmed when the overlap exceeds a threshold value, such as 0.2.

[0085] Such overlapping partial image regions can, in principle, indicate the same or similar related partial image regions. Therefore, it is advantageous to select only one partial image region from such overlapping partial image regions, since the identified related partial regions of the cell matrix should be included only once, rather than multiple times, in the later evaluation used to determine confidence levels.

[0086] Figure 6 Therefore, a preferred embodiment of the method according to the invention is shown.

[0087] The order of steps S1 and S2 is the same as... Figure 4 The implementation form.

[0088] In step S2, specifically for each relevant partial region, location information LI, a first partial confidence level ETKM, and a confidence level OB are determined. Step S21 then follows step S2, in which it is confirmed whether the location information given by data item LI indicates a certain number of overlapping partial regions. Specifically, this certain number is determined in step S21. The number of overlapping partial image regions and their location information can be provided as a certain amount of location information MLI as a data item.

[0089] From the overlapping partial image regions of the quantity, preserve the following partial image regions or positioning information, wherein the partial image regions or positioning information have the highest value of the confidence OB.

[0090] In step S22, the location information of the determined partial image regions is identified and retained from the overlapping partial image regions. Location information of other partial image regions is discarded from the overlapping partial image regions. Furthermore, location information of partial image regions that do not indicate overlap is retained. The retained location information forms the basis for further processing. The retained location information is shown as data item VLI. Therefore, based on the retained location information VLI, it can be ensured that, in overlapping partial image regions indicating the same related areas of the matrix, only the identified partial regions enter the final evaluation or processing. This ensures that the related cellular matrix region enters the subsequent evaluation only once as a partial image region without distorting the results.

[0091] In step S3, the corresponding partial image regions are extracted from the fluorescence image based on the remaining or retained positioning information VLI.

[0092] Figure 7 The preferred steps for determining the brightness level of the presence of a fluorescent pattern are shown. In step S100, a certain number of partial image regions having the presence of a fluorescent pattern are determined. Thus, for example, a certain number of partial image regions can be determined based on a weighted partial image confidence level (GTKM), where the partial image confidence level (GTKM) or its value R1 exceeds a threshold.

[0093] The portion of the image region, or the remaining portion of the image region, can be shown as a data item VTB.

[0094] In step S101, the brightness level HM of the fluorescent pattern present on the immunofluorescence image is determined based on the partial image region VTB of the amount, wherein the partial image region has the presence of the fluorescent pattern.

[0095] Therefore, in particular, only the following partial image regions are considered as partial image regions VTB, which have class-determining "positive fluorescence".

[0096] This analysis is performed on these partial image regions, particularly by statistically analyzing the pixel values ​​within these regions.

[0097] For each partial image region, a histogram of pixel luminance values ​​is constructed, and then luminance values ​​are determined, where 80% of the pixels have an intensity value greater than the 0.8 quantile. This 0.8 quantile value can then be used as a partial luminance value for the partial image region. For all the partial image regions to be observed, the corresponding partial image luminance values ​​are preferably averaged. Therefore, it is preferable to average the luminance quantile values ​​of the corresponding partial image regions. Alternatively, median determination can be performed.

[0098] The luminance quantile value determined using the median or average value can then be used as the luminance level HM output.

[0099] Particularly preferably, the generated luminance quantization values ​​are also quantized in four quantization ranges. For example, the luminance quantization values ​​can be in the range of [0…255], thereby quantizing with order values ​​of [1…5] in the four quantization ranges.

[0100] Therefore, this order value can be output to the user.

[0101] Figure 8 The preferred structure of the second neural network NN2 is shown. The neural network NN2 analyzes the corresponding partial image TB and determines the corresponding second part confidence level ZTKM for the corresponding partial image TB.

[0102] Therefore, the neural network NN2 can employ a sequence of corresponding convolutional modules CM. Preferably, the corresponding convolutional module CM here has a sequence of convolutional steps CS, batch normalization steps BS, another convolutional step CS, and another batch normalization step BS, followed by a max pooling step MAS.

[0103] Multiple such convolutional modules (CMs) can be sequentially linked.

[0104] The final convolutional module CM can then be further processed using multiple densely connected layers (DL), also known as fully connected layers. The second confidence level (ZTKM) can be determined using a Softmax layer (SML).

[0105] Figure 9The apparatus V1 is shown, by means of which the method according to the invention can preferably be implemented. The apparatus V1 may be referred to as a fluorescence microscope. The apparatus V1 has a holding device H for a matrix S or a slide comprising such a matrix S, which is cultured in the manner previously described. Excitation light AL of an excitation source LQ is guided toward the matrix S through an optical system O. The resulting fluorescence radiation FL is then emitted back through the optical system O and passes through a dichroic mirror SP1 and an optional filter F2. Preferably, the fluorescence radiation FL is filtered out by a filter FG for the green channel. The camera K1 is preferably a monochrome camera, which then detects the fluorescence radiation FL in the green channel in the presence of the filter FG. According to an alternative embodiment, the camera K1 is a color imaging camera, which is sufficient without using the filter FG and detects fluorescence images in the corresponding color channels as the green channel using a Bayer array. The camera K1 provides image information BI or a fluorescence image to a computing unit R, which processes the image information BI. Preferably, the computing unit R can output or provide data ED, such as fluorescence images and / or confidence levels, through the data interface DS1.

[0106] Figure 10 A computing unit according to the invention is shown, wherein the computing unit preferably receives a fluorescence image FB as a data signal SI via a data interface DS2, according to a preferred embodiment. The computing unit R can then determine and provide the aforementioned information as a data signal SI3 via a data interface DS3. Preferably, this can be done via a wired or wireless data network. Particularly preferably, the computing unit R has an output interface AS for outputting information via an output unit AE. The output unit AE is preferably a display unit for optical display of the aforementioned information.

[0107] Figure 11 A data network device DV according to a preferred embodiment of the invention is shown. The data network device DV receives a fluorescence image FB as a data signal SI1 via a data interface DS4. The data network device DV has the previously described computing unit R and storage unit MEM. The computing unit R, storage unit MEM, and data interface DS4 are preferably interconnected via an internal data bus IDB.

[0108] Figure 12 An embodiment of the proposed computer program product CPP is shown. The computer program product CPP can receive its data signal SI2 through the data interface DSX via the computer CO.

[0109] Figure 13 Showing the use of Crithidia luciliae Results panel of results for implementation of cell smears in biological cell matrix.

[0110] According to factual information, 4,923 fluorescent images that did not exhibit fluorescence or were classified as "negative" were also reported as negative by the method according to the present invention.

[0111] Twenty-three negative fluorescence images were classified as positive by the method according to the present invention.

[0112] 2,532 fluorescent images showing a "positive" fluorescence class were also classified as "positive" according to the method of the present invention.

[0113] 195 fluorescent "positive" images were classified as fluorescent "negative" by the method according to the present invention.

[0114] Embodiments of the present invention can be implemented in hardware or software according to the defined implementation requirements. The implementation can be carried out using digital storage media, such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, flash memory, hard disks, or other magnetic or optical storage, on which electronically readable control signals are stored. These control signals, in conjunction with or in combination with programmable hardware components, enable the implementation of the corresponding methods.

[0115] Programmable hardware components, particularly computing units, can be formed from processors, computer processors (CPU), graphics processing units (GPU), computers, computer systems, application-specific integrated circuits (ASIC), integrated circuits (IC), systems on chips (SOC), programmable logic elements, or field-programmable gate arrays (FPGA) that include microprocessors.

[0116] The digital storage medium can therefore be machine-readable or computer-readable. Some embodiments therefore include a data carrier having electronically readable control signals that can interact with a programmable computer system or programmable hardware components to implement one of the methods described herein. One embodiment is therefore a data carrier (or digital storage medium or computer-readable medium) on which a program for implementing one of the methods described herein is recorded.

[0117] Generally, embodiments of the present invention can be implemented as a program, firmware, computer program, or computer program product including program code, or as data, wherein the program code or the data is effective for implementing one of the methods when the program runs on a processor or programmable hardware component. The program code or data may also be stored, for example, on a machine-readable medium or data medium. The program code or data may also exist as source code, machine code, or bytecode, as well as as other intermediate code.

[0118] Another embodiment is a data stream, signal sequence, or sequence of signals describing a procedure for implementing one of the methods described herein. The data stream, signal sequence, or sequence of signals may, for example, be configured for transmission via a data communication connection, such as via the Internet or other networks. The embodiments are also, therefore, signal sequences representing data suitable for transmission via a network or data communication connection, wherein the data describes the procedure.

[0119] According to one embodiment, the program can implement one of the methods during its implementation, for example, by reading a memory location or writing one or more data into the memory location, thereby causing, if necessary, switching processes or other processes in transistor structures, amplifier structures, or other electrical, optical, magnetic, or otherwise operating components. Accordingly, data, values, sensor values, or other information of the program can be detected, determined, or measured by reading the memory location. The program can therefore detect, determine, or measure parameters, values, measurement parameters, and other information by reading one or more memory locations.

Claims

1. Methods for digital image processing, The method has the following characteristics: - Provides immunofluorescence images (FB) representing biological cell matrix (S) stained with fluorescent dyes. - Determine the corresponding localization information (LI), which indicates the corresponding localization of the relevant partial region of the cell matrix in the fluorescence image, and simultaneously determine the corresponding first partial confidence level (ETKM) of the fluorescence pattern in the corresponding partial region based on the entire fluorescence image using a combined first neural network (NN1). -Based on localization information (LI), extract the corresponding partial image region (TB) from the fluorescence image that corresponds to the corresponding partial region of the cell matrix. -Based on the corresponding partial image region (TB), the corresponding second partial confidence level (ZTKM) of the fluorescent pattern in the corresponding partial region is determined by using a second neural network (NN2). - Determine the confidence (KM) of the presence of the fluorescent pattern in the fluorescent image (FB) based on the first part confidence (ETKM) and the second part confidence (ZTKM).

2. The method according to claim 1, The method also has a confidence level (KM) output.

3. The method according to claim 1, The method also has the following characteristics: - Determine the weighted partial confidence level (GTKM) using the weights of the first part confidence level (ETKM) and the second part confidence level (ZTKM). - The confidence level is determined based on the weighted partial confidence level.

4. The method according to claim 3, The method also includes: determining the confidence level (KM) by applying a threshold to a weighted partial confidence level (GTKM).

5. The method according to claim 1, The method also has the following characteristics: - Based on the entire fluorescence image, a first neural network is used to determine the corresponding presence confidence (OB), which indicates the localization of the corresponding relevant partial region: to what extent the actual relevant partial region of the cellular matrix is ​​present at the corresponding location. - Confirm whether the location information (LI) indicates multiple overlapping partial image regions of a certain amount (MLI). -Based on the presence confidence (OB) of overlapping partial image regions, the localization information of certain partial image regions is retained from the certain number of overlapping partial image regions. Furthermore, the positioning information of other partial image regions is discarded from the certain amount of overlapping partial image regions. - Extract the corresponding partial image region from the fluorescence image (FB) based on the remaining location information (VLI).

6. The method according to claim 1, The method also has - Identify a certain amount of image regions containing fluorescent patterns. - The brightness value (HM) of the fluorescent pattern on the immunofluorescence image is determined based on the certain amount of partial image regions containing the fluorescent pattern.

7. A computer program product (CPP) comprising instructions that, when executed by a computer, cause the computer to perform the method for digital image processing as described in claim 1.

8. An apparatus for detecting at least one fluorescent pattern on an immunofluorescence image (FB) of a biological cell matrix. The device has: - A holding device (H) for a slide comprising a cell matrix (S) cultured with a patient sample containing autoantibodies and a secondary antibody, the autoantibodies and the secondary antibody being labeled with fluorescent dyes. - At least one image acquisition unit (K1) for detecting immunofluorescence images (FB) of the cell matrix (S). The device further includes at least one computing unit (R) configured to perform the following steps: - Determine the corresponding location information (LI), which indicates the corresponding location of the corresponding relevant partial region of the cell matrix in the fluorescence image (FB), and simultaneously determine the corresponding first partial confidence (ETKM) of the fluorescence pattern corresponding to the corresponding partial region based on the entire fluorescence image (FB) by means of a combined first neural network (NN1). - Based on localization information (LI), extract corresponding partial image regions, which correspond to corresponding partial regions of the cell matrix. -Based on the corresponding partial image regions, the second partial confidence level (ZTKM) of the fluorescent pattern in the corresponding partial regions is determined by using a second neural network (NN2). - Determine the confidence (KM) of the presence of the fluorescent pattern in the fluorescent image (FB) based on the first part confidence (ETKM) and the second part confidence (ZTKM).

9. Data network device (DV). The data network device has at least one data interface (DS4) for receiving immunofluorescence images (FB), which represent the cell matrix stained with fluorescent dyes. The data network device further includes at least one computing unit (R) configured to perform the following steps during digital image processing: - Determine the corresponding localization information (LI), which indicates the corresponding localization of the corresponding relevant partial region of the cell matrix (S) in the fluorescence image (FB), and simultaneously determine the corresponding first partial confidence level (ETKM) of the fluorescence pattern in the corresponding partial region based on the entire fluorescence image (FB) by means of a combined first neural network (NN1). - Based on the location information (LI), extract the corresponding partial image region (TB), which corresponds to a corresponding partial region of the cell matrix. -Based on the corresponding partial image regions, the second partial confidence level (ZTKM) of the fluorescent pattern in the corresponding partial regions is determined by using a second neural network (NN2). - The confidence (KM) of the presence of the fluorescent pattern in the fluorescent image is determined based on the first part confidence (ETKM) and the second part confidence.

Images