Generating quantitative ground truth for ihc stained slides

Abstract

This disclosure provides methods and apparatus for generating training data, training machine learning models, and analyzing biological samples. Generating training data includes: acquiring a first image of a biological sample; generating a first set of annotations for the first image based on a second image of the biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; and outputting the first image and a benchmark truth including the first set of annotations as data for training a first machine learning model (“ML”) to analyze the biological sample. Furthermore, the generated (output) training data can then be used to train the ML model, and the trained ML model can be used to analyze the biological sample.

Description

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Applications Nos. 63 / 605,424 and 63 / 605,435, filed December 1, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to methods and apparatus for detecting targets in biological tissues with imaging assistance, and more specifically to digital microscopy. This disclosure also relates to methods and apparatus for generating ground truth for training machine learning models to analyze images of biological samples, training such models, and performing analysis using the trained models. Background Technology

[0003] Histological specimens are typically set on glass slides in the form of thin slices of patient tissue fixed to the surface of each slide. Various chemical or biochemical processes can be used to stain the tissue with one or more staining compounds to distinguish cellular components, which can then be further evaluated using a microscope. Brightfield slide scanners are commonly used for digital analysis of these slides.

[0004] When analyzing tissue samples on microscope slides, staining the tissue or parts thereof with stained or fluorescent dyes can aid in the analysis. Histological staining agents typically enhance the ability to visualize or differentially identify microstructures. Hematoxylin and eosin (“H&E”) staining agents are the most commonly used staining agents for histological samples in optical microscopy.

[0005] In addition to H&E staining agents, other staining agents or dyes are applied to provide more specific staining and a more detailed view of tissue morphology. Immunohistochemical (“IHC”) staining agents are highly specific because they use peroxidase substrates or alkaline phosphatase (“AP”) substrates for IHC staining, providing a uniform staining pattern that appears to the observer as a uniform color with intracellular resolution of cellular structures such as membranes, cytoplasm, and nucleus. Formalin-fixed paraffin-embedded (“FFPE”) tissue samples, metaphases, or histological smears are typically analyzed by staining on glass slides, where specific biomarkers (such as target proteins or nucleic acids) can be stained with H&E and / or with staining dyes (hereinafter referred to as “chromogens” or “chromogen fractions”). IHC staining is a commonly used tool for assessing the presence of specific biomarkers in tissue samples. In situ hybridization (“ISH”) can be used to detect target nucleic acids in tissue samples. ISH can use nucleic acids labeled with directly detectable portions (such as fluorescent portions) or indirectly detectable portions (such as portions recognized by antibodies that can subsequently be used to generate a detectable signal). Further methods such as fluorescent ISH (“FISH”) have also been applied.

[0006] Chromatography typically has lower sensitivity compared to other detection techniques such as radioactivity, chemiluminescence, or fluorescence, but it offers the advantage of permanent, clearly visible color, which can be observed with the naked eye, for example, under a bright-field microscope. However, there is a need for substrates with additional properties that can be used in a variety of applications, including multiplex detection such as IHC or ISH assays.

[0007] Additional capabilities for advanced image analysis of histological sections, such as in digital pathology, can improve the detection and evaluation of specific molecular markers, tissue features, and organelles. However, achieving high accuracy remains a challenging problem, not only for machine learning-based algorithms but also for experienced professionals. Summary of the Invention

[0008] In recent years, digital pathology has become more prevalent as many stained tissue sections are digitally scanned at high resolution (e.g., 40×) and viewed in whole-slide images (“WSI”) using digital devices (e.g., PCs, tablets, etc.) instead of standard microscopes. Information in digital format enables digital analysis applicable to WSI for diagnostic purposes. Recently, quantitative staining methods have been developed that convert antibody / antigen complexes into dots. These dots can then be detected and provide quantitative measurements of the expression of desired molecules, such as proteins.

[0009] Developing robust automated methods is particularly challenging due to the significant variations in the shape, color, orientation, and density of staining across different tissues and staining agent types. Therefore, there is a need for more robust and scalable solutions for digital microscopy imaging, and more specifically, for methods, systems, and apparatuses that enable digital microscopy imaging using deep learning-based segmentation, instance segmentation based on annotation, and / or implement user interfaces configured to facilitate user annotation of instance segments within biological samples.

[0010] Therefore, in some aspects, this disclosure provides systems and methods for using artificial intelligence (“AI”) (e.g., machine learning (“ML” models) to facilitate the detection of spots generated by staining biological samples using quantitative methods. Such detection can be used to generate a ground truth (the basis for which the artificial intelligence can be further used to train the artificial intelligence.

[0011] According to a first aspect, a method for generating training data is provided, the method comprising: i) acquiring a first image of a biological sample; ii) generating a first set of annotations for the first image based on a second image of the biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; and iii) outputting the first image and a benchmark truth including the first set of annotations as data for training a first ML model to analyze the biological sample.

[0012] According to the second aspect, in addition to the first aspect, the generation of the first set of annotations is based on a statistical measure of the points in the second image.

[0013] According to the third aspect, in addition to the second aspect, statistical measures include the number of points and / or the density of points.

[0014] According to the fourth aspect, in addition to the second or third aspect, generating the first set of annotations includes generating annotations for each of the multiple regions of the first image by determining a statistical measure of the matching regions in the second image that match the regions in the first image.

[0015] According to the fifth aspect, in addition to the second or third aspect, generating the first set of annotations includes: i) acquiring multiple regions in the second image; ii) for each of the multiple regions in the second image: a) calculating a statistical metric, and b) associating the calculated statistical metric as an annotation in the first set of annotations with the location of the region in the first image that matches the region in the second image.

[0016] According to the sixth aspect, except for the fourth or fifth aspect, each region is a rectangle, square, or circle at a pre-configured position in the first image.

[0017] According to the seventh aspect, in addition to any one of the fourth to sixth aspects, the first set of annotations includes: the statistical measure associated with the location of the corresponding region in the first image and / or associated with the location of the corresponding matching region in the second image.

[0018] According to the eighth aspect, except for any one of the fourth to seventh aspects, at least one of the plurality of regions is a closed region selected in the first image.

[0019] According to the ninth aspect, in addition to any one of the first to eighth aspects, generating the first set of annotations includes: inputting the second image into a second process; and obtaining the output of the second process as the first set of annotations.

[0020] According to the tenth aspect, except for any one of the first to ninth aspects, the first image is an image of a biological sample stained with IHC, H&E and FISH.

[0021] According to the eleventh aspect, in addition to any one of the first to tenth aspects, the method further includes: i) acquiring a second image stained with an antibody / antigen at a first concentration using a quantitative method for converting antibody / antigen complexes into dots; ii) acquiring a third image of a biological sample, the third image being stained with an antibody / antigen at a second concentration different from the first concentration using a quantitative method for converting antibody / antigen complexes into dots, wherein the second image and the third image are images of respective slides of sequential sections of the biological sample; iii) generating a second set of annotations for the first image based on the third image; and iv) outputting the first image and a baseline truth including the second set of annotations as data for training a first ML model to analyze the biological sample.

[0022] According to aspect 12, except for any one of aspects 1 through 11, the staining method is IHC staining for visualizing HER2 epitopes in formalin-fixed paraffin-embedded tissues.

[0023] According to the thirteenth aspect, a computerized method for training an ML model is provided, the method comprising: inputting a first image and a benchmark truth output by a method for generating training data according to any one of the first to twelfth aspects into the ML model; and modifying at least one parameter of the machine learning model based on the input first image and benchmark truth.

[0024] According to the fourteenth aspect, in addition to the thirteenth aspect, the method further includes: iteratively executing the method for generating training data according to any one of the first to twelfth aspects, as well as the steps of inputting and modifying.

[0025] According to the fifteenth aspect, an analysis method is provided, comprising: inputting a fourth image of a biological sample or a first image of a second biological sample into an ML model trained using the method according to the eleventh aspect; and obtaining an analysis of the biological sample or the second biological sample from the output of the ML model.

[0026] According to aspect sixteen, except for aspect fifteen, the results of the analysis indicate the presence, absence, or amount of antigen in the biological sample or the second biological sample.

[0027] According to aspect seventeen, except for aspect fifteen or sixteen, this analysis is a classification of biological samples or second biological samples.

[0028] According to the eighteenth aspect, in addition to any one of the first to seventeenth aspects, the ML model is a first ML model, and the generation of the first set of annotations includes: i) detecting objects in an image of a biological sample, the detection including: a) acquiring an image of the biological sample stained by a quantitative method of converting antibody / antigen complexes into points, and b) detecting points in the image of the biological sample using a trained second ML model, the second ML model being pre-trained to detect points; ii) inputting a second image into a second processing; and iii) acquiring the output of the second processing as the first set of annotations.

[0029] According to the nineteenth aspect, in addition to the eighteenth aspect, the detection also includes: generating an image of the biological sample and one or more annotations associated with the points detected in the detection step.

[0030] According to aspect 20, in addition to aspect 18 or 19, the one or more markings indicate the location of the corresponding one or more detected points.

[0031] According to aspect 21, apart from aspect 20, the position of a point is its position in three-dimensional space.

[0032] According to aspect 22, in addition to any one of aspects 19 to 21, the detection further includes: a) providing an input interface that allows a human user to obtain one or more updated labels by: i) deleting a label from one or more labels, ii) modifying a label in one or more labels, and / or iii) adding a label to one or more labels; and b) storing one or more updated labels.

[0033] According to aspect 23, in addition to aspect 22, the method further includes: training the second ML model or another AI model for detection points using one or more updated labels as baseline truth.

[0034] According to aspect 24, in addition to any one of aspects 19 to 23, the method further includes: providing a label viewing interface configured to display the image of the biological sample together with the one or more labels.

[0035] According to aspect 25, in addition to aspect 24, the annotation viewing interface is a side-by-side viewing interface configured to display a first view and an adjacent second view, the first view showing the image based on the biological sample and including the first view image of the one or more annotations superimposed, and the second view showing a second view image of the biological sample.

[0036] According to aspect twenty-six, apart from aspect twenty-five, the second view was acquired in a different setting than the image of the biological sample and includes annotations corresponding to one or more annotations of the first view image.

[0037] According to aspect 27, except for either aspect 25 or 26, the second view image of the biological sample is based on or includes a z-stack, which comprises two or more focal plane images of the same field of view (FOV) of the biological sample that are different from each other.

[0038] According to the twenty-eighth aspect, in addition to the nineteenth aspect, the second view image is obtained by tiling or 3D deconvolution of a z-axis stack, including combining two or more focal plane images into a single image of a biological sample.

[0039] According to aspect 29, except for any one of aspects 24 to 28, the annotation viewing interface is configured to view each of the one or more annotations as a graph of an unfilled outline with a pre-configured shape surrounding one of the detected points.

[0040] According to aspect 30, except for aspect 29, the annotation viewing interface is configured to enable switching between focal planes of the z-axis stack.

[0041] According to aspect thirty-one, in addition to aspect twenty-nine or thirty, the annotation viewing interface is configured to: i) enable the user to select a focal plane from the z-axis stack and mark a point on the selected focal plane, and ii) store the identifier associated with the selected focal plane and the marked point.

[0042] According to aspect thirty-two, except for any one of aspects twenty-nine to thirty-one, the shape is a rectangle or a square, which is positioned such that one of the detected points is located at its geometric center.

[0043] According to aspect thirty-three, in addition to any one of aspects twenty-nine to thirty-two, the annotation viewing interface is configured to allow switching between: i) viewing each annotation in the annotation as a point graph co-located with one of the detected points, and ii) viewing each annotation in the annotation as a graph with an unfilled outline having a pre-configured shape surrounding one of the detected points.

[0044] According to aspect thirty-four, except for aspect thirty-three, the switching is triggered by changing the field of view of the image of the biological sample.

[0045] According to aspect thirty-five, in addition to any one of aspects eighteen to thirty-four, the detection point also includes: classifying or screening the one or more points based on additional cellular or non-cellular markers.

[0046] According to aspect thirty-six, in addition to any one of aspects one through thirty-five, the detection point further includes: classifying or screening the one or more points based on one or more additional images of the biological sample, the additional one or more images including one or more images of the same slice or consecutive slices.

[0047] According to aspect thirty-seven, in addition to any one of aspects eighteen to thirty-six, the method further includes: training the second ML model to detect objects in an image of a biological sample, the training comprising the steps of: a) acquiring an image of the biological sample stained by a quantitative method of converting antibody / antigen complexes into points; b) acquiring the image of the biological sample and one or more annotations associated with the points; and c) adjusting one or more parameters of the second ML model based on the image of the biological sample and the one or more annotations as a baseline truth input into the AI ​​model.

[0048] According to aspect thirty-eight, in addition to aspect thirty-seven, the training also includes: a) acquiring images of negative control biological samples stained with IHC or fluorescence-based methods; and b) adjusting one or more parameters of the AI ​​model based on the images of the negative control biological samples and unlabeled or unindicating points as a baseline truth.

[0049] According to the thirty-ninth aspect, a computer program is provided, stored on a non-transitory medium, including instructions that, when executed on one or more processors, cause the one or more processors to perform the steps of the method according to any one of the first to thirty-eighth aspects.

[0050] According to a fortieth aspect, a training data generation apparatus is provided, comprising: a data interface; and a processing circuit that, in operation, performs the following operations: i) acquiring a first image of a biological sample via the data interface; ii) generating a first set of annotations for the first image based on a second image of the biological sample stained by a quantitative method, the quantitative method converting antibody / antigen complexes into dots; and iii) outputting the first image and a benchmark truth including the first set of annotations as data for training a first ML model to analyze the biological sample.

[0051] According to aspect 41, methods are provided for determining or estimating antibody expression levels using any analytical or training methods described herein, wherein such methods further include determining the number of points per cell in one or more cells of a biological sample or a second biological sample using a first ML model or a second ML model as described herein, and determining or estimating the antibody expression level in the biological sample or the second biological sample based on the determined number of points per cell. In some aspects, the antibody expression level in the biological sample or the second biological sample is determined or estimated based on the average number of points per cell. In some aspects, the biological sample or the second biological sample comprises formalin-fixed paraffin-embedded cells.

[0052] It should be noted that this disclosure also provides an apparatus for its processing circuitry to perform any of the methods described herein. This disclosure provides an integrated circuit for implementing the processing circuitry described above. Attached Figure Description

[0053] A further understanding of the nature and advantages of the specific embodiments can be achieved by referring to the remainder of the specification and the accompanying drawings, wherein the same reference numerals are used to refer to similar components. In some instances, sub-reference numerals are associated with reference numerals to indicate one of a plurality of similar components. When reference is made to the reference numerals without describing the existing sub-reference numerals, it is intended to refer to all such plurality of similar components.

[0054] Figure 1 The diagram illustrates the acquisition of two slices of a biological sample and the quantitative staining of one of them.

[0055] Figure 2 This is a flowchart of an exemplary method for generating training data.

[0056] Figure 3 This is a flowchart of an exemplary method for utilizing the generated training data.

[0057] Figure 4 This is a schematic diagram illustrating an exemplary implementation of training using the generated training data.

[0058] Figure 5This is a flowchart of an exemplary method for generating region-based annotations.

[0059] Figure 6 This is a schematic diagram of an exemplary view and region of an image.

[0060] Figure 7 It is a view of a set of quantitative staining images, stained with different concentrations of antigen.

[0061] Figure 8 This is a schematic diagram illustrating an exemplary implementation of annotation using heatmaps.

[0062] Figure 9 This is a flowchart of an exemplary method for analyzing biological samples.

[0063] Figure 10 The block diagram illustrates a device that enables the generation of benchmark truth, its use for training or ML models, and the analysis of biological samples using trained ML models.

[0064] Figure 11 The flowchart exemplifies a method for point detection and includes schematic representations of images of labeled and unlabeled biological samples.

[0065] Figure 12 The block diagram illustrates an exemplary device for point detection.

[0066] Figure 13 The flowchart illustrates methods for generating and updating annotations, as well as methods for training artificial intelligence at runtime.

[0067] Figure 14 This is a schematic diagram of a graphical user interface that allows the display of images of biological samples and associated annotations, and also shows some exemplary buttons for updating annotations.

[0068] Figure 15 This is an exemplary partial screenshot of an image of a biological sample stained with qIHC.

[0069] Figure 16 The diagram illustrates a side-by-side view of the graphical user interface.

[0070] Figure 17 The diagram illustrates some possible forms of labeling.

[0071] Figure 18 The exemplary flowchart illustrates the stages related to the application of artificial intelligence.

[0072] Figure 19 An exemplary flowchart illustrates a method for training an artificial intelligence model.

[0073] Figure 20 This is an example of the contents of a memory that stores functional modules for configuring processing circuitry to perform functions of a module.

[0074] Figure 21 An exemplary block diagram shows that includes Figure 2 The system of point detection equipment.

[0075] Figure 22 The diagram illustrates the application of qIHC.

[0076] Figure 23 The graph illustrates the performance of the exemplary implementation of this disclosure, specifically showing a comparison between the qIHC point count in the circular block and the GE051 detailed score.

[0077] Figure 24 The graph illustrates the performance of the exemplary implementation of this disclosure, specifically demonstrating the modeling performance on a validation set with region separation but no organization separation: the graph shows a comparison of the predicted values ​​after averaging the organization on the test set with the baseline truth.

[0078] Figure 25 A set of graphs illustrates the performance of the exemplary implementation of this disclosure, specifically showing (a) the qIHC block count distribution of the training set and the threshold for bin labeling, (b) the confusion matrix of the block predictions on the test set and the baseline truth, (c) the comparison of the predicted values ​​across regions within each organization on the test set with the baseline truth, and (d) the comparison of the predicted values ​​across organizations on the test set with the baseline truth.

[0079] Figure 26 A set of images of formalin-fixed paraffin-embedded cell lines with different HER2 expression levels is provided, with the top row stained with IHC and the bottom row stained with qIHC.

[0080] Figure 27 The graph shows the qIHC points / cells measured in a set of exemplary cell lines with different HER2 expression.

[0081] Figure 28 Images of the same biological sample stained with qIHC are provided, where the detected qIHC points are labeled with dots (left image) and boxes (right image). Detailed Implementation

[0082] This disclosure generally relates to methods, procedures, systems, and apparatuses for facilitating digital microscopy imaging (e.g., digital pathology or live-cell imaging). More specifically, this disclosure relates to achieving digital microscopy imaging using quantitative staining methods that convert antibody / antigen complexes into dots, and using these dots as quantitative measures to generate a baseline truth for analyzing other stained or unstained portions.

[0083] This disclosure generally relates to methods and apparatus for use, for example, in multiplex assays or in sequential slices that can be observed together to detect target molecules or other parts of biological tissue. Such methods have broad applicability in diagnostic applications, selecting appropriate therapies for individual patients, or training neural networks or developing algorithms for such diagnostic applications or therapies. This disclosure can also facilitate the collection and autonomous annotation of labeled data, and more specifically, facilitate the imaging of biological samples to generate training data for developing deep learning-based models for image analysis, cell classification, target feature recognition, and / or virtual staining of biological samples.

[0084] The following detailed description illustrates several exemplary embodiments in more detail to enable those skilled in the art to practice these embodiments. The described examples are provided for illustrative purposes and are non-limiting and non-exclusive.

[0085] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that other embodiments of this disclosure may be practiced without some of these specific details. In other instances, certain structures and devices are illustrated in block diagram form. Several embodiments are described herein, and while different embodiments are given different features, it should be understood that features described with respect to one embodiment may also be incorporated into other embodiments. However, similarly, no single feature or multiple features of any described embodiment should be considered essential to every embodiment described and / or claimed herein, as such features may be omitted in other embodiments.

[0086] In clinical pathology, particularly digital pathology, the detection of proteins in intact formalin-fixed paraffin-embedded tissue can be performed using semi-quantitative immunohistochemical methods. For example, a recent approach has proposed quantitative immunohistochemistry (“qIHC”), which allows for the direct quantification of proteins, for instance, in formalin-fixed paraffin-embedded tissue by counting spots. qIHC can be combined with standard immunohistochemistry and evaluated using standard bright-field microscopy or image analysis. The qIHC method is similar in principle to classical IHC, and like classical immunohistochemistry, its amplification is based on enzyme deposition (typically horseradish peroxidase (“HRP”). ​​The primary antibody binds to the target (protein / receptor). The HRP-labeled secondary antibody recognizes and binds to the primary antibody. These two initial steps in the qIHC reaction are directly comparable to standard IHC. However, in qIHC, only a predetermined proportion of the secondary antibody is labeled. The labeled secondary antibody is mixed with unlabeled antibody to improve the robustness of the detection. Finally, the amplification reaction generates spots centered on the labeled antibody, i.e., directly at a single target site. The number of spots can be counted, and since the ratio between labeled and unlabeled secondary antibodies is known, qIHC assays allow for a direct correlation between the number of spots and the amount of biomarkers present in the tissue.

[0087] However, quantitative methods are not limited to qIHC; they can also be applied to ISH methods for DNA or RNA, imaging mass cytometry, or immunofluorescence-based methods.

[0088] Some embodiments of this document provide methods and apparatus for generating accurate quantitative baseline truths for, for example, IHC-stained slides using qIHC points (or points obtained by similar methods).

[0089] Currently, the leading solution for generating benchmark truth for interpreting and scoring tissue sections with various staining methods (e.g., IHC and H&E) remains manual annotation. However, manual annotation is tedious, time-consuming, and, more importantly, subjective, thus requiring consensus scoring by a committee of pathologists in many cases. Other existing solutions exist, such as methods for fluorescent IHC on serial sections. Therefore, while using fluorescence as an objective chemical “annotation” layer to provide a quantitative signal, it primarily involves comparing signals and requires guessing / calibrating the true expression of the antibody being tested. Other methods use molecular analysis / measurement to obtain accurate quantitative estimates; however, these methods may lack spatial resolution and are generally more complex and expensive.

[0090] Based on the truth of quantitative staining generation benchmark Therefore, one of the problems addressed by this disclosure is the challenging task of collecting large-scale, accurate benchmark truth data to train computational statistical models (e.g., machine learning or AI models), such as in the interpretation of whole-slice pathological images. Achieving reliable benchmark truth is even more challenging when processing medical data and requires highly trained personnel. Points (e.g., qIHC points) provide chemically based signals of protein expression, rather than based on human interpretations of staining that are known to be biased and / or noisy in some cases.

[0091] A trained AI model for IHC staining interpretation based on qIHC benchmark truth can achieve IHC staining quantification beyond human visual perception. This approach may allow for the computational alignment of aligned qIHC WSIs with test WSIs, projecting a measure of the point distribution onto the test WSI as a benchmark truth label, thereby generating large-scale, quantitative, and objective benchmark truth labels. This, in turn, can lead to the development of more accurate computational AI models applied to WSIs. For example, the computational model can be implemented as a classification / regression model for predefined regions within the WSI. In other scenarios, the computational model can be used as a segmentation model for desired tissue structures.

[0092] A trained AI model for IHC staining interpretation, based on the qIHC benchmark truth, can achieve IHC staining quantification beyond human visual perception. This method is not limited to qIHC but can be applied to any other quantitative staining method.

[0093] In this disclosure, non-limiting examples of solving the above problems include those based on, for example... Figure 1 The methods and apparatus shown illustrate quantitative staining to generate benchmark truth annotations (e.g., large-scale annotations of whole-slice images (WSI) of pathologically stained tissue slides). This benchmark truth supports the development and / or improvement of AI-based methods (or, in general, ML-based methods) for quantitative estimation of protein expression in tissues stained by routine IHC or H&E, or for other applications.

[0094] like Figure 1 As shown, biological sample 110 is sliced ​​to obtain two or more consecutive slices of the same tissue. In this example, the biological sample is a block of breast tissue. Therefore, two or more consecutive tissue slices are first cut from the tissue block. These tissue blocks can then be processed into, for example, formalin-fixed paraffin-embedded slices. Here, the first slice is clinical tissue slice 120, and the second slice 130 is a tissue slice used to generate one or more labeled sections.

[0095] One slide (e.g., the first slide) can be stained with the desired IHC staining, and another slide (e.g., the second slide) can be stained with qIHC against the antibody to be tested. Figure 1In this study, two sections, 120 and 130, were excised from a block containing very low levels of human epidermal growth factor receptor 2 (HER2) protein expression. Section 120 was stained with membrane-bound HER2 staining (GE001), and almost no stained expression was detected. Section 130 was stained with quantitative IHC (qIHC). The number of qIHC spots in the second section was used as a baseline truth label for HER2 expression in the matched region of the GE001-stained tissue. Figure 1 The upper right shows two examples, 140 and 150, of the first section of tissue stained with GE001. The tumor area (corresponding to the biological sample) harvested here showed no visible staining. Figure 1 The lower right side shows two corresponding examples 160 and 170 of a second section of tissue stained with qIHC. The number of spots is proportional to HER2 expression and can be used as a baseline truth for the corresponding images 140 and 150.

[0096] It is important to note that qIHC staining and / or IHC staining can be performed on more than one consecutive tissue section. Using more than one consecutive tissue section to generate additional baseline truth (if qIHC staining is used) or to generate other relevant inputs (if IHC staining is used, or unstained, or other staining is performed) can improve overall accuracy.

[0097] According to the embodiments, as follows are provided Figure 2 The method 200 shown is for generating training data. Method 200 includes step 210: acquiring a first image of a biological sample. Such a first image can be, for example... Figure 1 The clinical tissue section 120 shown is shown. It should be noted that the first image can be any image of the biological sample (stained or unstained).

[0098] Method 200 further includes step 220: generating a first set of annotations for the first image based on a second image of the biological sample stained by a quantitative method that converts the antibody / antigen complex into dots. The first set of annotations includes one or more annotations. As described below, the format of the annotations has various possibilities. For example, the second image itself can serve as a baseline truth (annotation) for the first image. On the other hand, dots obtained after applying detection to the second image can serve as annotations, or a count of dots or any statistical measure of dots can also be used.

[0099] The second image can be an image of the same biological sample slice as the first image. For example, a biological sample is collected to produce the first image, then stained using a quantitative method and collected again to produce the second image. Alternatively or alternatively, a biological sample can be stained and collected to produce the first image, then stained again using a quantitative method and collected again to produce the second image. Here, the term "alternatively or alternatively" means that the same slice can be stained using different quantitative or non-quantitative methods (e.g., sequentially) to obtain more than one image.

[0100] However, the first and second images can alternatively be as follows: Figure 1 Images of two corresponding sections of the same biological sample are shown. Specifically, one section may be unstained, while the other is stained using a quantitative method. Alternatively or additionally, one section may be stained using a first method, and the other using a second method. The first method may be non-quantitative, and the second method may be quantitative. Here, the term "alternatively or additionally" indicates that more than one section may be stained using different quantitative or non-quantitative methods.

[0101] Quantitative methods for converting antibody / antigen complexes into dots can include, for example, qIHC (see below). Figure 22 (For example). However, this disclosure is not limited to qIHC staining, and any other quantitative staining method known in the art may be used.

[0102] Method 200 further includes step 230: outputting a first image and a benchmark truth including the first set of annotations as data for training a first ML model to analyze biological samples.

[0103] Therefore, step 230 outputs a training data pair of the input image and the benchmark truth. The first ML model can be any type of machine learning model, such as AI, which can be formed by one or more layers of a neural network (e.g., a deep network). For image processing purposes, a convolutional neural network is suitable as an example. For example, ResNet or any other type of classification architecture can be used. However, this disclosure is not limited to convolutional networks or networks that include convolutional layers. Generally, any neural network can be used, such as a multilayer perceptron, etc.

[0104] The analysis of biological samples described above can be, for example, the classification of input images. For instance, the output of the first ML model could be a category specifying whether (or with what probability) the analyzed biological sample contains certain objects or structures, such as tumor cells. However, this disclosure is not limited to this; the first ML model is not necessarily a classification model that categorizes the input image into some discrete groups. Instead, the first ML model could be a regression model. For example, when using qIHC, since it is quantitative, the first ML model can be trained to estimate the model beyond classification into discrete groups. Instead, it can output the position on the gradient. In this way, for example, different cancer grades forming a continuous spectrum rather than discrete groups might be more suitable for analysis.

[0105] It should be noted that this disclosure is not limited to applications in cancer cell analysis or digital pathology. For example, any other type of immune disease can be analyzed based on the corresponding biological samples.

[0106] According to the example, step 220, which generates the first set of annotations, is based on statistical measures of points in the second image. For example, these statistical measures include the number of points and / or the density of points. To obtain such measures, in some exemplary implementations, points (e.g., their respective locations) can be obtained through automated processing to identify them. The identified points can then be counted to obtain their number or other measures (such as density).

[0107] However, this disclosure is not limited to these metrics and does not necessarily involve the counting of points. For example, it is conceivable to use the (second) image of quantitative staining directly as a baseline truth without prior detection or identification of points. Furthermore, it is conceivable that, especially where the point staining has a unique color different from the primary color of the biological sample, a color histogram can indicate the presence and / or quantity of points.

[0108] The results of method 200 can be further used for training. This is in Figure 3 As shown in the image. Specifically, Figure 3 A computerized method 300 for training a first ML model is illustrated. This method includes inputting 310 into the first ML model via the reference above. Figure 2 The method 200 outputs a first image and a benchmark truth. The training method 300 further includes step 320: modifying at least one parameter of the first ML model based on the input first image and benchmark truth.

[0109] In an exemplary implementation, method 300 includes iteratively executing the steps of method 200 for generating training data, as well as the input 310 and modification 320. This is in Figure 3This is illustrated by the dashed arrows indicating the return from step 320 to step 200. In other words, the generation of the baseline truth can be performed "at runtime," for example, during training (interleaved with training). However, it is important to note that training does not need to be performed at runtime, or even on an instance of the first ML model used for inference (analysis of biological samples). For example, the first ML model can be trained during the training phase, prior to the inference phase where the ML model is used to analyze biological samples. However, the instance of the first ML model used for inference does not need to be trained during the training phase at all. For example, one or more parameters of the first ML model can be obtained from another source, such as from the training results of another instance of the ML model (e.g., a model with a similar structure / architecture including corresponding adjustable parameters).

[0110] In the exemplary implementation, Figure 4 The diagram schematically illustrates the training process and the actions that might be taken before training. The first ML model in... Figure 4 The diagram exemplarily illustrates a neural network 470 with adjustable weights. In this specific example, a first ML model 470 is configured for HER2 expression quantification. This configuration is obtained through training (either during runtime, with pre-trained ML model 470, or by copying trained weights from another ML model similar to the first ML model 470). The first ML model 470 can be used to quantify HER2 expression over a very low range, which is one of the advantages of this disclosure. The first ML model 470 outputs 480 in the range of 0 to 1, indicating HER2 quantification, e.g., how much HER2 is present in the input image and / or in different regions of the input image. Figure 4 The application of HER2 in this study is merely exemplary and does not limit this disclosure. Furthermore, it should be noted that the above ranges are only exemplary, as HER2 expression can range from 0 to a score of 1 according to the exemplary clinical protocol. However, any other ranges may be used.

[0111] The inputs to the first ML model 470 are the annotations 440 representing the baseline truth and the input image 460 (e.g., corresponding to the reference image 460). Figure 1 and Figure 2 (The first image acquired). Therefore, based on these inputs, the weights of the ML model 470 can be adjusted. The output 480 can be used to test the quality of inference during the training phase. During the inference phase, it represents the results of the analysis of the biological sample.

[0112] Region-based labeling A single label 440 can be set for the first image. For example, this could be the number of points within the first image. Alternatively, it may be advantageous to associate different regions of the first image with their own independent labels (i.e., labels determined separately for the matching regions). It should be noted that the size and shape of the regions in the first image and the matching regions in the second image are not necessarily the same.

[0113] For example, step 220 of generating the first set of annotations includes generating annotations for each of a plurality of regions in the first image, achieved by determining a statistical measure of matching regions in the second image that correspond to the regions in the first image. The regions here are not limited to any particular shape, size, or method of obtaining the regions.

[0114] To obtain a matching region, an alignment of 450 can be applied to the region. This alignment can be global alignment and / or local alignment. Global alignment refers to the alignment of the entire region or image. Local alignment aligns a region and / or a portion of the image. This disclosure is not limited to any particular alignment method. Region matching can be performed by finding a match that minimizes a certain metric (e.g., the sum of absolute differences, etc.). In other words, alignment can be performed automatically. Additionally or alternatively, alignment can be performed manually or assisted by a human using a graphical user interface (“GUI”).

[0115] like Figure 5 As shown, for example, step 220 of generating the first set of annotations includes step 510: obtaining multiple regions in the second image.

[0116] according to Figure 5 Step 220 further includes: for each i-th region in the plurality of regions in the second image: calculating a statistical metric 520; and associating the calculated statistical metric as a label in a first set of annotations with the location of the region in the first image that matches the region in the second image 530.

[0117] In other words, the method performs a loop, determining the i-th label (or multiple labels) of a region and associating it with the i-th region in each i-th loop iteration. The loop iteration can begin with the first region among the multiple regions (e.g., an iteration of i=1 (or i=0, which is simply a region numbering convention)) and end with the last region among the multiple regions. At the end (or beginning) of each loop iteration, i is incremented, as shown in step 540. In other words, the loop repeats steps 520, 530, and 540, with i continuously increasing until all regions are associated with their respective labels. For example, the first set of labels includes the statistical measure associated with the location of the corresponding region within the first image and / or with the location of the corresponding matching region on the second image.

[0118] For example, each region is a rectangle, square, or circle at a pre-configured location in the first image. Figure 6 Such an area is shown. Figure 6 In this process, the number of qIHC points located within circle 620 (exemplary radius = 256 pixels) is calculated. Therefore, labels are generated for the regions in the first image corresponding to the region 620 in the second image where points are counted.

[0119] Figure 6 The first view 610 of the GUI is shown, in which the region for calculating points is a circle 620. Therefore, the position of this region can be given by the center of the circle. Figure 6 A second view 650 of the GUI is also shown, in which the area for calculating points is either a circle 660 or a square 670. (See also...) Figure 4 It is also shown that, in an exemplary implementation, each region of the qIHC (second) image is used to generate a label for its corresponding block in the IHC-stained (first) image.

[0120] Typically, the shape of the area is not limited to a circle, square, or rectangle. These shapes are easy to handle and provide simple reference points for their position (e.g., the center of a circle, the center of a square or rectangle, the top left corner of a square or rectangle, or another corner of a square or rectangle, etc.).

[0121] Typically, regions can have any shape or size. In the example, at least one of the multiple regions is a closed region selected in the first image. Such a closed region can be enclosed, for example, by a polygon or a hand-drawn outline.

[0122] Figure 5 Step 510, which involves obtaining a region, may include automatically and / or manually determining the region. For example, the first image may be segmented into regions of equal size, and then labels may be determined for matching regions in the second image. In this example, regions of square or rectangular shapes (such as...) are used. Figure 6 (As shown in Figure 660) is likely computationally efficient. In this way, non-overlapping regions covering the entire first image can be defined, along with annotations for each region.

[0123] However, it should be noted that this disclosure is not limited to non-overlapping regions. Typically, regions can overlap. In the example, the region is circular (e.g., ...). Figure 6 (As shown in 620 and 670). Circles can overlap, covering the entire first image. However, non-overlapping regions can also be defined.

[0124] In the example above, the first image was segmented into regions, and labels were determined in the second image for regions that matched those in the first image. However, this disclosure is not limited to this approach. The second image could be segmented into regions, these regions labeled, and then the labels assigned to matching regions in the first image.

[0125] The size of the region can be set by the user (e.g., the GUI can provide the user with corresponding input possibilities) or determined automatically, for example, based on the image or field of view (FOV) resolution.

[0126] The regions into which an image is segmented can all be the same size, or they can be different sizes. For example, it's conceivable to determine the region size based on the density of points. For instance, the lower the density, the larger the circle.

[0127] It is important to note that the regions do not necessarily cover the entire image. These regions can be user-defined or automatically defined target regions that are labeled, while the rest of the image remains unlabeled. For example, target regions can have associated semantics, such as ducts or non-tumor components.

[0128] When referring to an image, as mentioned above, it can refer to either the first image or the second image, because segmentation can be performed on either the first or second image. It should also be noted that when performing... Figure 4 In the alignment case shown in step 450, the same type of segmentation can be applied to the first image and the second image, and it is assumed that the co-located regions in the first image and the second image correspond to each other, that is, they match each other in the first image and the second image.

[0129] Typically, different partitions / covers (segments) can be applied to the input image (first image) and the baseline truth (second image). For example, the input can be a rectangular or square block (due to how the image is represented in memory), but the baseline truth can be... Figure 6 As shown in view 650, calculations are performed on a circle centered on a square. For example, the first image can be divided into square regions 670. Then, in the second image, matching circular regions (620, 670) are determined. These circular regions can surround, be surrounded by, or be centered on a square region but have the same area as the square region (circles of equal area). The latter is shown in view 650. The circles of equal area will lie between the circumcircle and incircle of the square.

[0130] As mentioned above, the image is not necessarily covered by regions / labels. Alternatively or additionally, the GUI can provide the user with the possibility of drawing or indicating regions, and then automatically obtain the labels for those regions. For example, the GUI can provide the user with one or more of the following functions: drawing a freehand outline (enclosing a region), drawing a polygon outline, drawing a circle / square / rectangle at a user-defined location and size, etc. After the user draws the polygon (in a view of the first or second image), method 200 can be applied, which calculates the labels, for example, counting the number of points located within the user-drawn region, and displays the count to the user in the GUI. It should be noted that, as mentioned above, the labels are not necessarily point counts.

[0131] Back Figure 4 As an example, it's important to note that WSI can first be divided into blocks, and then the first and second images can be processed as described above. For example, in... Figure 4 In this context, input block 460 consists of multiple blocks from the same WSI, corresponding to the first image. Each block can be processed as the first image as described above (independent of other blocks). Label generation is performed in box 440. Correspondingly, the second image can also take the form of multiple blocks 410, stained using a quantitative method (qIHC, as an example here).

[0132] The generation of the first set of annotations 220 may include: inputting the second image into a second process; and obtaining the output of the second process as the first set of annotations. The second process can be any type of processing. For example, it may include automatic point detection (recognition). For example, boxes 420 and 430 represent specific exemplary methods that can automatically detect points in the second image and will be described in more detail later (see, for example, the following "..."). Point detection in stained images ("part"). The detected points can then be used to calculate one or more labels in box 440.

[0133] However, this disclosure is not limited to such MLM-based methods. Instead, the second processing can be any type of processing, including point detection using algorithms such as pattern matching or feature extraction or similar methods.

[0134] It should be noted that the second processing does not limit this disclosure. The above embodiments and exemplary implementations can also work without the second processing, for example, by directly inputting the point-colored image 410, etc. Furthermore, annotation generation can be assisted or performed by a human user; for example, qIHC images can typically be manually graded.

[0135] In short, Figure 4In this process, the WSIs of slices 410 / 430 and 460 are aligned 450 and used in an end-to-end pipeline to train a first AI model 470 to score the (first) tissue slice 460 to be examined, with its labels generated based on a metric of qIHC points. Here, HER2 qIHC-stained slice 410 is used to generate benchmark truth labels 440 by detecting and counting qIHC points in a predefined region 420 / 430. The qIHC “labels” along with the corresponding image patches from the (aligned) tissue slice 460 to be examined are used to train the HER2 scoring model 470.

[0136] It is important to note that the selection of region locations and the calculation of annotations can be performed once for all locations at the start of the training process. Alternatively, annotations can be calculated for a location each time it is sampled during training (which may in principle be more than once).

[0137] exist Figure 4 In specific examples, similar to Figure 1 The first image was stained with HercepTest™ mAb pharmDx (DakoOmnis), a semi-quantitative immunohistochemical assay based on a rabbit monoclonal primary antibody (clone DG44) and assay-specific visualization reagents. This assay identified HER2 protein overexpression in histologically evaluated formalin-fixed paraffin-embedded (FFPE) breast cancer tissue.

[0138] The first image can be an image of a biological sample stained with immunohistochemistry (IHC), hematoxylin and eosin (H&E), fluorescence in situ hybridization (FISH), or any other staining method (such as phase contrast, differential interference contrast, dark field). It can even be unstained.

[0139] As mentioned above, in Figure 1 and 4 In the example, the staining method is based on HER2 IHC staining. However, this disclosure is not limited to this, and any staining method for visualizing a specific antibody or molecule in formalin-fixed paraffin-embedded tissue may be applied. FFPE is also not limiting, depending on the type of biological sample, and other fixation and embedding methods may be used instead.

[0140] Labels for staining at different concentrations The performance of biological sample analysis may depend on the density of qIHC spots, which in turn depends on the labeled secondary antibody (see [link to analysis]). Figure 22 The concentration levels of (2230) in the image. Therefore, when generating baseline truth (e.g., annotation), more than one image of the biological sample can be considered, acquiring biological samples stained with different concentrations of labeled secondary antibodies / antigens. For example, the concentration can be selected based on the desired dynamic range of target protein expression.

[0141] Figure 7 Three different qIHC dot densities corresponding to different antibody concentrations are shown, which have been examined exemplarily: 2 pM, 4 pM, and 8 pM for labeled secondary antibody (2230), respectively, in their respective views (a), (b), and (c). See above reference. Figure 1 and 4 In the example, a concentration of 4 pM was chosen to obtain a reliable baseline truth label within the ultra-low expression range. For cases requiring a wider dynamic range, several sequential sections stained with qIHC can be used.

[0142] According to an exemplary implementation, a method for generating a baseline truth includes: acquiring a second image obtained by a quantitative method of converting an antibody / antigen complex into a dot and staining with a first concentration of labeled antibody / antigen (e.g., qIHC reagent, such as labeled secondary antibody); and acquiring a third image of a biological sample stained with a second concentration of qIHC, different from the first concentration, wherein the second and third images are images of corresponding slides of sequential sections of the biological sample.

[0143] Then, a second set of annotations is generated for the first image based on the third image. The first image and the baseline truth including the second set of annotations are output as data for training a first ML model to analyze biological samples. The second set of annotations can be used together with the first set of annotations to train the MLM.

[0144] Analysis of biological samples can also involve measuring antibody expression in tissue images as described above, which is sometimes nearly unobservable. Coverage of the desired protein expression range can be achieved by adjusting the concentration of the qIHC reagent. This coverage can then be achieved using a robust number of qIHC spots. Furthermore, when the dynamic range of the antibody is too large, a series of consecutive slices with different qIHC spot concentrations can be used to cover this dynamic range, thereby achieving the desired antibody expression resolution, as described above (second set of annotations). It should be noted that the first and second sets of annotations described above do not limit this disclosure. Typically, more than two different sets of annotations can be associated with the input image (first image) of the biological sample.

[0145] Visualization In practice, it may be necessary to visualize the first image of the labeling, for example, to enable the display of the first image and indications of one or more labels on a GUI. For instance, the point distribution or any derived indices thereof can be overlaid / visualized on the WSI to be tested (the first image) and used as an auxiliary layer for pathology clinicians. Alternatively or additionally, the point distribution or any derived indices thereof can be overlaid / visualized on a second image (e.g., a qIHC image).

[0146] One possible method for visualizing qIHC-based antibody distribution / information is to use a semi-transparent heatmap indicating the local qIHC point count for each pixel. This is in... Figure 8 shown in . Specifically, Figure 8 A first image (e.g., IHC-stained) is shown in view 810, and a second image (e.g., qIHC-stained) is shown in view 820. In view 830, the heatmap is translucently overlaid on the first image. The transparency and / or smoothness of the heatmap can be varied to more easily visualize the number or density of qIHC points.

[0147] However, this disclosure is not limited to this visualization, or may not provide any visualization at all.

[0148] Analyzing biological samples Figure 9 A method 900 for analyzing biological samples is illustrated. Method 900 includes inputting a fourth image of the biological sample to be tested into 910 using the method described above. Figure 3 and Figure 4 The method described above trains an ML model. The method also includes obtaining 920 analyses of the biological samples to be tested from the output of the ML model.

[0149] For example, the analysis results indicate the presence, absence, or amount of antigens in the tested biological sample. In some implementations, this analysis is a classification of the biological sample. However, this disclosure is not limited to such applications. As described above, MLM can be configured (trained or using parameters acquired through training) to output the degree / quantity of a desired antigen or molecule in general, rather than outputting discrete groups or categories.

[0150] Systems and equipment refer to Figure 2 , 3 The methods described in points 5 and 9 can be executed by a device with a corresponding configuration. For example, a training data generation device 1000 is provided, such as... Figure 10 As shown. Device 1000 includes a data interface 1080 and a processing circuit 1010, which performs the following operations during operation: i) acquiring a first image of a biological sample via the data interface; ii) generating a first set of annotations for the first image based on a second image of the biological sample, the biological sample being stained by a quantitative method of converting antibody / antigen complexes into dots; and iii) outputting the first image and a baseline truth including the first set of annotations as data for training a first ML model to analyze the biological sample.

[0151] In addition, device 1010 may also include memory 1020, display control device 1030, communication interface 1040, and operation input interface. Memory 1020 may store programs that, when executed on processing circuitry 1010, perform the steps of any of the methods described above. Processing circuitry 1010 may include one or more processors. However, processing circuitry is not limited to general-purpose or special-purpose processors. It may include programmable hardware, special-purpose hardware, and / or other types of electronic devices. Memory 1020 may include one or more volatile and / or non-volatile memory modules, and may also store training data, first and / or second images and / or annotations, etc. However, it should be noted that this data may also be stored in external storage. Display control 1030 is a module that controls a display device (which may, but is not necessarily, built into device 1000) during operation to display, for example, a user interface suitable for and / or configured to display, for example, a first image and / or a second image and / or annotations. For example, the display may include, as in reference... Figure 6 , Figure 7 or Figure 8 The view described above. This display can show annotations overlaid on the first and / or second images.

[0152] Communication interface 1040 can be used to communicate with external devices (such as storage, servers, or clients) using one or more standardized or proprietary, wireless or wired technologies. For example, communication interface 1040 can implement a protocol stack to access the Internet based on Ethernet, wireless LAN, or cellular communication. It can implement a protocol stack that supports communication via Bluetooth, etc. Operation input interface 1050 enables the user to provide input. For example, operation input interface can be formed by a user interface displayed on the monitor in combination with a touch screen (which can be the same as the aforementioned display device) and / or a mouse, and / or a standard or dedicated keyboard and / or specific operation panels / buttons, etc. Device 1000 can be... Figure 21 It is part of the system described herein, serving as a supplement or replacement for device 1200.

[0153] The embodiments and examples of this disclosure (particularly using qIHC-stained tissue to generate baseline truth for WSI) can provide advantages such as: accurate alignment of IHC-stained WSI and qIHC-stained sequential sections, calibration of existing scoring methods (automatic / manual), development of accurate models for qIHC spot detection, and development of statistical indicators for quantifying qIHC spot distribution analysis (e.g., nuclear density estimation, spatial analysis) (possibly in conjunction with cell distribution).

[0154] Quantitative staining methods can typically be used to train an MLM to analyze samples stained with other methods. This results in more accurate labeling, leading to better training and inference.

[0155] Point detection in stained images As mentioned above, identifying the location of points in the second image may be advantageous in order to generate annotations (220, 440), as referenced above. Figure 4 (Specifically boxes 420 and 430) are briefly shown.

[0156] Point detection can be performed in any manner, such as with or without human assistance, which can be inefficient. Classical computer vision algorithms can be used to detect points, for example, based on color differences, morphology, shape, and size. However, these methods may have some limitations. For example, they may not be robust enough to variations in staining quality, color, and the irregular shape and / or size of the points. Furthermore, these methods are not easily tuned to avoid or at least reduce false positives caused by point-like structures that may exist in biological samples. For example, in biological tissues, objects or structures may look similar to points but are not necessarily points; these objects could be cell nucleoli, etc.

[0157] To provide a more efficient method, Figure 11 In the illustrated embodiment, an image of a biological sample 1110 (i.e., the second image described above) was acquired, which was stained using a quantitative method that converts antibodies or antigens into dots. Figure 11 The image 1160 shown is an exemplary image of a biological sample, which includes multiple points, one of which is marked by arrow 1150.

[0158] Then, a second machine learning model (MLM) is used to detect points in image 1160. To be able to detect points, the second MLM is pre-trained to detect points. Therefore, point detection is performed as follows: image 1160 is input to the second MLM at 1120, and the detection results (i.e., the locations of the detected points within image 1160) are obtained at 1130. These locations are shown in image 1190 (corresponding to image 1160) by rectangular outlines, such as rectangle 1170 surrounding the location of the point pointed to by arrow 1150 in image 1160. In the following text, when referring to MLM, it refers to the second MLM used for point detection, not the first MLM used for analyzing biological samples.

[0159] In this context, "pre-trained" means that the MLM is configured with one or more parameters acquired during a training phase before the MLM is applied to detection points (inference phase). It is not necessary to perform an actual training phase for each MLM instance used for detection points. Training can be performed once on an MLM instance, and the parameters obtained from that trained MLM instance can then be stored and made available to configure other MLM instances. This disclosure is not limited to any particular MLM parameterization method. As described below, the training phase can even be performed at runtime.

[0160] Quantitative methods for converting antibodies or antigens into dots include using the staining methods described above to convert antigen-antibody complexes into dots, for example, for qIHC. These dots are obtained by amplification through several enzymatically catalytic deposition steps at the primary antibody site bound to the secondary antibody-dextran-HRP polymer.

[0161] Figure 11 Method 1100 is a computerized method for detecting objects in an image (e.g., 1160) of a biological sample. In some embodiments, the biological sample may include, but is not limited to, human tissue samples, animal tissue samples, or plant tissue samples, wherein the target object may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures.

[0162] Compared with classical image processing methods, Figure 11 Some advantages of Method 1100 include: more accurate and robust detection due to the use of MLM (e.g., AI-based models). Figure 11 Image 1160 of tissue stained with qIHC is shown, scanned and displayed at 40x magnification. qIHC spots can be viewed as DAB (3,3'-diaminobenzidine) brown spots. Cell nuclei are stained with hematoxylin (actually blue). However, other methods and resulting colors are also possible.

[0163] Figure 11 The computerized method 1100 can be provided by a computerized method 1100 with Figure 12 The exemplary structure shown is implemented by device 1200. Specifically, device 1200 may be a point detection device. Processing circuitry 1210 may perform the reference... Figure 11Steps 1110, 1120, and 1130 are described. Specifically, the acquisition of the biological sample image 1160 can be performed via a processing circuit interface 1280. For example, the processing circuit interface can be an interface to a bus 1290 or another communication medium. Through the processing circuit interface 1280, the processing circuit 1210 can be connected to various other modules of the device 1200, such as a memory 1220, a display control 1230, a communication interface 1240, and an operation input interface 1250. It should be noted that this structure is merely exemplary and illustrative. For example, the bus 1290 can actually be a communication system including one or more buses and / or other communication media. The processing circuit can be any hardware, including one or more processors (e.g., a general-purpose processor) and / or programmable hardware (e.g., an FPGA) and / or special-purpose hardware (e.g., an ASIC) and / or electronic circuits or components. When a processor is included, the memory 1220 can store software capable of (running on the processor) configuring the processing circuit to perform method 1100. The memory 1220 may, for example, store the MLM and / or the trained parameters of the MLM.

[0164] The dot detection device 1200 may also include a display control module (display control) 1230, which controls, during operation, a display device that can be connected to or is part of the device 1200. For example, the display control module 1230 may control the display device to display image 1160 and / or detection results, etc. The communication interface 1240 is an interface for the device 1200 to communicate with other devices. For example, it may include communication interfaces such as Ethernet, WiFi (IEEE 802.11), Bluetooth, cellular communication networks (such as LTE, new radio, etc.), or any other standardized or proprietary communication interface. The operation input interface 1250 is an interface for inputting operation commands. It may be an interface that supports human user input (e.g., via a keyboard, dedicated keys or buttons, mouse, joystick, etc.).

[0165] like Figure 13 As shown, the point detection method 1100 can be used as input to generate 1300 an image 1160 of a biological sample and one or more labels (e.g., 1170) associated with points (e.g., 1150) detected in the detection (step 1120).

[0166] The generation of annotations 1300 typically refers to processing the output of the MLM (Multi-Level Model) to obtain annotations in a form that can be further used by humans or machines. Annotations can be generated by the MLM or by another processing step following the MLM. For example, the one or more annotations indicate the location of one or more corresponding points detected in the detection step. Typically, this location can be labeled by providing the coordinates (e.g., horizontal and vertical spatial coordinates x and y) of the point detected within image 1160. In some exemplary implementations, the location of the point is its location in three-dimensional space. In other words, in addition to the horizontal and vertical spatial coordinates x and y, the location also includes the depth coordinate z.

[0167] Annotations can indicate location by explicitly providing numbers representing coordinates, and / or by providing markers superimposed on image 1160 at or near the location of the detected point. These markers can be hollow rectangles (such as...). Figure 11 The rectangle (1170) can also have other shapes or sizes.

[0168] In summary, one advantage of the point detection proposed in this paper is that MLM can be trained to distinguish between points generated by quantitative methods (such as qIHC) and other point structures or objects in biological samples.

[0169] Update label Notice, Figure 13 The annotations generated in step 1300 can be used in various ways. For example, they can be displayed in 1350 to help human users (e.g., pathologists or other clinical practitioners) and / or stored together with or associated with image 1160.

[0170] However, the annotations can also be used as baseline truth for training the MLM. The annotations can be further refined by 1360 (possibly iteratively) and then used as baseline truth for additional training of the MLM for point detection or for training other MLMs.

[0171] Accordingly, refer to Figure 13 The described method may include the following steps: providing an input interface that enables a human user to obtain one or more updated annotations. This interface may correspond to... Figure 12 The operation input interface 1250 shown allows a human user to update one or more labels in the following ways: i) delete labels from one or more labels, ii) modify labels in one or more labels, and / or iii) add labels to one or more labels; and the updated one or more labels can be associated with and stored with image 1160. They can then be displayed again 1350. This can be achieved by... Figure 12 The display control 1230 of the device 1200 is used to perform the operation.

[0172] Therefore, the annotation process can be used in an improved loop, where the user adjusts the annotations 1100 and 1300 generated by the MLM at 1360. A new MLM can then be trained based on the refined annotations, and this loop can be repeated until the desired model (e.g., with the desired accuracy) is achieved. In other words, the method can include the following steps: storing 1310 the updated annotations (e.g., in memory 1220, or in another storage device possibly located outside of device 1200), and training 1320 the MLM using one or more updated annotations as baseline truth. This on-the-fly additional training can help continuously improve the performance of the MLM. The additionally trained MLM can then be used again to detect 1100 points.

[0173] However, it should be noted that this disclosure is not limited to the implementation that allows for training 1320 in progress. Retraining the MLM model used for inference is not required. The updated annotations can be stored 1310 for other purposes, such as training another MLM for detection points, or simply for viewing.

[0174] Any of the methods and devices mentioned above can also provide an annotation viewing interface configured to display the image of the biological sample together with the one or more annotations. For example, in Figure 12 In the display control 1230, the display can be controlled to view images of biological samples and one or more annotations.

[0175] Figure 14 An exemplary user interface 1400 is illustrated schematically, having an image and annotation display area 1410 and display buttons 1420 (for adding annotations), 1430 (for modifying annotations, such as moving them to a different location), 1440 (for deleting annotations), and 1450 (for saving updated annotations). The image and annotation display area 1410 is an example of an annotation viewing interface. Within the image and annotation display area 1410, as shown below, various elements can be displayed. Figure 15 The image shown is 1500 pixels long and can be overlaid with annotations.

[0176] Figure 14 An example of an input interface formed by buttons 1420 to 1450 is also shown, which support updates to the annotations as described above. In this example, the input interface is a GUI, which includes active areas represented by images of the various buttons, and an input device capable of recording user input. This can be any device, including a computer mouse, touchscreen, touchpad, pen, arrow keys, voice control device, etc. Figure 14Only exemplary user interface buttons for updating annotations are shown. However, this disclosure is not limited to such an interface. Typically, not all four buttons are required. For example, the modify button 1430 may not be necessary, as the same effect can be achieved by deleting annotation 1440 and adding annotation 1420. It should also be noted that the GUI buttons are merely one example of input possibilities. Physical keys (standalone or on a standard keyboard) may be present to associate with the corresponding actions of adding, modifying position, deleting, and / or saving annotations.

[0177] Furthermore, GUI 1400 is not limited to these four annotation update actions. Additional buttons may be present, such as those for triggering retraining of the MLM using the currently displayed labeled image (in GUI area 1410). Alternatively or additionally, settings may exist that allow selection of different types of annotations (e.g., annotations with different sizes and shapes) and / or different views, etc.

[0178] Regarding the view configuration, according to an exemplary implementation, the annotation viewing interface is a side-by-side viewing interface, which is configured to display a first view and an adjacent second view. The first view shows the image based on the biological sample and includes a first view image of the one or more annotations superimposed, and the second view shows a second view image of the biological sample.

[0179] exist Figure 16 The diagram schematically illustrates a side-by-side interface 1600. It includes a first viewing area 1610 and a second viewing area 1620. This arrangement may be particularly ergonomic for human users. The first viewing area 1610 may display an labeled biological sample image 1160 (or an image based on that image), while the second viewing area 1620 may display an unlabeled biological sample image 1160. One advantage of this viewing method is that users can simultaneously see both the labels and the original image without the labels overlapping the content. This may be particularly suitable if points in close proximity exist.

[0180] When referring to a first view image based on a biological sample, it means that the first view image can be the image of the biological sample directly, or it can be an image based on that image. For example, an image based on a biological sample can be an image obtained by filtering or otherwise processing the image of the biological sample (e.g., to generate an image with lower / higher contrast or brightness).

[0181] The second view image may also include an image of the biological sample, or it may include another image taken of the same biological sample. For example, the second view may have been acquired with settings different from those of the image of the biological sample, and may include annotations corresponding to one or more annotations in the first view image.

[0182] In an exemplary implementation, the second view image of the biological sample is based on or includes a z-axis stack comprising two or more different focal plane images of the same field of view (FOV) of the biological sample. Different focal planes refer to focal planes that are distinct from each other. Generally, the term "z-axis stack" imaging refers to acquiring multiple images of a sample (e.g., a biological entity or tissue, or any sample in general) at set intervals between a first and a last focal plane. Thus, each image corresponds to a specific focal length setting.

[0183] In an exemplary implementation, the second view image is obtained by tiling or 3D deconvolution of a z-axis stack, including combining two or more focal plane images into a single image of a biological sample. Tiling or 3D deconvolution can be performed by any known method. For example, for tiling, extended depth of focus (“EDF”) can be used. EDF is an algorithm designed to scan each of a set of images taken at different focal lengths and form a single composite image with all determined to be in focus.

[0184] The advantage of providing a single image based on several images with different focal lengths is that such a single image carries information from the combination of different images, thus providing the user with the possibility of better distinguishing points. However, this disclosure is not limited to displaying a single combined image. In some exemplary implementations, the annotation viewing interface 600 may provide the user with the ability to browse the images of the z-axis stack in the second view 620, and / or provide the ability to select a specific image from the images of different focal planes for viewing.

[0185] Specifically, the annotation viewing interface can be configured to allow switching between various focal planes in the z-axis stack. For example, the annotation viewing interface is configured to: i) allow the user to select a focal plane from the z-axis stack and mark a point on the selected focal plane, and ii) store an identifier associated with the selected focal plane and the marked point. In this way, some depth (z-coordinate) information is obtained.

[0186] Note that in the example described above, it is assumed that the first view image is in viewing area 1610, and the second view image is in viewing area 1620. However, this is merely for illustration. In reality, the first view image can be in the second view area 1620, and the second view image can be in the first view area 1610. Figure 16 The side-by-side view shown is merely illustrative and could be part of a larger GUI that includes other features such as references. Figure 4 The functions described, etc. For example, view 1600 can correspond to... Figure 14Viewing area 1410.

[0187] This disclosure is not limited to displaying images acquired at various focal planes. Additionally or alternatively, images acquired from different angles, by different acquisition devices, or using different settings may be displayed. For example, a second-view image may be a series of slices of the same biological sample, or an image of a biological sample but with different staining, and so on.

[0188] Note that the second view image can also include labels. In the exemplary implementation, labels are presented in two views, 1610 and 1620, and they are linked (related to each other). Linking means that when a label is moved in one view 1610, 1620, it will move accordingly in the other view 1620, 1610. This can improve the accuracy of the labels (because the signal may not always be clear enough in the original view). For example, the link can be established by a pre-step of aligning the images of the two views together (global alignment or block alignment). The GUI can provide the possibility of selecting the "Link Views" feature. If the user selects the Link Views feature, the two image views (one or both of which include labels) will be linked, and if the user moves (pans, rotates, and / or zooms) one view within the viewing area, the other view will move accordingly.

[0189] Typically, whether to display annotations for first-view and / or second-view images can be configured by the user. This is done via a user GUI (e.g., reference). Figure 14 (or those described in 16) may also include a scaling function that allows the user to change the field of view (“FOV”) of the image 1160, for example by zooming in or out, by panning or rotating, etc.

[0190] Show annotations This disclosure is not limited to any particular shape, color, or size of the annotations. In the example, the annotation viewing interface is configured to view each of the one or more annotations as a graph with an unfilled outline having a pre-configured shape surrounding one of the detected points. For example, the shape is a rectangle or square, positioned such that one of the detected points is located at its geometric center.

[0191] exist Figure 17 The image 1700 provides an example of such annotations surrounding the points. Annotations 1710 (rectangle or square), 1720 (circle), or 1730 (rotated rectangle or square) are hollow outlines. It should be noted that the point does not necessarily need to be completely enclosed within the annotation outline. Furthermore, annotations that only point to the point are also conceivable, as exemplified by arrow 1740, or even annotations that at least partially overlap the point and mark its center, such as... Figure 17The crosshair in the image is 1750. The latter allows for more precise visual positioning of points.

[0192] Generally, it may be advantageous to provide annotations such as 1710, 1720, 1730, 1740 that do not touch or overlap with the points they mark. Furthermore, it may be advantageous to provide annotations that do not touch or overlap with any points (even those near the marked point). In special cases, this exemplary display rule can be relaxed. For example, in cases where the point is highly diffuse (out of focus), it may be permissible for the outline (annotation) to overlap with the outer portion of the diffuse point. There may be display rules according to which the central portion of the point is not obscured or overlapped by the annotation or any other annotation. The latter may be more difficult to implement, especially in cases of high point density. If the annotations do not overlap, it may be more ergonomic for human viewers. On the other hand, some overlap will not cause any problems, and such annotations can still allow human users to clearly see the marked points.

[0193] In an exemplary implementation, the annotation viewing interface is configured to allow switching between: i) viewing each annotation in the annotation as a graph co-located with one of the detected points, and ii) viewing each annotation in the annotation as a graph with an unfilled outline having a preconfigured shape surrounding one of the detected points.

[0194] Note that the graphic that co-occupies (and at least partially overlaps) one of the detected points can be, for example, a dotted graphic whose size corresponds to, for example, the size of the detected point, or a fixed size regardless of the size of the detected point. This does not limit the present disclosure: the graphic that co-occupies one of the detected points can have different forms, such as a cross 1750, etc.

[0195] The switching can be triggered automatically or by the user. Regarding automatic triggering, in an exemplary implementation, the switching is triggered by changing the field of view of the image of the biological sample. For example, when the zoom level is below a certain threshold (pre-configured or fixed), co-position (at least partially overlapping) display is applied to the annotations. When the zoom level is above (or equal to) that specific threshold, hollow-shaped annotations can be applied.

[0196] Alternatively or additionally, automatic triggering can be based on the content of the field of view, such as the density of points and / or the proximity of points to each other. For example, for densities above a threshold (pre-configured or fixed), labels that at least partially overlap with and co-locate the points are displayed. For densities below or equal to the threshold, hollow outline labels are displayed.

[0197] Regarding user-triggered annotations, the input interface may provide users with the ability to switch between annotation types (either partially overlapping and co-located annotations, or hollow outline annotations around points). Additionally or alternatively, the input interface may provide users with the ability to switch (configure) between annotation size, color, shape, and / or fill.

[0198] Preprocessing or postprocessing The point detection using a machine learning model (MLM) described above can be enhanced with additional processing steps. For example, the point detection step may further include classifying or filtering the one or more points based on additional cellular or non-cellular markers.

[0199] Cellular markers include, for example, nuclear markers, membrane markers, or cytoplasmic markers. Classification can be performed using MLM (e.g., neural network-based AI or algorithmic methods). When using MLM, the same MLM used for point detection can be used, but it can be trained with additional markers (e.g., to distinguish which cellular structure the point is located in). The markers can also include labels corresponding to point attributes, such as point size, point shape, point intensity, focal plane, etc.

[0200] The classification category can be one or more categories that distinguish the following items: Is the point located within the tumor area? Is the spot located in the normal / matrix area? Is the dot located in the cell nucleus? Whether the point is located on the membrane; and / or Is the spot located near immune cells?

[0201] In other words, points can be classified based on the surrounding biological structures.

[0202] Additionally or alternatively, the step of detecting points may further include classifying or screening the one or more points based on one or more additional images of the biological sample, including one or more images of the same slice or consecutive slices.

[0203] For example, additional staining on consecutive slices (and / or on the same slice) can be used to generate markers. Alternatively or additionally, points can be filtered based on the z-plane of the points; for example, blurred (out-of-focus) points can be excluded from the set of detected (labeled) points.

[0204] (Second) Training of MLM Figure 18 The exemplary flowchart illustrates the various stages in a task using MLM, or more specifically, using AI (such as AI based at least in part on neural network architectures).

[0205] Step 1810 represents the training dataset. In this stage, training images of the stained biological samples and corresponding benchmark truth data are collected. Exemplary benchmark truth data may be a list of coordinates of points in the corresponding training images. However, benchmark truth does not necessarily have to be a list; it can be a 2D image simply labeled with the locations of detected points, or it can be directly a training image with added annotations. This disclosure is not limited to any particular format of benchmark truth data.

[0206] As mentioned above, training data can be obtained by using MLM to detect points and enabling users to update the annotations (the positions of points within the corresponding training images). Training data can also be obtained alternatively or additionally by human users labeling points in the corresponding training images, or by another method capable of labeling points (e.g., any known algorithm specifically developed for point detection, such as feature extraction, pattern matching, etc.).

[0207] Step 1820 represents the training phase. The training phase will be described in more detail below. During the training phase, training images associated with their respective baseline truth data are fed into the MLM so that it can learn (train).

[0208] Step 1830 can be performed, but is not mandatory. Step 1830 is the testing phase. In principle, testing step 1830 corresponds to inference phase 1840. However, the testing phase is performed on training data not used in training and allows the quality of the MLM to be evaluated by testing the output of the MLM against a benchmark truth relative to the training images not used in the training phase.

[0209] Step 1840 represents the inference phase, that is, the phase of applying MLM to new data for which the baseline truth is unknown. In other words, the inference phase is as described above (e.g., refer to...). Figure 11 Point detection.

[0210] If it is possible Figure 18 Seen in, and as referenced Figure 13 Briefly, the inference phase can be supplemented by a human user or another algorithm (such as filtering or additional classification) to modify the MLM's detection results. These modified detection results can then serve as new benchmark truth data for the training images and can be fed back into the training phase. Similar methods can be implemented during the training phase.

[0211] Figure 19Training method 1900 is illustrated. Training method 1900 is a computerized method for training an AI model (or, in general, an MLM) for detecting objects in images of biological samples. The method includes the following steps: acquiring 1910 an image of a biological sample stained using a quantitative method that converts antibody / antigen complexes into dots; acquiring 1920 an image of the biological sample and one or more annotations associated with the dots; and adjusting 1930 one or more parameters of the AI ​​model based on inputting the image of the biological sample and the one or more annotations as a baseline truth into the AI ​​model.

[0212] Adjusting one or more parameters of an AI can include, for example, modifying certain weights and / or biases of a neural network. However, this disclosure is not limited to this. The adjustment can include changing the model itself, or switching to a non-neural network-based machine learning approach, etc.

[0213] Note that the annotations here are not limited to actual displayed annotations, but only represent data (information) associated with the image. For example, such annotations are the locations of points within the image. These locations can be two-dimensional or three-dimensional.

[0214] To further improve training, the method may also include the following steps: acquiring images of negative control biological samples stained with IHC or fluorescence-based methods; and adjusting one or more parameters of the AI ​​model based on the images of the negative control biological samples and unlabeled or unindicating points that the method does not produce as baseline truth.

[0215] Negative controls can be stained using standard qIHC (or a similar method also used for positive biological samples). However, to obtain negative controls, a standard protocol (e.g., qIHC) can be used for quantification, but without the dextran-HRP-ab polymer that produces "true" spots, so that all generated spots will be ghost spots. Specifically, this can be achieved by adding a labeled secondary antibody step ( Figure 12 In step 2230), a buffer solution is used instead of the reagent to generate a negative control. Otherwise, all procedures are identical to the positive reaction. This means that in... Figure 12 The HRP enzyme required for catalyzing substrate precipitation in step 2260 is absent.

[0216] It should be noted that training can be performed in a device similar to the point detector 1200. Specifically, to implement both training 1820 and inference 1840, the processing circuitry 1210 can perform reference operations during operation. Figure 11 , Figure 13 , Figure 18 and / or Figure 19 The steps described. This can be achieved by storing the corresponding program modules in memory 1220, such as... Figure 20 As shown schematically.

[0217] Figure 20 The memory portion 1222 of memory 1220 is shown, which has functional modules dedicated to point detection (module 2010), user input processing (module 2020), GUI operation module (2030), and training (module 2040). It should be noted that not all modules are required.

[0218] This disclosure may only provide a point detector, in which case only the point detection module 1010 will be included. A training module may not be necessary, as the MLM may only be capable of inference and the MLM parameters may be stored statically. These parameters may come from some pre-training performed on a similar MLM (implemented by a different device).

[0219] Device 1200 may implement a user input processing module 2020, which receives user input via an operation input interface 1250 and determines the action to be taken in response to that specific user input. Additionally or alternatively, device 1200 may implement a GUI operation module 2030 for representing the GUI (generating images representing the GUI), and then providing these images to a display device for display via display control 1230. This representation of the GUI may rely on the input processing module 2020 (from which input is received).

[0220] In addition to or as a replacement for the point detection module 2010, the device 1200 may include a training module 2040 for performing the training described above.

[0221] Figure 21 It has been shown that references have been made. Figure 12 The described device 1200 is now part of a system capable of interacting with a human user. Specifically, device 1200 is connected to display device 1260 via its display control interface. Display device 1260 can be an external, stand-alone screen that can be connected to and disconnected from device 1200, or it can be a display device permanently connected to and part of device 1200. Such a display device can be any type of monitor, such as OLED, LCD, etc.

[0222] Furthermore, user input device 1280 can be connected via operation input interface 1250 of device 1200. User input device 1280 can be a keyboard including one or more keys, such as a standard computer keyboard or any type of keyboard. Alternatively or additionally, input device 1280 may include a touchscreen, a mouse, or other means for positioning a cursor at various locations within a graphical user interface displayed, for example, on display device 1260. Communication interface 1240 of device 1200 can be configured to connect device 1200 to network 1295. Figure 21 As shown, the image acquisition device 1270 (e.g., a slide scanner) that serves as the source of the first and second images can also be connected to the network 1295, allowing device 1200 to directly acquire one or more images of the biological sample from the image acquisition device 1270. However, this method of image acquisition is merely exemplary. The images are not necessarily acquired directly from the image acquisition device 1270. They can be acquired from an external storage device via the network 1295. The communication interface 1240 may include a USB interface, or may be a USB interface, thereby enabling image acquisition, for example, from a USB storage device.

[0223] These images can be stored and / or retrieved along with a set of corresponding annotations (e.g., as training data). Alternatively or additionally, as described above, the annotations can be determined by device 1200 and stored along with the images.

[0224] Exemplary model based on qIHC An exemplary embodiment can apply qIHC staining (generating spots). This qIHC staining in... Figure 22 The diagram illustrates (with an exemplary qIHC application, namely the point expressing human epidermal growth factor receptor 2 (HER2)). An example of qIHC application can be found in K. Jensen et al., “A novel quantitative immunohistochemical method for precise protein measurements directly in formalin-fixed paraffin-embedded specimens: analytical performance for measuring HER2,” Mod. Pathol., Vol. 30, No. 2, pp. 180–193, February 2017 (“Jensen 2017”). As with classic immunohistochemistry, its amplification is based on enzyme deposition, typically horseradish peroxidase (“HRP”).

[0225] exist Figure 12In step 1, primary antibody 2210 binds to the target (protein / receptor) 2220. In step 2, the HRP-labeled secondary antibody recognizes and binds to primary antibodies 2230 and 2240. These two initial steps in the qIHC reaction are directly comparable to standard immunohistochemistry. However, in qIHC, only a predetermined proportion of the secondary antibody (e.g., 2230) is labeled with HRP (2250). The labeled secondary antibody 2230 is mixed with unlabeled antibody 2240 to improve the robustness of the assay. In step 3, enzyme substrate 2260 is added and deposition occurs. In step 4, the HRP-labeled antibody 2270 binds to the deposited substrate 2260. Finally, in step 5, the amplification reaction generates a dot 2280 centered on the labeled antibody, i.e., directly at a single target site. The number of dots can be counted, and since the ratio between labeled and unlabeled secondary antibodies is known, qIHC assays allow a direct correlation between the number of dots and the amount of biomarkers present in the tissue.

[0226] Classical computer vision algorithms can be disadvantageous for detecting qIHC points, especially due to the highly localized nature of points in both three-dimensional (3D) and two-dimensional (2D) dimensions. For example, out-of-focus points may appear much larger and more diffuse than in-focus points, making them nearly undetectable using classical image processing methods. Traditional image processing methods are also ineffective at detecting individual points that may appear clustered or seemingly clustered when viewed in a single 2D focal plane but are separated in the third dimension. To overcome these issues, 3D deconvolution can be used, for example, which requires a z-axis stacked 3D scan of the tissue during inference. While point locations can be manually labeled as baseline truth for training AI-based models, accurate labeling is more difficult without using a tiled z-axis stacked scan or a side-by-side view of the focal plane scan, because qIHC points are highly localized in 3D and therefore may appear diffuse and confusing when out of focus.

[0227] Without using perspective annotation, marking point locations can be more difficult because points can often be very small, and filled markers (such as filled circles) or generally overlapping markers (such as crosses 750) can obscure points. Reducing false positives can be achieved by manually marking non-qIHC structures as non-points. This has the disadvantage of being very time-consuming. However, this practice of providing negative annotations (which mark dot-like structures that are not qIHC points or are not points of the desired staining method) can accelerate training and / or improve its accuracy when used as the baseline truth for training data.

[0228] Improving the generation of baseline truth for model training using z-axis stacks (multiple focal planes) scans and / or tiled z-axis stacks or 3D deconvolutional z-axis stacks can provide a more ergonomic experience for human users, including as an auxiliary layer for manual annotators to mark point locations. An iterative annotation process can be provided, in which, in each iteration, qIHC points on the scanned image are pre-annotated using the previously trained model, and human annotators correct the pre-annotations, making changes, additions, or deletions. The corrected annotations are used to train the next generation of detection models. Using negative control stained tissue (parts stained with qIHC but without connection markers) can help efficiently generate baseline truth for reducing false positive detections. Applying the trained model to negative control slices, any points detected by the model are definitely false positives because there are no true points in the slice.

[0229] User interface improvements for accurate and efficient point location labeling can be provided as described above: a perspective annotation mode that facilitates switching between annotations presented as (filled) points and (unfilled) rectangles. Unfilled shapes offer some advantages, as filled markers hide the labeled qIHC points. This can be useful for benchmark truth labeling and point detection for viewing model inference in a synchronized manner. Viewing side-by-side panels showing the labeled image and panels showing the selected z-axis stacked focal plane or tiled z-axis stacked image can further enhance the user interface's character. Spatial relationships between detected qIHC points and the representation of additional markers, including nuclear, membrane, or cytoplasmic markers, as well as non-cellular markers, can be used to classify or filter detected qIHC points.

[0230] In-focal analysis using z-axis stacked scans allows for the localization of detected qIHC points in both the z-axis and 2D, resulting in 3D point localization. This 3D localization enables more accurate assignment of detected points to neighboring cells, as indicated by additional markers. Advantages of the method described herein include improved detection and localization of qIHC points in both 2D and 3D, overcoming the difficulty in point identification due to disordered tissue morphology, staining variations, point aggregation, and off-focality. Furthermore, this disclosure addresses the problem of efficiently and accurately generating labeled data for training AI-based models to detect qIHC points, including generating labeled data to reduce false positives.

[0231] Thanks to user interface improvements and an iterative process aided by additional information based on z-axis stacks, a more efficient and accurate data annotation process can be achieved. These improvements and information make annotations more precise and easier to provide. False positive detections are reduced by applying the trained model to negative controls and utilizing a large number of efficiently generated false positive annotations.

[0232] MLM can be any type of AI or an integration thereof. For example, a model using deep neural networks (such as convolutional neural networks) can be used.

[0233] Methods for detecting qIHC points may include training an AI-based (e.g., neural network) qIHC point detection model, similar to those described in U.S. Patent No. 11,748,881. This utilizes fully convolutional networks (“FCNs”), specifically the so-called U-net architecture, as suggested in O. Ronneberger et al.'s 2015 paper, "U-Net: A Convolutional Network for Biomedical Image Segmentation," published on arXiv:1505.04597. U-net has a shrinking path and an expanding path, giving it a U-shaped architecture. The shrinking path is typical of convolutional networks, consisting of repeatedly applied convolutions, each followed by a rectified linear unit (“ReLU”) and a max-pooling operation. During shrinking, spatial information decreases while feature information increases. The expanding path combines feature and spatial information through a series of up-convolutional operations and concatenation with high-resolution features from the shrinking path. Convolutional neural networks can be applied to image patches of the input image (with a predetermined and potentially configurable size), which, in some applications, can be processed in parallel. However, this disclosure is not limited to this implementation of MLM.

[0234] Summary of implementation examples related to point detection According to a first aspect, a computerized method for detecting objects in an image of a biological sample is provided, the method comprising the steps of: acquiring an image of a biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; and using a trained AI model pre-trained to detect the dots in the image of the biological sample.

[0235] According to the second aspect, in addition to the first aspect, the method further includes the following steps: generating an image of the biological sample and one or more annotations associated with the points detected in the detection step.

[0236] According to the third aspect, in addition to the first or second aspect, the one or more markings indicate the location of the corresponding one or more points detected in the detection step.

[0237] According to the fourth aspect, apart from the third aspect, the position of a point is its position in three-dimensional space.

[0238] According to the fifth aspect, in addition to any one of the second to fourth aspects, the method further includes the steps of: providing an input interface that allows a human user to obtain one or more updated labels by: i) deleting a label from one or more labels, ii) modifying a label in one or more labels, and / or iii) adding a label to one or more labels; and storing the one or more updated labels.

[0239] According to the sixth aspect, in addition to the fifth aspect, the method also includes the following steps: training the AI ​​model or another AI model for detecting points using one or more updated labels as baseline truth.

[0240] According to the seventh aspect, in addition to any one of the second to sixth aspects, the method further includes the step of: providing a label viewing interface configured to display the image of the biological sample together with the one or more labels.

[0241] According to the eighth aspect, in addition to the seventh aspect, the annotation viewing interface is a side-by-side viewing interface, which is configured to display a first view and an adjacent second view, the first view showing the image based on the biological sample and including the first view image of the one or more annotations superimposed, and the second view showing a second view image of the biological sample.

[0242] According to the ninth aspect, in addition to the eighth aspect, the second view was acquired with settings different from those of the image of the biological sample, and includes annotations corresponding to one or more annotations of the first view image.

[0243] According to the tenth aspect, in addition to the eighth or ninth aspect, the second view image of the biological sample is based on or includes a z-axis stack, which includes two or more focal plane images of the same field of view (FOV) of the biological sample that are different from each other.

[0244] According to the eleventh aspect, except for any of the eighth to tenth aspects, the second view image is obtained by tiling or 3D deconvolution of a z-axis stack, including combining two or more focal plane images into a single image of a biological sample.

[0245] According to the twelfth aspect, in addition to any one of the seventh to eleventh aspects, the annotation viewing interface is configured to view each of the one or more annotations as a graph of an unfilled outline with a pre-configured shape surrounding one of the detected points.

[0246] According to the thirteenth aspect, in addition to the twelfth aspect, the pre-configured shape is a rectangle or square, which is positioned such that one of the detected points is located at its geometric center.

[0247] According to aspect fourteen, except for any of aspects seven through thirteen, the annotation viewing interface is configured to enable switching between focal planes of the z-axis stack.

[0248] According to aspect fifteen, except for any one of aspects seven through fourteen, the annotation viewing interface is configured to: i) enable a user to select a focal plane from the z-axis stack and mark a point on the selected focal plane, and ii) store an identifier associated with the selected focal plane and the marked point.

[0249] According to the sixteenth aspect, in addition to any one of the second to fifteenth aspects, the annotation viewing interface is configured to allow switching between: i) viewing each annotation in the annotation as a point graph co-located with one of the detected points, and ii) viewing each annotation in the annotation as a graph with an unfilled outline having a preconfigured shape surrounding one of the detected points.

[0250] According to the seventeenth aspect, in addition to the sixteenth aspect, the switching is triggered by changing the field of view of the image of the biological sample.

[0251] According to aspect eighteen, in addition to any one of aspects one through seventeen, the step of detecting points further includes classifying or screening the one or more points based on additional cellular or non-cellular markers.

[0252] According to the nineteenth aspect, in addition to any one of the first to eighteenth aspects, the step of detecting points further includes: classifying or screening the one or more points based on one or more additional images of the biological sample, the additional one or more images including one or more images of the same slice or consecutive slices.

[0253] According to the twentieth aspect, a computerized method is provided for training an AI model for detecting objects in images of biological samples, the method comprising the steps of: i) acquiring an image of a biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; ii) acquiring an image of the biological sample and one or more annotations associated with the dots; and iii) adjusting one or more parameters of the AI ​​model based on the image of the biological sample and the one or more annotations as a baseline truth input into the AI ​​model.

[0254] According to aspect twenty-one, in addition to aspect twenty, the method further includes: acquiring an image of a negative control biological sample stained with IHC or a fluorescence-based method; and adjusting one or more parameters of the AI ​​model based on the image of the negative control biological sample and unlabeled or unmarked points indicating the truth as a baseline.

[0255] According to aspect 22, in addition to aspect 20 or 21, the one or more markings indicate the location of the corresponding one or more points detected in the detection step.

[0256] According to aspect 23, except for aspect 20 or 22, the position of a point is its position in three-dimensional space.

[0257] According to the twenty-fourth aspect, a computerized method is provided for training a first AI model for detecting objects in an image of a biological sample, the method comprising the steps of: i) acquiring an image of a biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; ii) acquiring an image of the biological sample and one or more annotations associated with the dots, wherein the acquisition includes: a) acquiring an image of the biological sample stained by a quantitative method that converts antibody / antigen complexes into dots, and b) using a trained second AI model pre-trained to detect dots in the image of the biological sample; and iii) adjusting one or more parameters of the first AI model based on the image of the biological sample and the one or more annotations as a baseline truth input into the first AI model.

[0258] According to the twenty-fifth aspect, a computer program is provided, stored on a non-transitory medium, including instructions that, when executed on one or more processors, cause the one or more processors to perform the steps of the method according to any one of the first to twenty-fourth aspects.

[0259] According to the twenty-sixth aspect, a detection device is provided, comprising: a data interface; a storage device; and a processing circuit that, in operation, performs the following operations: i) acquiring, via the data interface, an image of a biological sample stained by a quantitative method that converts antibody / antigen complexes into dots; ii) detecting dots in the image of the biological sample using a trained AI model pre-trained to detect the dots; and iii) storing indications of the location of each detected dot as one or more labels.

[0260] According to aspect 27, in addition to aspect 20 or 26, the detection device also includes an input interface that enables a human user to update one or more labels by: i) deleting a label from one or more labels, ii) modifying a label in one or more labels, and / or iii) adding a label to one or more labels; wherein the processing circuitry stores the updated one or more labels in a storage device during operation.

[0261] According to aspect 28, in addition to aspect 26 or 27, the detection device further includes: a label viewing interface configured to display the image of the biological sample together with the one or more labels.

[0262] According to a twenty-ninth aspect, a training apparatus is provided, comprising: a data interface; a storage device; and processing circuitry, which, in operation, performs the following operations: i) acquiring via the data interface an image of a biological sample stained using a quantitative method that converts antibody / antigen complexes into dots; ii) acquiring via the data interface an image of the biological sample and one or more labels associated with the dots; iii) adjusting one or more parameters of an AI model based on the image of the biological sample and the one or more labels as a baseline truth, thereby training the AI ​​model to detect objects in the image of the biological sample.

[0263] It should be noted that this disclosure also provides an apparatus for its processing circuitry to perform any of the methods described herein. This disclosure provides an integrated circuit for implementing the processing circuitry described above.

[0264] Experimental results Figure 23 The number of qIHC points, used as an estimate of HER2 expression, is compared with a manual score based on membrane-bound HER2 IHC staining. Specifically, Figure 23 The correlation between the point count and the manual estimate of GE051 staining is shown. In particular, tissues are scored only in selected rectangular areas (rather than the entire WSI area).

[0265] The dashed lines represent a linear fit to the data. The points in the figure represent different cases, where a human score is given on one image and the points are counted on a second (continuous) slice. Within a defined region, qIHC points are counted in many (possibly overlapping) circles with a defined radius (here r = 256 pixels), so for each rectangular region, many counts are available, thus providing a standard deviation for each case (as indicated by the vertical lines accompanying the points). It can be seen from the figure that the number of qIHC points is a suitable measure of HER2 antibody expression. Accordingly, Figure 4 A computational flow is presented, where the actual HER2 expression is derived from the qIHC point count (GT). For Figure 23 HER2 expression was manually scored and compared with the point counts of MLM annotations.

[0266] Figure 24 As shown in the reference Figure 4The AI-based HER2 expression estimates (x-axis shows the inference results for MLM 470) are compared with the actual qIHC point counts (y-axis), and these results are summarized for the entire tumor region in each slice.

[0267] Figure 25 The prediction results of the trained qIHC AI model are shown. In (a), the distribution of qIHC points for each block of the model training dataset is shown. The dashed vertical lines represent the threshold of the 10th percentile binning. Figure 25 Part (b) shows the confusion matrix between the true labels and the predicted labels (obtained by the trained AI model from blocks in the test dataset). As can be seen from the figure, the correlation between the predicted and true values ​​is quite significant. Figure 25 Section (c) shows a comparison between the true labels and the predicted labels (which are obtained by the trained AI model predicting blocks in the test dataset and then averaging them over each region in each slice). Figure 25 Part (d) shows a comparison between the true labels and the predicted labels (predicted by a trained AI model on blocks of the test dataset, then averaged over each slice). The labels in (b)-(d) represent quantile bins calculated based on the distribution shown by the red dashed line in (a). The dot sizes in (c) and (d) represent the number of nuclei in each region and the number of nuclei in each slice, respectively. It can be seen that the correlation coefficients of these graphs are above 0.8, indicating that the model is very close to the baseline truth.

[0268] Example Example 1: Evaluation of qIHC detection model on formalin-fixed paraffin-embedded cell lines.

[0269] An experimental study was conducted to evaluate an exemplary qIHC detection model based on this disclosure, using various formalin-fixed paraffin-embedded cell lines. This study demonstrated a linear relationship between HER2 expression levels and qIHC spot counts with qIHC chemical substrate concentrations. As the results of this study show, qIHC can be used as a tunable system where the number of qIHC spots is proportional to the concentration of the spot-generating component. This allows for quantitative HER2 detection at varying protein expression levels with a high signal-to-noise ratio.

[0270] Methods and Results Samples were taken from several formalin-fixed paraffin-embedded cell lines (MDA-MB-468, MDA-MB-231, MDA-MB-175, MDA-MB-453, and SK-BR-3) with different HER2 expression levels (classification range from "0" to "3+") and stained using HercepTest™ mAb (Dako Omnis) or qIHC. Figure 26 The sample preparation and staining protocols used in this study are described in Jensen 2017.

[0271] The number of qIHC points for each cell line was counted, and the average results for each cell line are shown in the figure. Figure 27 The provided charts have shading around each mean to indicate the 95% confidence interval. Figure 28 Representative images of qIHC-stained cells examined in this study are provided. Detected qIHC spots are labeled with dots (left image) or boxes (right image). Figure 27 As shown, there is a linear relationship between the mean point / cell count and HER2 expression level in these cell lines. Therefore, this method provides reliable and consistent quantitative expression information in FFPE cell lines with different HER2 expression levels.

[0272] It is important to note that the term "first" (e.g., in the terms "first ML model," "first set of annotations," etc.) is used here for labeling purposes only. It does not imply any quantity or quality; for example, it does not suggest any type of application order, nor any specific characteristic of the model itself. The same applies to the terms "second" image, "third" image, "fourth" image, "second" set of annotations, "third" set of annotations, "fourth" set of annotations, etc.

[0273] While certain features and aspects have been described with respect to exemplary embodiments, those skilled in the art will recognize many possible modifications. For example, the methods and processes described herein can be implemented using hardware components, software components, and / or any combination thereof. Furthermore, while the various methods and processes described herein may be described with respect to specific structural and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any particular structural and / or functional architecture, but can be implemented on any suitable hardware, firmware, and / or software configuration. Similarly, while a function may be attributed to certain system components, unless the context otherwise specifies, this function may be distributed among various other system components according to several embodiments.

[0274] Furthermore, although the processes of the methods and procedures described herein have been presented in a specific order for ease of description, various processes may be reordered, added, and / or omitted according to the various embodiments unless the context otherwise requires. Moreover, the procedures described with respect to a method or process may be incorporated into other described methods or processes; similarly, system components described with respect to a particular architecture and / or a system may be organized in an alternative architecture and / or incorporated into other described systems. Therefore, although various embodiments have been described with (or without) specific features for ease of description and illustration of exemplary aspects of these embodiments, various components and / or features described herein with respect to specific embodiments may be replaced, added, and / or subtracted from other described embodiments unless the context otherwise requires. Therefore, although several exemplary embodiments have been described above, it should be understood that this disclosure is intended to cover all modifications and equivalents within the scope of the appended claims.

[0275] Various embodiments of the present disclosure have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0276] It is anticipated that many related machine learning models will be developed during the lifetime of the patents that mature from this application, and the scope of the term machine learning model is intended to a priori include all such new technologies.

[0277] The term “about” as used in this article refers to + / - 10%.

[0278] The terms “including,” “comprising,” “containing,” “having,” and their variations mean “including but not limited to.” This term includes the terms “consisting of” and “substantially composed of.”

[0279] The phrase “consistently of” means that a composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not substantially alter the fundamental and novel characteristics of the claimed composition or method.

[0280] The singular forms “a,” “an,” and “the” used herein include plural references unless the context clearly indicates otherwise. For example, the terms “compound” or “at least one compound” can include a variety of compounds, including mixtures thereof.

[0281] As used herein, the term "exemplary" means "as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or as excluding the incorporation of features from other embodiments.

[0282] As used herein, the term "optionally" means "provided in some embodiments but not in others." Any particular embodiment may include multiple "optional" features unless these features conflict.

[0283] Throughout this application, various embodiments may be presented in a range format. It should be understood that the range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the various embodiments described and / or claimed herein. Therefore, the range description should be considered to specifically disclose all possible subranges and individual numerical values ​​within that range. For example, a range description such as 1 to 6 should be considered to have specifically disclosed subranges, such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, etc., and individual numbers within that range, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the width of the range.

[0284] Whenever a range of numbers is indicated in this document, it is intended to include any referenced numbers (fractions or integers) within the indicated range. The phrases “range between the first indicated number and the second indicated number” and “range from the first indicated number to the second indicated number” may be used interchangeably in this document and are intended to include the first and second indicated numbers as well as all fractions and integers in between.

[0285] It should be understood that certain features of the embodiments described and / or claimed herein (which, for clarity, are described in the context of individual embodiments) may also be provided in combination in a single embodiment. Conversely, for brevity, various features described in the context of a single embodiment may also be provided individually or in any suitable sub-combination or as suitably as in any other described embodiment. Certain features described in the context of various embodiments are not considered essential features of those embodiments unless the embodiment is inoperable without those elements.

[0286] While this disclosure has been described with reference to specific embodiments, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, this disclosure is intended to cover all such alternatives, modifications, and variations that fall within the spirit and broad scope of this disclosure and the claims.

[0287] The applicant intends that all publications, patents, and patent applications referenced in this specification are incorporated herein by reference in their entirety, as if each individual publication, patent, or patent application were specifically and separately mentioned when incorporated herein by reference. Furthermore, any reference or designation of any reference in this application shall not be construed as an admission that such reference is prior art to this application or any patent derived therefrom. The use of section headings shall not be construed as a necessary limitation. In addition, any priority documents of this application are incorporated herein by reference in their entirety.

Claims

1. A method for generating training data, the method comprising: Acquire the first image of the biological sample; A first set of annotations for the first image is generated based on a second image of the biological sample stained by a quantitative method, wherein the quantitative method converts the antibody / antigen complex into dots; as well as The first image and the benchmark truth including the first set of annotations are output as data for training a first machine learning ("ML") model to analyze biological samples.

2. The method according to claim 1, wherein, The first set of annotations was generated based on statistical measurements of points in the second image.

3. The method according to claim 2, wherein, The statistical measures include the number of points and / or the density of points.

4. The method according to claim 2 or 3, wherein, Generating the first set of annotations includes generating annotations for each of the multiple regions of the first image by determining a statistical measure of the matching regions in the second image that match the regions in the first image.

5. The method according to claim 2 or 3, wherein, Generating the first set of annotations includes: Obtain multiple regions from the second image; For each of the multiple regions in the second image: Calculate the statistical metric, and The calculated statistical metric is used as a label in the first set of labels and associated with the location of the region in the first image that matches the region in the second image.

6. The method according to claim 4 or 5, wherein, Each region is a rectangle, square, or circle located at a pre-configured position in the first image.

7. The method according to any one of claims 4 to 6, wherein, The first set of annotations includes: statistical measures associated with the location of the corresponding region in the first image and / or associated with the location of the corresponding matching region in the second image.

8. The method according to any one of claims 4 to 7, wherein, At least one of the plurality of regions is a closed region selected in the first image.

9. The method according to any one of claims 1 to 8, wherein, The generation of the first set of annotations includes: The second image is input into the second process; and The output of the second processing is used as the first set of labels.

10. The method according to any one of claims 1 to 9, wherein, The first image is an image of the biological sample stained with immunohistochemistry ("IHC"), hematoxylin and eosin ("H&E"), and / or fluorescence in situ hybridization ("FISH").

11. The method according to any one of claims 1 to 10, further comprising: Acquire a second image using a first concentration of antibody / antigen and stained using a quantitative method that converts the antibody / antigen complex into dots; A third image of the biological sample is obtained, the third image being stained using a second concentration different from the first concentration, and using a quantitative method for converting antibody / antigen complexes into dots, wherein the second image and the third image are images of respective slides of sequential sections of the biological sample; and Generate a second set of annotations for the first image based on the third image; and The first image and the baseline truth including the second set of annotations are output as data for training the first ML model to analyze biological samples.

12. The method according to any one of claims 1 to 11, wherein, The staining method described is IHC staining, used to visualize HER2 epitopes in formalin-fixed paraffin-embedded tissues.

13. A method for training a machine learning ("ML") model, the method comprising: The first image and the benchmark truth, output by the method for generating training data according to any one of claims 1 to 12, are input into the ML model; as well as Modify at least one parameter of the machine learning model based on the input first image and the benchmark truth.

14. The method of claim 13, comprising one or more iterations of the method for generating training data according to any one of claims 1 to 12, and the steps of inputting and modifying.

15. An analytical method, comprising: The fourth image of the biological sample or the first image of the second biological sample is input into the ML model trained using the method according to claim 11; as well as The analysis of the biological sample or the second biological sample is obtained from the output of the ML model.

16. The method according to claim 15, wherein, The analysis results indicate the presence, absence, or amount of antigens in the biological sample or the second biological sample.

17. The method according to claim 15 or 16, wherein, The analysis is a classification of the biological sample or the second biological sample.

18. The method according to any one of claims 1 to 17, wherein, The ML model is a first ML model, and generating the first set of annotations includes: Detecting objects in an image of a biological sample, the detection including: Acquire images of biological samples stained using a quantitative method that converts antibody / antigen complexes into dots, and Points in the image of the biological sample are detected using a trained second ML model, which is pre-trained to detect points; The second image is input into the second process; and The output of the second processing is used as the first set of labels.

19. The method according to claim 18, wherein, The detection also includes generating the image of the biological sample and one or more annotations associated with the detected points.

20. The method according to claim 18 or 19, wherein, The one or more labels indicate the location of the corresponding one or more detected points.

21. The method according to claim 20, wherein, The location of the point is its position in three-dimensional space.

22. The method according to any one of claims 19 to 21, wherein, The detection also includes: Provide an input interface that allows human users to obtain one or more updated labels in the following ways: i) Delete the label from the one or more labels. ii) Modify the annotations in one or more of the above annotations, and / or iii) Adding annotations to the one or more annotations; and Store the updated one or more labels.

23. The method of claim 22, further comprising: The updated one or more labels are used as baseline truth to train the second ML model or another AI model for detecting points.

24. The method according to any one of claims 19 to 23, further comprising: A label viewing interface is provided, which is configured to display the image of the biological sample together with one or more labels.

25. The method according to claim 24, wherein, The annotation viewing interface is a side-by-side viewing interface, which is configured to display a first view and an adjacent second view. The first view shows the image based on the biological sample and includes a first view image with one or more annotations superimposed, and the second view shows a second view image of the biological sample.

26. The method according to claim 25, wherein, The second view was acquired with settings different from those of the image of the biological sample, and includes annotations corresponding to one or more annotations in the first view image.

27. The method according to claim 25 or 26, wherein, The second view image of the biological sample is based on or includes a z-stack, which comprises two or more focal plane images of the same field of view (FOV) of the biological sample that are different from each other.

28. The method according to claim 19, wherein, The second view image is obtained by tiling or 3D deconvolution of the z-axis stack, including combining two or more focal plane images into a single image of the biological sample.

29. The method according to any one of claims 24 to 28, wherein, The annotation viewing interface is configured to view each of the one or more annotations as a graphic of an unfilled outline with a pre-configured shape surrounding one of the detected points.

30. The method according to claim 29, wherein, The annotation viewing interface is configured to allow switching between the focal planes of the z-axis stack.

31. The method according to claim 29 or 30, wherein, The annotation viewing interface is configured as follows: This allows the user to select a focal plane from the z-axis stack and mark a point within the selected focal plane. Store the identifiers associated with the selected focal plane and the marked points.

32. The method according to any one of claims 29 to 31, wherein, The shape is a rectangle or a square, which is positioned such that one of the detected points is located at its geometric center.

33. The method according to any one of claims 29 to 32, wherein, The annotation viewing interface is configured to allow switching between the following modes: i) View each annotation in the annotation as a point graph co-located with one of the detected points, and ii) View each of the annotations as a graph of an unfilled outline with a pre-configured shape surrounding one of the detected points.

34. The method according to claim 33, wherein, The switching is triggered by changing the field of view of the image of the biological sample.

35. The method according to any one of claims 18 to 34, wherein, The detection points also include: classifying or screening the one or more points based on additional cellular or non-cellular markers.

36. The method according to any one of claims 1 to 35, wherein, The detection points also include: classifying or filtering the one or more points based on one or more additional images of the biological sample, wherein the one or more additional images include one or more images of the same slice or consecutive slices.

37. The method of any one of claims 18 to 36, further comprising training the second ML model to detect objects in an image of a biological sample, the training comprising: Acquire images of biological samples stained using a quantitative method that converts antibody / antigen complexes into dots; Acquire the image of the biological sample and one or more annotations associated with the point; One or more parameters of the second ML model are adjusted by inputting the image of the biological sample and one or more annotations as a baseline truth into the AI ​​model.

38. The method according to claim 37, wherein, The training also includes: Acquire images of negative control biological samples stained with IHC or fluorescence-based methods; One or more parameters of the AI ​​model are adjusted based on the image of the negative control biological sample and the unlabeled or unmarked data that serves as a baseline of truth.

39. A computer program stored on a non-transitory medium and comprising instructions that, when executed on one or more processors, cause the one or more processors to perform the steps of the method according to any one of claims 1 to 38.

40. A training data generation device, comprising: Data interface; as well as The processing circuit performs the following operations during operation: The first image of the biological sample is obtained via the data interface; A first set of annotations for the first image is generated based on a second image of the biological sample stained by a quantitative method, wherein the quantitative method converts the antibody / antigen complex into dots; as well as The first image and the benchmark truth including the first set of annotations are output as data for training a first machine learning ("ML") model to analyze biological samples.

41. The method according to any one of claims 19 to 21, wherein, The method further includes: The first ML model or the second ML model is used to determine the number of points for each cell in one or more cells of the biological sample or the second biological sample, and The antibody expression level in the biological sample or the second biological sample is determined or estimated based on the number of points for each cell.

42. The method according to claim 41, wherein, The antibody expression level in the biological sample or the second biological sample is determined or estimated based on the average number of points per cell.

43. The method according to claim 41 or 42, wherein, The biological sample or the second biological sample includes formalin-fixed paraffin-embedded cells.

Images