Automated detection of dots in images of stained biological samples

Abstract

The present disclosure provides methods and apparatuses for detecting objects in an image of a biological sample. In particular, an image is obtained of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots. The dots are detected in said image using a trained artificial intelligence ("AI") model that has been pre-trained to detect dots. Moreover, methods and apparatuses for training such AI model are provided.

Description

AUTOMATED DETECTION OF DOTS IN IMAGES OF STAINEDBIOLOGICAL SAMPLESCross-Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Application Nos. 63 / 605,424, and 63 / 605,435, which were filed on December 1, 2023, the entire contents of which are hereby incorporated by reference.Technical Field

[0002] The present disclosure relates generally to methods and devices for use in detecting targets in biological tissues aided by imaging, and, in particular, to digital microscopy.Background

[0003] Histological specimens are frequently disposed upon glass slides as a thin slice of patient tissue fixed to the surface of each of the glass slides. Using a variety of chemical or biochemical processes, one or several colored compounds may be used to stain the tissue to differentiate cellular constituents, which can be further evaluated utilizing microscopy. Bright- field slide scanners are conventionally used to digitally analyze these slides.

[0004] When analyzing tissue samples on a microscope slide, staining the tissue or certain parts of the tissue with a colored or fluorescent dye can aid the analysis. The ability to visualize or differentially identify microscopic structures is frequently enhanced using histological stains. Hematoxylin and eosin (“H&E”) stains are the most commonly used stains in light microscopy for histological samples.

[0005] In addition to H&E stains, other stains or dyes have been applied to provide more specific staining and provide a more detailed view of tissue morphology. Immunohistochemistry (“IHC”) stains have great specificity, as they use a peroxidase substrate or alkaline phosphatase (“AP”) substrate for IHC staining, providing a uniform staining pattern that appears to the viewer as a homogeneous color with intracellular resolution of cellular structures, e.g., membrane, cytoplasm, and nucleus. Formalin Fixed Paraffin Embedded (“FFPE”) tissue samples, metaphase spreads or histological smears are typically analyzed by staining on a glass slide, where a particular biomarker, such as a protein or nucleic acid of interest, can be stained with H&E and / or with a colored dye, hereafter “chromogen” or “chromogenic moiety.” IHC staining is a common tool in evaluation of tissue samples for the presence of specific biomarkers. In situ hybridization (“ISH”) may be used to detect targetnucleic acids in a tissue sample. ISH may employ nucleic acids labeled with a directly detectable moiety, such as a fluorescent moiety, or an indirectly detectable moiety, such as a moiety recognized by an antibody which can then be utilized to generate a detectable signal. Further approaches such as Fluorescence ISH (“FISH”) have been also applied.

[0006] Compared to other detection techniques, such as radioactivity, chemo-luminescence or fluorescence, chromogens generally suffer from much lower sensitivity, but have the advantage of a permanent, plainly visible color which can be visually observed, such as with bright field microscopy. However, more substrates with additional properties which may be useful in various applications, including multiplexed assays, such as IHC or ISH assays, are needed.

[0007] Additional capabilities for advanced image analysis of histological slides which may be used, for example, in digital pathology may improve detection and assessment of specific molecular markers, tissue features, and organelles, or the like. However, achieving high accuracy has been a challenging issue not only for machine learning based algorithms but also for experienced professionals.Brief Summary

[0008] In recent years, digital pathology has gained more popularity as many stained tissueslides are digitally scanned with high resolution (e.g., 40*) and viewed as whole slide images (“WSIs”) using digital devices (e.g., PCs, tablets, etc.) instead of standard microscopes. Having the information in a digital format enables digital analyses that may be applied to WSI to facilitate diagnoses. Recently, quantitative staining approaches have been developed, which convert antibody / antigen complexes into dots. These dots may then be detected, and they provide a quantitative measure for expression of desired molecules (e.g., proteins).

[0009] Developing a robust automated approaches is particularly challenging due to the huge diversity of shape, color, orientation, and density of staining in different tissue and stain types. Hence, there is a need for more robust and scalable solutions for implementing digital microscopy imaging, and, more particularly, to methods, systems, and apparatuses for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on annotations, and / or implementing user interface configured to facilitate user annotation for instance segmentation within biological samples.

[0010] Accordingly, in some aspects the present disclosure provides systems and methods that facilitate detection of dots that result from staining a biological sample with a quantitative approach using artificial intelligence, e.g., machine learning (“ML”) models. Such detectionmay be used to generate (a basis for) ground truth that may be further used to train the artificial intelligence.

[0011] According to a first aspect, a computerized method is provided of detecting objects in an image of a biological sample, the method comprising: obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; and detecting dots in said image of a biological sample using a trained artificial intelligence (“Al”) model pre-trained to detect dots.

[0012] According to a second aspect, in addition to the first aspect, the method further comprises generating one or more annotations associated with the image of a biological sample and with the detected dots.

[0013] According to a third aspect, in addition to the first or second aspect, said one or more annotations indicate location of respective one or more of the detected dots.

[0014] According to a fourth aspect, in addition to the third aspect, the location of dots is a location in three spatial dimensions.

[0015] According to a fifth aspect, in addition to any of the second to fourth aspect, the method further comprises providing an input interface capable of enabling a human user to obtain an updated one or more annotations by: i) deleting an annotation from the one or more annotations, ii) modifying an annotation among the one or more annotations, and / or iii) adding an annotation to the one or more annotations; and storing the updated one or more annotations.

[0016] According to a fourth aspect, in addition to the fifth aspect, the method further comprises using the updated one or more annotations as a ground truth to train said Al model or another Al model for detecting dots.

[0017] According to a seventh aspect, in addition to any of the second to sixth aspect, the method further comprises providing an annotation viewing interface configured to display said image of the biological sample together with said one or more annotations.

[0018] According to an eighth aspect, in addition to the seventh aspect, the annotation viewing interface is a side-by-side viewing interface configured to display a first view showing a first-view image based on said image of the biological sample and including said one or more annotations overlaid beside a second view showing a second-view image of the biological sample.

[0019] According to a ninth aspect, in addition to the eighth aspect, said second-view is captured with settings different from those of said image of the biological sample and includes annotations corresponding to said one or more annotations of the first-view image. i

[0020] According to a tenth aspect, in addition to the eighth or ninth aspect, said second- view image of the biological sample is based on or includes a z-stack including two or more mutually different focal plane images of the same field of view, FOV, of the biological sample.

[0021] According to a eleventh aspect, in addition to any of the eight to tenth aspect, said second-view image is obtained by flattening or 3D deconvolution of the z-stack including combining the two or more focal plane images into a single image of the biological sample.

[0022] According to a twelfth aspect, in addition to any of the seventh to eleventh aspect, the annotation viewing interface is configured to view each of said one or more annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots.

[0023] According to a thirteenth aspect, in addition to the twelfth aspect, the preconfigured shape is a rectangle or a square positioned so that the one of the detected dots is in its geometric center.

[0024] According to a fourteenth aspect, in addition to any of the seventh to thirteenth aspect, the annotation viewing interface is configured to enable switching between focal planes of the z-stack.

[0025] According to a fifteenth aspect, in addition to any of the seventh to fourteenth aspect, the annotation viewing interface is configured to: i) enable a user to select a focal plane out of the z-stack and to mark a dot in the selected focal plane, and ii) store an identification of said selected focal place in association with the marked dot.

[0026] According to a sixteenth aspect, in addition to any of the second to seventh to fifteenth aspect, the annotation viewing interface is configured to enable switching between: i) viewing each of said annotations as a graphics of a dot co-located with one of the detected dots, and ii) said viewing each of said annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots.

[0027] According to a seventeenth aspect, in addition to the sixteenth aspect, said switching is triggered by changing a field of view of said image of the biological sample.

[0028] According to a eighteenth aspect, in addition to any of the first to seventeenth aspect, the step of detecting dots further includes categorizing or filtering said one or more dots based on additional cellular or non-cellular markers.

[0029] According to a nineteenth aspect, in addition to any of the first to eighteenth aspect, the step of detecting dots further comprises categorizing or filtering said one or more dots based on additional one or more images of the biological sample including one or more images of the same slide or of consecutive slides.

[0030] According to a twentieth aspect, a computerized method is provided of training an Al model for detecting objects in an image of a biological sample, the method comprising: i) obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; ii) obtaining one or more annotations associated with the image of the biological sample and with the dots; and iii) adapting one or more parameters of said Al model according to said image of a biological sample and, as a ground truth, said one or more annotations into the Al model.

[0031] According to a 21-st aspect, in addition to the twentieth aspect, the method further comprises obtaining an image of a negative control biological sample stained with an IHC or fluorescence-based approach; and adapting one or more parameters of said Al model according to said image of a negative control biological sample and, as a ground truth, no annotations or an annotation indicating no dots.

[0032] According to a 22-nd aspect, in addition to twentieth or 21 -st aspect, said one or more annotations indicate location of respective one or more dots detected in the detecting step.

[0033] According to a 23-rd aspect, in addition to twentieth or 22-nd aspect, the location of dots is a location in three spatial dimensions.

[0034] According to a 24-th aspect, a computerized method is provided of training a first Al model for detecting objects in an image of a biological sample, the method comprising: i) obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; ii) obtaining one or more annotations associated with the image of the biological sample and with the dots, wherein said obtaining includes: a) obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots, and b) detecting dots in said image of a biological sample using a trained second Al model pre-trained to detect dots; and iii) adapting one or more parameters of said first Al model according to said image of a biological sample and, as a ground truth, said one or more annotations into the first Al model.

[0035] According to a 25-th aspect, a computer program is provided, stored on a non- transitory medium and including instructions which when executed on one or more processors causes the one or more processors to perform the steps of the method according to any of the first to 24-th aspect.

[0036] According to a 26-th aspect, a detection device is provided comprising: a data interface; a storage; and processing circuitry that, in operation, i) obtains over said data interface an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots, ii) detects dots in said image of a biological sample using ia trained Al model pre-trained to detect dots, and iii) stores indication of locations of said respective detected dots as one or more annotations.

[0037] According to a 27-th aspect, in addition to twentieth or 26-th aspect, the detection device further comprises an input interface capable of enabling a human user to update one or more annotations by: i) deleting an annotation from the one or more annotations, ii) modifying an annotation among the one or more annotations, and / or iii) adding an annotation to the one or more annotations; wherein the processing circuitry, in operation, stores the updated one or more annotations in the storage.

[0038] According to a 28-th aspect, in addition to 26-th or 27-th aspect, the detection device further comprises an annotation viewing interface configured to display said image of the biological sample together with said one or more annotations.

[0039] According to a 29-th aspect, a training device is provided, comprising: a data interface; a storage; and processing circuitry that, in operation: i) obtains an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots via the data interface; ii) obtains one or more annotations associated with the image of the biological sample and with the dots via the data interface; iii) adapts one or more parameters of an Al model according to said image of a biological sample and, as a ground truth, said one or more annotations, thereby training the Al model for detecting objects in an image of a biological sample.

[0040] According to a 30-th aspect, a computer-implemented method of measuring antibody expression is provided, comprising: obtaining an image of a biological sample comprising one or more cells stained with a quantitative approach converting antibody / antigen complexes into dots; detecting dots in the image of the biological sample using a trained Al model pre-trained to detect dots; determining a number of dots per cell in the image of the biological sample using the trained Al model; determining or estimating a level of antibody expression in the biological sample based on the determined number of dots per cell. In some aspects, the level of antibody expression in the biological sample is determined or estimated based on a mean number of dots per cell. In some aspects, the biological sample comprises formalin-fixed and paraffin- embedded cells.

[0041] It is noted that the present disclosure also provides devices of which the processing circuitry performs any of the methods described herein. The present disclosure provides an integrated circuit which embodies the processing circuitry as described above.Brief Description of the Drawings

[0042] A further understanding of the nature and advantages of particular embodiments may be realized by reference to the remaining portions of the specification and the drawings, in which like reference numerals are used to refer to similar components. In some instances, a sub-label is associated with a reference numeral to denote one or multiple similar components. When reference is made to a reference numeral without specification to an existing sub-label, it is intended to refer to all such multiple similar components.

[0043] FIG. 1 is a flow diagram exemplifying a method for dot detection and including schematic representation of an image of a biological sample with and without annotations.

[0044] FIG. 2 is a block diagram illustrating an exemplary device for dot detection.

[0045] FIG. 3 is a flow diagram illustrating a method got annotation generation and updating, as well as a method for training of an artificial intelligence on the fly.

[0046] FIG. 4 is a schematic drawing of graphical user interface allowing displaying of an image of a biological sample and the associated annotations alongside with some exemplary buttons for updating annotations.

[0047] FIG. 5 is an exemplary excerpt of an image of a biological sample stained with qlHC.

[0048] FIG. 6 is a schematic drawing illustrating a side-by-side view of a graphical user interface.

[0049] FIG. 7 is a schematic drawing illustrating some possible forms of an annotation.

[0050] FIG. 8 is an exemplary flow diagram illustrating phases connected with application of an artificial intelligence.

[0051] FIG. 9 is an exemplary flow chart illustrating a method for training an artificinal intelligence model.

[0052] FIG. 10 is an exemplary content of a memory storing functional modules for configuring processing circuitry to perform the modules’ functionalities.

[0053] FIG. 11 is an exemplary block diagram illustrating a system including the dot detection device of Fig. 2.

[0054] FIG. 12 is a schematic drawing illustrating application of qlHC.

[0055] FIG. 13 provides a set of images of biological samples stained with IHC (top row) and qlHC (bottom row).

[0056] FIG. 14 is a graph showing the measured qlHC dots / cell in an exemplary set of cell lines with different HER2 expression.

[0057] FIG. 15 provides a pair of images of the same biological sample stained with qlHC, with detected qlHC dots annotated using points (left image) and boxes (right image). iDETAILED DESCRIPTION

[0058] The present disclosure relates, in general, to methods, programs, systems, and apparatuses for facilitating digital microscopy imaging (e.g., digital pathology or live cell imaging, etc.). More specifically, the present disclosure relates to implementing digital microscopy imaging using quantitative staining approaches converting antibody / antigen complexes into dots.

[0059] The present disclosure relates generally to methods and devices for use, for example, in multiplexed assays or in cases of consecutive slides that may be observed together for detecting target molecules or other parts of biological tissue. Such methods have a wide utility in diagnostic applications, in choosing appropriate therapies for individual patients, or in training neural networks or developing algorithms for use in such diagnostic applications or selection of therapies. The present disclosure may also facilitate implementing annotation data collection and autonomous annotation, and, more particularly, implementing imaging of biological samples for generating training data for developing deep learning based models for image analysis, for cell classification, for feature of interest identification, and / or for virtual staining of biological samples.

[0060] The following detailed description illustrates a few exemplary embodiments in further detail to enable one of skill in the art to practice such embodiments. The described examples are provided for explanatory purposes and are non-limiting and non-exhaustive.

[0061] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the described embodiments. It will be apparent to one skilled in the art, however, that other embodiments described herein may be practiced without some of these specific details. In other instances, certain structures and devices are shown in block diagram form. Several embodiments are described herein, and while various features are ascribed to different embodiments, it should be appreciated that the features described with respect to one embodiment may be incorporated with other embodiments as well. By the same token, however, no single feature or features of any described embodiment should be considered essential to every embodiment, as other embodiments may omit such features.

[0062] In clinical pathology and in particular in digital pathology, detection of protein in intact formalin-fixed, paraffin-embedded tissue may be performed using immunohistochemistry, which is semi-quantitative. For example, recent approaches have brought quantitative immunohistochemistry (“qlHC”) that enables quantification of e.g., a iprotein directly in formalin-fixed, paraffin-embedded tissue by counting of dots. The qlHC technology can be combined with standard immunohistochemistry, and assessed using standard bright-field microscopy or image analysis. The qlHC method is in principle similar to classic IHC, and like classic immunohistochemistry, the basis for the amplification is enzyme deposition (typically Horse radish peroxidase (“HRP”)). A primary antibody binds a target (protein / receptor) of interest. A secondary, HRP -labeled antibody recognizes the primary antibody and binds. These first two steps in the qlHC reaction are directly comparable to standard IHC. However, in qlHC, only a predetermined fraction of the secondary antibodies is labeled. The secondary-labeled antibody is mixed with non-labeled antibody to increase robustness of the assay. Finally, an amplification reaction generates a dot centered around the labeled antibody, i.e., directly at the single target. The number of dots can be counted and as the ratio between labeled and unlabeled secondary antibody is known, the qlHC assay allows for a direct correlation between number of dots and amount of biomarker present in the tissue.

[0063] However, quantitative approaches are not limited to qlHC and may be also applied with ISH methods against DNA or RNA, imaging mass cytometry or immunofluorescencebased methods, or the like.

[0064] Some embodiments herein provide computational methods for qlHC dot detection, as well as the corresponding apparatuses and systems.

[0065] Classical computer vision algorithms may be used for detecting dots, for example, based on color difference, morphology, shape and size. However, such methods may suffer from several limitations. For example, such methods may not be sufficiently robust to changes in staining quality, color and variable shape and / or size of dots. Moreover, such methods are not easy to tune in order to prevent or at least reduce false positive detections due to dot-like structures that may be present in a biological sample. For instance, in a biological tissue, objects or structures may look similar to dots but are not necessarily the dots - such objects may be cell nucleoli, or the like.

[0066] Dot detection in stained images

[0067] In order to provide a more efficient approach, in an embodiment illustrated in Fig. 1, an image of a biological sample is obtained 110, stained with a quantitative approach converting antibodies or antigens into dots. Such exemplary image 160 of a biological sample is shown in Fig. 1, including dots - one of the dots being marked by an arrow 150.

[0068] Then, dots are detected in said image 160 using a machine learning model (“MLM”). In order to be able to detect the dots, the MLM is pre-trained to detect dots. The detection of the dots is thus performed by inputting 120 the image 160 into the MLM and by obtaining 130as an output of the MLM the result of the detection, i.e., a location of the detected dots within the image 160. These locations are illustrated in an image 190 (corresponding to image 160) by rectangular contours, such as rectangle 170 surrounding position of the dot pointed to by the arrow 150 in image 160.

[0069] In this context, “pre-trained” means that the MLM is configured with one or more parameters that have been obtained in a training phase preceding the application of the MLM for the detection of dots (inference phase). It is not necessary to perform the actual training phase for each MLM instance used for detection of the dots. The training may be performed once for one MLM instance and the resulting parameters of such trained MLM instance may then be stored and provided to configure other MLM instances. The present disclosure is not limited to any particular way in which the MLM is parametrized. As will be described below, training phase may be even performed - even on the fly.

[0070] The quantitative approach converting antibodies or antigens into dots includes conversion of antigen-antibody complexes using staining into dots as described above, e.g., for qlHC. The dots are obtained by massive amplification by several steps of enzyme-catalyzed deposition at the sites of primary antibody where secondary antibody-dextran-HRP polymers are bound.

[0071] The method 100 of Fig. 1 is a computerized method for detecting objects in an image such as 160 of a biological sample. In some embodiments, the biological sample might include, without limitation, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, where the obj ects of interest might include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures, and / or the like.

[0072] Some advantages of the method 100 of Fig. 1 over classical image processing methods may include a more accurate and robust detection thanks to using an MLM, e.g., an Al-based model. Fig. 1 shows an image 160 of a tissue stained with qlHC scanned and shown at 40x magnification. qlHC dots can be seen as DAB (3,3 ’-Diaminobenzidine) brown dots. Cell nuclei are stained with Hematoxylin (in practice blue). However, other approaches and resulting colors are possible.

[0073] The computerized method 100 of Fig. 1 may be performed by a device 200 with an exemplary structure shown in Fig. 2. In particular, the device 200 may be a dot detection device. Processing circuitry 210 may perform steps 110, 120, and 130 described with reference to Fig. 1. In particular, the obtaining of the image 160 of a biological sample may be performed via a processing circuitry interface 280. For example, the processing circuitry interface may be aninterface to a bus 290 or another communication medium. Via the processing circuitry interface 280, the processing circuitry 210 may be connected to various other modules of the device 200 such as a memory 220, a display control 230, a communication interface 240, and an operation input interface 250. It is noted that this structure is only exemplary and schematic. For example, the bus 290 may in fact be a communication system including one or more buses and / or other communication media. The processing circuitry may be any piece of hardware including one or more processors (such as general-purpose processors) and / or programmable hardware pieces (such as FPGAs or the like) and / or application specific hardware (such as ASICs) and / or electronic circuits or elements. When including processors, the memory 220 may store software that may be capable of configuring the processing circuitry (when running on it) to perform the method 100. The memory 220 may, for example, store the MLM and / or the trained parameters of the MLM.

[0074] The dot detection device 200 may further include the display control module (display controller) 230 that, in operation, controls a display device which may be connected to or a part of the device 200. For example, the display control module 230 may control the display device to display the image 160 and / or the results of the detection or the like. The communication interface 240 is an interface for communication of the device 200 with other devices. For example, it may include communication interfaces such as Ethernet, WiFi (IEEE 802.11), Bluetooth, cellular communication networks (e.g., LTE, New Radio, or the like), or any other communication interfaces - standardized or proprietary. The operation input interface 250 is an interface for inputting operation commands. It may be an interface enabling a human user input (e.g., via a keyboard, specialized keys or buttons, mousejoystick or the like).

[0075] As shown in Fig. 3, the dot detection method 100 may be used as an input for generating 300 one or more annotations (such as 170) associated with the image 160 of a biological sample and with the dots (such as 150) detected in the detecting (step 120).

[0076] The generating 300 of annotations means in general processing the output of the MLM to obtain an annotation in a form further usable by a human or a machine. The annotation may be generated by the MLM or by another processing following the MLM. For example, said one or more annotations indicate location of respective one or more dots detected in the detecting step. In general, the location may be annotated by providing coordinates of the detected dots within the image 160, e.g., horizontal and vertical spatial coordinates x and y. In some exemplary implementations, the location of dots is a location in three spatial dimensions. In other words, in addition to the horizontal and vertical spatial coordinates x and y, the location includes depth coordinate z.

[0077] The annotations may indicate the location by way of explicitly providing numbers denoting the coordinates, and / or by providing markers that are superimposed to the image 160 on the positions or in the proximity of the positions of the detected dots. Such markers may be empty rectangles as the rectangle 170 in Fig. 1 or may have another shape or size.

[0078] In general, one of the advantages of the dot detection presented herein is that the MLM may be trained to distinguish between the dots resulting from a quantitative approach such as qlHC and other dot-like structures or objects in the biological sample.

[0079] Updating annotations

[0080] It is noted that the annotations generated in step 300 of Fig. 3 may be used in various ways. For example, they may be displayed 350 to assist human users (e.g., pathologists or other clinical practitioners) and / or stored together with or in association with the image 160.

[0081] However, the annotations may be also used as a ground truth for training MLMs. The annotations may be further refined 360 - possibly iteratively - and then used as a ground truth for additional training of the MLM used for dot detection or for training of other MLMs.

[0082] Correspondingly, the method described with reference to Fig. 3 may comprise a step of providing an input interface capable of enabling a human user to obtain 360 an updated one or more annotations. Such interface may correspond to the operation input interface 250 shown in Fig. 2, the human user may be enabled to update the one or more annotations by i) deleting an annotation from the one or more annotations, ii) modifying an annotation among the one or more annotations, and / or iii) adding an annotation to the one or more annotations; and the updated one or more annotations may be stored in association with the image 160. They may be displayed 350 again. This may be performed by the display controller 230 of the device 200 or Fig. 2.

[0083] The annotation process may thus be used in a cycle of improvement, where the user adjusts 360 the annotation produced 100, 300 by the MLM. Then, a new MLM may be trained based on the refined annotations and this cycle can be repeated until a desired model is achieved (e.g., having a desired accuracy). In other words, the method may comprise a step of storing 310 the updated annotations (e.g., in the memory 220 or in another storage which may ne external to the device 200) and using the updated one or more annotations as a ground truth to train 320 said MLM. Such additional training on the fly may help continuously improving the MLM performance. The MLM additionally trained may then be used again for detecting 100 dots.

[0084] However, it is noted that the present disclosure is not limited to implementations allowing for training 320 in the fly. It is not necessary to re-train the MLM model used forinference. The updated annotations may be stored 310 used for other purposes such as training another MLM for detecting dots or merely for viewing.

[0085] Any of the methods and apparatuses mentioned above may further provide an annotation viewing interface configured to display said image of the biological sample together with said one or more annotations. For example, in Fig. 2, the display control 230 may control a display to view the image of the biological sample together with said one or more annotations.

[0086] Fig. 4 illustrates, in a schematic manner, an exemplary user interface 400, with an image and annotation display field 410 and display buttons 420 (for adding an annotation), 430 (for modifying an annotation, e.g., moving it to a different location), 440 (for deleting an annotation), and 450 (for saving the updated annotations). The image and annotation display field 410 is an example for the annotation viewing interface. Within the image and annotation display field 410, an image 500 of the biological sample as shown in Fig. 5 may be shown and annotations may be overlaid on it.

[0087] Fig. 4 shows also an example of an input interface formed by buttons 420-450 that enable updating of the annotations as described above. In this example, the input interface is a graphic user interface, GUI, including active fields represented by images of a respective buttons together with an input device capable of registering a user input. This may be any device including a computer mouse, a touch screen, a touch pad, a pen, arrow keys, speech control device, or the like. Fig. 4 shows merely exemplary user interface buttons for updating annotations. However, the present disclosure is not limited to such interfaces. In general, not all four buttons must be present. For example, a modification button 430 may not be necessary, as the same effect may be achieved by deleting 440 and adding 420 annotations. It is further noted that the GUI buttons are only one example of input possibility. There may be physical keys (isolated or on a standard keyboard) associated with the respective actions of adding, modifying position, deleting, and / or saving the annotations.

[0088] Moreover, the GUI 400 is not limited to the four annotation updating actions. There may be further buttons such as a button for triggering re-training of the MLM with the currently displayed image with annotations (in the GUI field 410). Alternatively or in addition, there may be settings enabling selection of different kinds of annotations (e.g. annotations having different sizes and shapes) and / or different views or the like.

[0089] Regarding view configurations, according to an exemplary implementation, the annotation viewing interface is a side-by-side viewing interface configured to display a first view showing a first-view image based on said image of the biological sample and includingsaid one or more annotations overlaid beside a second view showing a second-view image of the biological sample.

[0090] A side-by-side interface 600 is schematically illustrated in Fig. 6. It comprises a first viewing field 610 and a second viewing field 620. Such arrangement may be particularly ergonomic for a human user. The first view filed 610 may show the biological sample image 160 (or an image based thereon) with annotations whereas the second view field 620 may show the biological sample image 160 without annotations. One advantage of such viewing is that the user may see at the same time the annotations and the original image in which the annotation do not overlap with the content. This may be particularly suitable if there are closely located dots.

[0091] When referring to the first-view image based on said image of the biological sample, what is meant is that the first-view image may be directly said image of the biological sample or may be an image based on it. For example, the image based on the image of the biological sample may be the image of the biological sample filtered or otherwise processed, e.g., to produce an image with a lower / higher contrast or brightness or the like.

[0092] The second-view image may also include the image of the biological sample or it may include another image taken of the same biological sample. For example, said second-view is captured with settings different from those of said image of the biological sample and includes annotations corresponding to said one or more annotations of the first-view image.

[0093] In an exemplary implementation, said second-view image of the biological sample is based on or includes a z-stack including two or more different focal plane images of the same field of view, FOV, of the biological sample. By different focal planes, what is meant is focal planes different from each other (mutually different). In general, the term “z-stack” imaging refers to obtaining a plurality of pictures taken at a set interval between the first and last planes of focus of a sample (e.g., a biological entity or tissue or in general any sample). Thus, each picture corresponds to a respective focus setting.

[0094] In an exemplary implementation, said second-view image is obtained by flattening or 3D deconvolution of the z-stack including combining the two or more focal plane images into a single image of the biological sample. The flattening or 3D deconvolution may be performed by any known approach. For example, for the flattening, extended depth of focus (“EDF”) may be used. EDF is an algorithm designed to scan through each picture in the set of pictures captured with different focus, and form a single composite image with all the parts that are determined to be in focus.Il

[0095] Provision of a single image based on several differently focused images has the advantage that such single image carries information combined from the different images and thus provides the user with a possibility to better distinguish the dots. Nevertheless, the present disclosure is not limited to displaying a single combined image. In some exemplary implementations, the annotation viewing interface 600 may provide a function of browsing through the images of the z-stack for a user in the second view 620 and / or a function for selecting a particular one of the different focal-plane images to be viewed.

[0096] In particular, the annotation viewing interface may be configured to enable switching between focal planes of the z-stack. For example, the annotation viewing interface is configured to: i) enable a user to select a focal plane out of the z-stack and to mark a dot in the selected focal plane, and ii) store an identification of said selected focal plane in association with the marked dot. In this way, some information of the depth (z coordinate) is obtained.

[0097] It is noted that in the above describes examples, it was assumed that the first-view image is in the view field 610 whereas the second-view image is in the view field 620. However, this was only for the sake of example. In practice, the first-view image may be in the second view field 620 whereas the second-view image may be in the first view field 610. In fact, the side-by-side view shown in Fig. 6 is only schematic and may be a part of a larger GUI including further functions such as those described with reference to Fig. 4 or the like. For example, the view 600 may correspond to the viewing field 410 in Fig. 4.

[0098] The present disclosure is not limited to displaying images captured at different respective focus planes. In addition or alternatively, images may be shown that were captured from a different angle or by different capturing apparatuses or with different settings or the like. For instance, the second-view image may be a consecutive slice of the same biological sample or an image of the biological sample, but differently stained, or the like.

[0099] It is noted that the second-view image may also include the annotations. In an exemplary implementation, annotations are presented in both views 610 and 620, and they are linked (associated with each other). The linking means that when annotation is moved in one of the views 610, 620 — it moves accordingly in the respective second view 620, 610. This may improve accuracy of the annotations (since the signal may not always be sufficiently clear in the original view). For example, the linking may be established by a preceding step of aligning the images of the two views together (globally, or per patch). The GUI may provide a possibility of selecting a function of a “linked view”. If a user selects the linked view function, the two image views (one of them or both including annotations) are linked - if one of the iiviews is moved (translated, rotated, and / or zoomed in or out) by a user within the viewing field, the other one of the views is moves correspondingly.

[0100] In general, the displaying or not of the annotations may be configurable by a user for the first-view image and / or the second-view image. The user GUI (such as those described with reference to Figs. 4 or 6) may further comprise a zooming functionality which enables a user to change a field of view (“FOV”) of the image 160 e.g., by zooming in or out, by translation or rotation, or the like.

[0101] Displaying annotations

[0102] The present disclosure is not limited to any particular shape, color or size of the annotations. In an example, the annotation viewing interface is configured to view each of said one or more annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots. For instance, the shape is a rectangle or a square positioned so that the one of the detected dots is in its geometric center.

[0103] Examples of such annotations encircling respective dots in an image 700 are provided in Fig. 7. Annotations 710 (rectangle or square), 720 (circle), or 730 (rotated rectangle or square) are empty contours. It is noted that it is not necessary to completely enclose the dots within the annotation contour. Moreover, annotations that merely point to the dots are also conceivable and exemplifies as an arrow 740 or even annotations that at least partially overlap with the dot and mark its center such as cross 750 in Fig. 7. The later may enable a visually more precise localization of the dots.

[0104] In general, it may be advantageous to provide annotations that do not touch the dot and do not overlap the dot they are marking (such as 710, 720, 730, 740). Moreover, it may be advantageous to provide annotations that do not touch or overlap any dot (not even the dots in the proximity of the marked dot). In special cases, such exemplary displaying rules may be lifted. For example, in case dots are highly diffused (out of focus) the contour (annotation) may be allowed to overlap outer parts of the diffused dot. It is possible to have a displaying rule according to which the central part of the dot is not obscured or overlapped by the annotation or by any annotation. The later may be more difficult to achieve especially in cases of high dot concentrations. It may be more ergonomic to a human viewer, if the annotations do not overlap with each other. On the other hand, some overlap would not cause any problems and such annotations may still enable a human user to clearly see the marked dots.

[0105] In an exemplary implementation, the annotation viewing interface is configured to enable switching between: i) viewing each of said annotations as a graphics co-located withone of the detected dots, and ii) said viewing each of said annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots.

[0106] It is noted that the graphics co-located (and at least partially overlapping) with one of the detected dots may be for instance a graphics of a dot, e.g., corresponding in size to the detected dot or fixed-size, irrespectively of the size of the detected dot. This is not to limit the present disclosure: the graphics co-located with one of the detected dots may have a different form, e.g., the cross 750 or the like.

[0107] The switching may be automatically triggered or triggered by a user. Regarding automatic triggering, in an exemplary implementation, said switching is triggered by changing a field of view of said image of the biological sample. For example, in case a zoom level is below a certain threshold (previously configured or fixed), the co-located (at least partially overlapping) displaying of the annotation is applied. In case the zoom level is above (or equal to) the certain threshold, the empty-shape annotation displaying may be applied.

[0108] Alternatively or in addition, the automatic triggering may be based on the contents of the field of view, for example based on the density of the dots and / or the proximity of the dots to each other. For instance, for densities higher than a threshold (pre-configured or fixed) annotations at least partially overlapping the dots and collocated with them are displayed. For the densities lower than or equal to the threshold, the empty contour annotations are displayed.

[0109] Regarding user-triggered annotations, an input interface may provide a possibility for a user to switch between annotation types (either at least partially overlapping and collocated or empty contours surrounding the dots). In addition or alternatively, the input interface may provide a possibility for a user to switch between (to configure) sizes, colors, shapes, and / or filling of the annotations.

[0110] Preprocessing or postprocessing[OHl] The above-mentioned dot detection using a MLM may be enhanced by additional processing steps. For example, the step of detecting dots further includes categorizing or filtering said one or more dots based on additional cellular or non-cellular markers.

[0112] Cellular markers are, for instance, nuclear, membranal or cytoplasmic markers or the like. The categorization can be performed using an MLM such as an Al based on a neural network or an algorithmic approach or the like. In case of using the MLM, it is possible to use the same MLM as the one for detecting dots, but it may be trained with additional labels (e.g. distinguishing in which cellular structure the dots are located). Labels may further include labels corresponding to properties of the dots such as dot sizes dot shapes, dot intensity, focal plane or the like. 1

[0113] The categorization classes can be, for example one or more classes distinguishing: whether or not a dot is located in a tumor region; whether or not a dot is located in a normal / stroma region; whether or not a dot is located in a cell nuclei; whether or not a dot is located on a membrane; and / or whether or not a dot is located in the vicinity of an immune cell.

[0114] In other words, the dots may be classified according to the surrounding biological structures.

[0115] In addition or alternatively, the step of detecting dots further comprises categorizing or filtering said one or more dots based on additional one or more images of the biological sample including one or more images of the same slide or of consecutive slides.

[0116] For example, additional stains on consecutive slides (and / or on the same slide) may be used to generate the labels. In addition or alternatively, the dots may be filtered based on their z plane - for example, the dots that are blurred (out-of-focus) may be excluded from the set of detected (annotated) dots.

[0117] Training of the MLM

[0118] Fig. 8 is an exemplary flow diagram that illustrates phases in tasks using an MLM or, more specifically, an Al such an Al based at least partially on a neural network architecture.

[0119] Step 810 represents training data collection. In this stage, training images of stained biological samples are collected together with the respective ground truth data. An exemplary ground truth data may be a list of coordinates of dots in a respective training image. However, the ground truth does not need to be list, it may be a 2D image in which merely the detected dot positions are marked or it may be directly the training image but with annotations added. The present disclosure is not limited to any particular format of the ground truth data.

[0120] The training data may be obtained as described above, by using an MLM to detect the dots and enabling a user to update the annotations (positions of the dots within the respective training image). The training data may be alternatively or in addition obtained purely by a human user marking the dots in the respective training image, or by another approach that enables marking the dots (such as any known algorithm specifically developed to detect dots (e.g., feature extraction, pattern matching, or the like).

[0121] Step 820 represents a training phase. The training phase will be described in more detail below. In the training phase, the training images associated with their respective ground truth data are fed to the MLM so as to enable it to learn (train).Il

[0122] Step 830 may be performed, but does not have to be. Step 830 is a testing phase. In principle, the testing step 830 corresponds to the inference phase 840. However, the testing phase is performed for training data that has not been used in the training and enables evaluating the quality of the MLM by testing its output against the ground truth for the training images that were not used in the training phase.

[0123] Step 840 represents an inference phase, i.e., the phase in which the MLM is applied to new data for which ground truth is unknown. In other words, the inference phase is the dot detection as described above, e.g., with reference to Fig. 1.

[0124] As can be seen in Fig. 8 and as already briefly discussed with reference to Fig. 3, the inference phase may be supplemented by a human user or another algorithm such as filtering or additional classification to modify the result of the detection by the MLM. Such modified result of detection may then be used as a new ground truth data for the training image and may be fed back to the training phase. Similar approach may be performed in the training phase.

[0125] Fig. 9 illustrates a training method 900. The training method 900 is a computerized method of training an artificial intelligence, Al, model (or, in general, an MLM) for detecting objects in an image of a biological sample. The method comprises steps of: obtaining 910 an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; obtaining 920 one or more annotations associated with the image of the biological sample and with the dots; and adapting 930 one or more parameters of said Al model according to said image of a biological sample and, as a ground truth, said one or more annotations into the Al model.

[0126] Adapting the one or more parameters of the Al may include, for instance, modification of some weights and / or biases of a neural network. However, the present disclosure is not limited to this. The adapting may include changing the model itself or switching to a non-neural -network based machine learning method or the like.

[0127] It is noted that annotation here are not limited to the actually displayed annotations, but merely represent data (information) that is associated with the image. For example, such annotations are locations of the dots within the image. These locations may be in two dimensions or in three dimensions.

[0128] In order to further improve the training, the method may further comprise a step of obtaining an image of a negative control biological sample stained with an IHC or a fluorescence based approach; and a step of adapting one or more parameters of said Al model according to said image of a negative control biological sample and, as a ground truth, no annotations or an annotation indicating no dots produced by the approach.

[0129] The negative controls may be stained with a regular qlHC (or a similar approach used also for the positive biological samples). However, in order to obtain the negative control, the quantitative approach may be run with a standard protocol (e.g., qlHC), but without the dextran-HRP-ab polymer capable of generating a “real” dot - so all generated dots would be ghost dots. Specifically, the negative controls may be produced by replacing the reagent in the labelled secondary antibody (1130 in Fig. 12) step with buffer. Apart from that, everything is run as a positive reaction. That means that the HRP enzyme needed to catalyze the precipitation of the substrate in step 1160 of Fig. 12 is not present.

[0130] It is noted that the training may be performed in a device similar to dot detector 200. In particular, in order to implement both, training 820 and inference 840, the processing circuitry 210 may, in operation, perform steps described with reference to Figs. 1, 3, 8, and / or 9. This may be facilitating by the memory 220 storing the corresponding program modules as shown schematically in Fig. 10.

[0131] Fig. 10 shows a memory portion 222 of the memory 220 with functional modules dedicated to dot detection (module 1010), user input processing (module 1020), GUI operation module (1030), and training (module 1040). It is note that not all modules must be present.

[0132] The present disclosure may provide merely a dot detector, in which case, only the dot detection module 1010 would be included. There may be no training module needed, as the MLM may be capable of only inference and may have the MLM parameter fixedly stored. The parameters may come from some preceding training, performed on a similar MLM implemented by a different device.

[0133] The device 200 may implement a user input processing module 1020 that receives a user input via the operation input interface 250, and determines an action to be taken upon that specific user input. In addition or alternatively, the device 200 may implement a GUI operation module 1030 for representing a GUI (producing images representing the GUI) that are then provided over the display controller 230 to a display device for displaying. This representing of the GUI may depend (receive input from) the input processing module 1020.

[0134] In addition or alternatively to the dot detection module 1010, the device 200 may include a training module 1040 for performing the training as described above.

[0135] Fig. 11 illustrates the device 200 described already with reference to Fig. 2, now as a part of a system that is capable of interacting with a human user. In particular, the device 200 is connected over its display control interface to a display device 260. The display device 260 may be a standalone screen external, and connectable and disconnectable from the device 200or it may be a display device permanently connected to and being a part of the device 200. Such display devise may be any kind of display such as an OLED, LCD or the like.

[0136] Moreover, a user input device 280 may be connected via the operation input interface 250 of the device 200. The user input device 280 may be a keyboard including one or more keys such as a standard computer keyboard or the like or any kind or keyboard. Alternatively, or in addition, the input device 280 may include a touch screen or a mouse or another means for pointing a cursor on various position for instance within a graphical user interface displayed on the display device 260. The communication interface 240 of the device 200 may be configured to connect the device 200 with a network 295. As illustrated in Fig. 11, an image capturing device 270 (such as a slide scanner or the like) that is a source of the first image and the second image may be also connected to the network 295, so that the device 200 may obtain the image(s) of a biological sample directly from the image capturing device 270. However, this way of obtaining the images is only exemplary. The images are not necessarily obtained directly from the image capturing device 270. They may be obtained from an external storage over the network 295. The communication interface 240 may include or be an USB interface, so that the images may be obtained e.g., from an USB storage device or the like.

[0137] The images may be stored and / or obtained together with a set of corresponding annotations (e.g., as training data). Alternatively or in addition, the annotations may be determined and stored together with the image by the device 200 as described above.

[0138] Exemplary embodiment based on qlHC

[0139] An exemplary embodiment may apply qlHC staining (generating dots). Such qlHC staining is illustrated in Fig. 12, (where an exemplary qlHC application is presented, namely dots that are expression of human epidermal growth factor receptor 2 (HER2). An example of a qlHC application can be seen in K. Jensen et al. “A novel quantitative immunohistochemistry method for precise protein measurements directly in formalin-fixed, paraffin-embedded specimens: analytical performance measuring HER2,” Mod. Pathol., vol. 30, issue 2, p. ISO- 193, Feb. 2017 (“Jensen 2017”).

[0140] Like in classic immunohistochemistry, the basis for the amplification is enzyme deposition —typically Horse radish peroxidase (“HRP”). In step 1 of Fig. 9, a primary antibody 1110 binds the target (protein / receptor) 1120 of interest. In step 2, a secondary, HRP -labeled antibody recognizes the primary antibody 1130, 1140 and binds. These first two steps in the qlHC reaction are directly comparable to standard immunohistochemistry. However, in qlHC, only a pre-determined fraction of the secondary antibodies (e.g., 1130) is labeled by HRP (1150). The secondary-labeled antibody 1130 is mixed with non-labeled antibody 1140 toincrease robustness of the assay. In step 3, enzyme substrate 1160 is added and deposited. In step 4, HRP labeled antibody 1170 binds deposited substrate 1160. Finally, in step 5, an amplification reaction generates a dot 1180 centered around the labeled antibody, i.e., directly at the single target. The number of dots can be counted and as the ratio between labeled and unlabeled secondary antibody is known, the qlHC assay allows for a direct correlation between number of dots and amount of biomarker present in the tissue.

[0141] Classical computer vision algorithms may be disadvantageous for detecting qlHC dots, inter alia due to a highly localized nature of the dots in three dimensions (3D) as well as in two dimensions (2D). For example, out-of-focus dots may look much larger and diffused in comparison with in-focus dots, making it nigh-impossible to detect them using classical image processing methods. The classical image processing methods are disadvantaged in dealing with detecting individual dots which are clustered or seem clustered when viewed in a single 2D focal plane, although being separated in the third dimension. In order to overcome these issues, for example, 3D deconvolution might be used, which requires a z-stack 3D scanning of the tissue at inference time. While it is possible to manually mark dot locations to serve as the ground truth for training an Al-based model, without using a side-by-side view of flattened z- stack scans or selectable focal plan scans it is much harder to do so accurately, since qlHC dots are highly localized in 3D and so might look diffused and confusing when out-of-focus.

[0142] Marking of dot locations without the use of see-through annotation mode may be more difficult because the dots may often be quite small and a filled marker (such as a filled circle) or in general an overlapping marker (such as the cross 750) might hide the dots. Reducing false-positive detection may be achieved by manual marking of non-qlHC structures as non-dots. This has the disadvantage of being a very time-consuming process. However, such provision of negative annotations (that mark dot-like structures that are not qlHC dots or dots of the desired staining approach) may speed up the training and / or increase its accuracy when used as ground truth for training data.

[0143] Using z-stack (multiple focal planes) scanning and / or flattened z-stack or 3D deconvolved z-stack for improved generation of ground truth for model training may provide more ergonomy to the human users, including as an assistive layer for manual annotators marking dot locations An iterative annotation process can be provided, where at each iteration the previously trained model is used to pre-annotate qlHC dots on a scan and a human annotator corrects the pre-annotations, changing, adding or deleting. The corrected annotations are used to train the next iteration of the detection model. Using negative control stained tissues (stained with qlHC without the part connecting to the marker) may help to efficiently generate groundtruth for reducing false positive detection. A trained model is applied to a negative control slide, and any dot detected by the model is assured to be a false-positive, as there are no actual dots in the slide.

[0144] User interface improvements for accurate and efficient marking dot locations may be provided as described above: A see-through annotation mode, enabling easy transition e.g., between annotations presented as (filled) dots and (unfilled) rectangles. The unfilled shape may provide some advantages, as filled markers hide the qlHC dot being marked. It may be useful for both ground truth annotation and for viewing model-inferred dot detections in a synchronized manner. Side-by-Side viewing of a panel showing the image being annotated and a panel showing selected z-stack focal planes or a flattened z-stack image may further improve the features of the user interface. Using the spatial relation of detected qlHC dots to expression of additional markers may be used to categorize or filter detected qlHC dots, including nuclear, membranal or cytoplasmic markers, as well as non-cellular markers.

[0145] Using in-focus analysis of z-stack scans, to localize detected qlHC dots in z-axis as well as in 2D, obtaining 3D localization of dots. Such 3D localization can enable more accurate assignment of detected dots to adjacent cells as indicated by additional markers. Advantages of approached described herein include improving the solution to the problem of detecting qlHC dots and localizing them in 2D and 3D, overcoming the difficulties of identifying dots due to confusing tissue morphology, changes in staining, dots clustering and out-of-focus dots. In addition, the present disclosure addresses the problem of efficiently and accurately generating annotated data for training an Al-based model for detecting qlHC dots, including the generation of annotated data for reducing false-positive detections.

[0146] More efficient and accurate data annotation process may be achieved thanks to an iterative process assisted by user interface improvements and z-stack based additional information making annotations more precise and easier to provide. Reducing false-positive detection using large amounts of efficiently generated false-positive annotations by applying trained model to negative controls.

[0147] The MLM may be an Al of any kind, or an ensemble thereof. For example, a model using deep neural networks such as a convolutional neural network may be used.

[0148] Methods for detection of qlHC dots may include training of an Al-based (e.g., neural network) qlHC dot detection model similar to those described in U.S. Patent No. 11,748,881. Therein, a fully convolutional network (“FCN”) and in particular the so-called U-net architecture is used, as suggested by O. Ronneberger et al. “U-Net: Convolutional Networks for Biomedical Image Segmentation”, 2015, arXiv: 1505.04597. The U-net has a contractingpath and an expansive path, which gives it the u-shaped architecture. The contracting path is a typical convolutional network that consists of repeated application of convolutions, each followed by a rectified linear unit (“ReLU”) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. The expansive pathway combines the feature and spatial information through a sequence of up- convolutions and concatenations with high-resolution features from the contracting path. Convolutional neural networks may be applied to patches (of a predetermined and possibly configurable size) of the input image that may be, in some applications, processed in parallel. However, the present disclosure is not limited to such implementations of MLM.Examples

[0149] Example 1. Evaluation of a qlHC detection model on formalin-fixed paraffin- embedded cell lines.

[0150] An experimental study was conducted to evaluate an exemplary qlHC detection model based on the present disclosure, with various formalin-fixed paraffin-embedded cell lines. This study demonstrated a linear relationship between expression level of HER2 and the qlHC dot count with concentration of qlHC chemical substrate. As illustrated by the results of this study, qlHC can be used as a tunable system, where the qlHC dot count is directly proportional to the concentration to the dot-generating component. This allows for quantitative HER2 detection with a high signal / noise ratio at different protein expression levels.

[0151] Methods and Results

[0152] Samples from several formalin-fixed paraffin-embedded cell lines (MDA-MB-468, MDA-MB-231, MDA-MB-175, MDA-MB-453, and SK-BR-3) with different levels of HER2 expression (classified in a range from “0” to 3+”) were stained with IHC using a HercepTest™ mAb (Dako Omnis) or qlHC (FIG. 13). The sample preparation and staining protocol used for this study is described in Jensen 2017.

[0153] The number of qlHC dots per cell line were counted and the mean result for each cell line is shown in the graph provided as FIG. 14, with shading around each mean to show the 95% confidence interval. FIG. 15 provides representative images of qlHC-stained cells examined in this study, with the detected qlHC dots annotated as points (left) or with boxes (right). As illustrated by FIG. 14, a linear relationship exists between the mean dots / cell count and the level of HER2 expression in these cell lines and, as such, the present methods provide reliable and consistent quantitative expression information in FFPE cell lines that have different HER2 expression levels.* * *

[0154] While certain features and aspects have been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and / or any combination thereof. Further, while various methods and processes described herein may be described with respect to particular structural and / or functional components for ease of description, methods provided by various embodiments are not limited to any particular structural and / or functional architecture but instead can be implemented on any suitable hardware, firmware and / or software configuration. Similarly, while certain functionality is ascribed to certain system components, unless the context dictates otherwise, this functionality can be distributed among various other system components in accordance with the several embodiments.

[0155] Moreover, while the procedures of the methods and processes described herein are described in a particular order for ease of description, unless the context dictates otherwise, various procedures may be reordered, added, and / or omitted in accordance with various embodiments. Moreover, the procedures described with respect to one method or process may be incorporated within other described methods or processes; likewise, system components described according to a particular structural architecture and / or with respect to one system may be organized in alternative structural architectures and / or incorporated within other described systems. Hence, while various embodiments are described with or without certain features for ease of description and to illustrate exemplary aspects of those embodiments, the various components and / or features described herein with respect to a particular embodiment can be substituted, added and / or subtracted from among other described embodiments, unless the context dictates otherwise. Consequently, although several exemplary embodiments are described above, it will be appreciated that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

[0156] The descriptions of the various embodiments of the present disclosure have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0157] It is expected that during the life of a patent maturing from this application many relevant machine learning models will be developed and the scope of the term machine learning model is intended to include all such new technologies a priori.

[0158] As used herein the term “about” refers to + / - 10 %.

[0159] The terms “comprises”, “comprising”, “includes”, “including”, “having” and their conjugates mean “including but not limited to”. This term encompasses the terms “consisting of’ and “consisting essentially of’.

[0160] The phrase “consisting essentially of’ means that the composition or method may include additional ingredients and / or steps, but only if the additional ingredients and / or steps do not materially alter the basic and novel characteristics of the claimed composition or method.

[0161] As used herein, the singular form “a”, “an” and “the” include plural references unless the context clearly dictates otherwise. For example, the term “a compound” or “at least one compound” may include a plurality of compounds, including mixtures thereof.

[0162] The word “exemplary” is used herein to mean “serving as an example, instance or illustration”. Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments.

[0163] The word “optionally” is used herein to mean “is provided in some embodiments and not provided in other embodiments”. Any particular embodiment may include a plurality of “optional” features unless such features conflict.

[0164] Throughout this application, various embodiments may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the various embodiments described and / or claimed herein. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

[0165] Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging / ranges between” a first indicate number and a second indicate number and “ranging / ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and aremeant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

[0166] It is appreciated that certain features of the embodiments described and / or claimed herein, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

[0167] Although the disclosure has been described with reference to specific embodiments, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations thereof that fall within the spirit and broad scope of the present disclosure and claims.

[0168] It is the intent of the applicant(s) that all publications, patents and patent applications referred to in this specification are to be incorporated in their entirety by reference into the specification, as if each individual publication, patent or patent application was specifically and individually noted when referenced that it is to be incorporated herein by reference. In addition, a citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present application or any patent resulting therefrom. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is / are hereby incorporated herein by reference in its / their entirety.

Claims

CLAIMS1. A computer-implemented method of detecting objects in an image of a biological sample, the method comprising: obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; and detecting dots in said image of a biological sample using a trained artificial intelligence (“Al”) model pre-trained to detect dots.

2. The computer-implemented method of claim 1, further comprising generating one or more annotations associated with the image of a biological sample and with the dots detected in the detecting step.

3. The computer-implemented method of claims 1 or 2, wherein said one or more annotations indicate a location of respective one or more dots detected in the detecting step.

4. The computer-implemented method of claim 3, wherein the location of dots is a location in three spatial dimensions.

5. The computer-implemented method of any one of claims 2 to 4, further comprising: providing an input interface capable of enabling a human user to obtain an updated one or more annotations by: deleting an annotation from the one or more annotations, modifying an annotation among the one or more annotations, and / or adding an annotation to the one or more annotations; and storing the updated one or more annotations.

6. The computer-implemented method of claim 5, further comprising using the updated one or more annotations as a ground truth to train said Al model or another Al model for detecting dots.

7. The computer-implemented method of any one of claims 2 to 6, further comprising providing an annotation viewing interface configured to display said image of the biological sample together with said one or more annotations.

8. The computer-implemented method of claim 7, wherein the annotation viewing interface is a side-by-side viewing interface configured to display a first view showing a first- view image based on said image of the biological sample and including said one or more annotations overlaid beside a second view showing a second-view image of the biological sample.

9. The computer-implemented method of claim 8, wherein said second-view is captured with settings different from those of said image of the biological sample and includes annotations corresponding to said one or more annotations of the first-view image.

10. The computer-implemented method of claims 8 or 9, wherein said second-view image of the biological sample is based on or includes a z-stack including two or more mutually different focal plane images of the same field of view, FOV, of the biological sample.

11. The computer-implemented method of any one of claims 8 to 10, wherein said second-view image is obtained by flattening or 3D deconvolution of the z-stack including combining the two or more focal plane images into a single image of the biological sample.

12. The computer-implemented method of any one of claims 7 to 11, wherein the annotation viewing interface is configured to view each of said one or more annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots.

13. The computer-implemented method of claim 12, wherein the preconfigured shape is a rectangle or a square positioned such that the one of the detected dots is in its geometric center.

14. The computer-implemented method of any one of claims 7 to 13, wherein the annotation viewing interface is configured to enable switching between focal planes of the z- stack.

15. The computer-implemented method of any one of claims 7 to 14, wherein the annotation viewing interface is configured to: enable a user to select a focal plane out of the z-stack and to mark a dot in the selected focal plane, and store an identification of said selected focal place in association with the marked dot.

16. The computer-implemented method of any one of claims 7 to 15, wherein the annotation viewing interface is configured to enable switching between: i) viewing each of said annotations as a graphics of a dot co-located with one of the detected dots, and ii) said viewing each of said annotations as a graphics of an unfilled contour of a preconfigured shape surrounding one of the detected dots.

17. The computer-implemented method of claim 16, wherein said switching is triggered by changing a field of view of said image of the biological sample.

18. The computer-implemented method of any one of claims 1 to 17, wherein said detection of dots further includes categorizing or filtering said one or more dots based on additional cellular or non-cellular markers.

19. The computer-implemented method of any one of claims 1 to 18, wherein said detection of dots further comprises categorizing or filtering said one or more dots based on additional one or more images of the biological sample including one or more images of the same slide or of consecutive slides.

20. A computer-implemented method of training an artificial intelligence (“Al”) model for detecting objects in an image of a biological sample, the method comprising: obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; obtaining one or more annotations associated with the image of the biological sample and with the dots; and adapting one or more parameters of said Al model according to said image of a biological sample and, as a ground truth, said one or more annotations into the Al model.

21. The computer-implemented method of claim 20, wherein the method further comprises obtaining an image of a negative control biological sample stained with an IHC or fluorescence-based approach; and adapting one or more parameters of said Al model according to said image of a negative control biological sample and, as a ground truth, no annotations or an annotation indicating no dots.

22. The computer-implemented method of claims 20 or 21, wherein said one or more annotations indicate location of respective one or more dots detected in the detecting step.

23. The computer-implemented method of claim 22, wherein the location of dots is a location in three spatial dimensions.

24. A computer-implemented method of training a first artificial intelligence (“Al”) model for detecting objects in an image of a biological sample, the method comprising: obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots; obtaining one or more annotations associated with the image of the biological sample and with the dots, wherein said obtaining includes: obtaining an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots, and detecting dots in said image of a biological sample using a trained second Al model pre-trained to detect dots; and adapting one or more parameters of said first Al model according to said image of a biological sample and, as a ground truth, said one or more annotations into the first Al model.

25. A computer program stored on a non-transitory medium and including instructions which when executed on one or more processors causes the one or more processors to perform the method of any one of claims 1 to 24.

26. A detection device comprising: a data interface; a storage; and processing circuitry that, in operation, Iobtains over said data interface an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots, detects dots in said image of a biological sample using a trained artificial intelligence (“Al”) model pre-trained to detect dots, and stores indication of locations of said respective detected dots as one or more annotations.

27. The detection device of claim 26, further comprising: an input interface capable of enabling a human user to update one or more annotations by: deleting an annotation from the one or more annotations, modifying an annotation among the one or more annotations, and / or adding an annotation to the one or more annotations; and wherein the processing circuitry, in operation, stores the updated one or more annotations in the storage.

28. The detection device of claim 26 or 27, further comprising: an annotation viewing interface configured to display said image of the biological sample together with said one or more annotations.

29. A training device comprising: a data interface; a storage; and processing circuitry that, in operation: obtains an image of a biological sample stained with a quantitative approach converting antibody / antigen complexes into dots via the data interface; obtains one or more annotations associated with the image of the biological sample and with the dots via the data interface; adapts one or more parameters of an artificial intelligence (“Al”) model according to said image of a biological sample and, as a ground truth, said one or more annotations, thereby training the Al model for detecting objects in an image of a biological sample.

30. A computer-implemented method of measuring antibody expression, comprising:obtaining an image of a biological sample comprising one or more cells stained with a quantitative approach converting antibody / antigen complexes into dots; detecting dots in the image of the biological sample using a trained artificial intelligence (“Al”) model pre-trained to detect dots; determining a number of dots per cell in the image of the biological sample using the trained Al model; determining or estimating a level of antibody expression in the biological sample based on the determined number of dots per cell.

31. The method of claim 30, wherein the level of antibody expression in the biological sample is determined or estimated based on a mean number of dots per cell.

32. The method of claim 30 or claim 31, wherein the biological sample comprises formalin- fixed and paraffin-embedded cells.

Images