Automated chromogenic in situ hybridization quantification and immunohistochemistry evaluation from whole slide images

Abstract

A system and method are provided that uses a control tissue to perform color calibration and automate detection. The system can receive calibration images from the scanner. Each calibration image can depict a portion of a stained tissue sample having a known score value. The system can determine a correction function for the scanner based on the plurality of calibration images. The system can receive unscored slide images depicting the portions of the stained tissue. The unscored slide images can be captured by the scanner. The system can display scores for the unscored slide images generated using the correction function for the scanner in a graphical user interface of an application executed by the system.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 309,933, filed February 14, 2022, the entire contents of which are incorporated herein by reference for all purposes.

[0002] This application relates generally to color calibration of whole slide images to reduce inaccuracies in determining cellular or tissue abnormalities. [Background technology]

[0003] Color differences detected in stains can be used to detect the presence of abnormal tissue. However, the quality of slide images can vary across machines, locations, and scanning techniques. Thus, a slide initially scanned at a first scanning location may appear different in color than when the same slide is scanned at a second scanning location, or at the same scanning location in the future. These inconsistencies pose challenges when attempting to automatically detect tissue or cellular abnormalities, such as markers for cancer cells. Summary of the Invention

[0004] The disclosed systems and methods provide techniques for calibrating color differences between different scanning devices and within slides over different time periods by using control tissues and / or calibration slides. A control tissue or calibration slide can be scanned at a specific time, and a color map can be generated that maps different colors to corresponding DAB (3,3'-diaminobenzidine) concentrations. The color map can be stored in association with a timestamp of the day the tissue was scanned and the scanned slide image of the control tissue or calibration slide. The color map (e.g., provided on a calibration slide) can be used to calibrate other scans of other tissue samples, automating the detection of cellular features of interest. By utilizing this technique, color differences between institutions and scanners do not affect the accuracy of the automated detection. Thus, the systems and methods described herein provide improvements to scanning technology.

[0005] At least one aspect of the present disclosure relates to a method of performing color calibration using a control tissue or calibration slide to automate detection. The method may be performed, for example, by one or more processors coupled to a non-transitory memory. The method includes receiving a plurality of calibration images from a scanner, each of the plurality of calibration images showing a portion of a respective stained tissue sample having a known score value. The method includes determining, by the one or more processors, a correction function for the scanner based on the plurality of calibration images. The method includes receiving an unscored slide image showing the portion of the stained tissue, the unscored slide image being captured by the scanner. The method includes showing a score of the unscored slide image generated using the correction function for the scanner in a graphical user interface of an application executing on the one or more processors.

[0006] In some implementations, the method includes determining a score for the unscored slide image based on the intensity profile. In some implementations, each of the plurality of calibration images is selected to correspond to a respective score value. In some implementations, determining the intensity profile includes performing a color calibration process. In some implementations, the color calibration process includes calculating a color correction matrix to be applied to the unscored slide image. In some implementations, determining the intensity profile includes determining one or more color thresholds corresponding to the plurality of calibration images.

[0007] In some implementations, the method includes annotating, by one or more processors, the unscored slide images based on the intensity profile. In some implementations, determining the intensity profile includes determining one or more intensity values ​​over multiple days. In some implementations, the method includes receiving a second unscored slide image generated by a second scanner during a second time period. In some implementations, the method includes identifying a second intensity profile associated with the second scanner and the second time period. In some implementations, the method includes presenting the second unscored slide image along with the second intensity profile to an application. In some implementations, the multiple calibration images correspond to respective HER2 scores.

[0008] At least one other aspect of the present disclosure relates to a system for performing color calibration using a control tissue or calibration slide to automate detection. The system may include one or more processors coupled to a non-transitory memory. The system may receive a plurality of calibration images from a scanner, each of the plurality of calibration images showing a portion of a respective stained tissue sample having a known score value. The system may determine a scanner correction function based on the plurality of calibration images. The system may receive an unscored slide image showing a portion of the stained tissue, the unscored slide image being captured by the scanner. The system may display a score of the unscored slide image generated using the scanner correction function in a graphical user interface of an application executing on the one or more processors.

[0009] In some implementations, the system can determine a score for the unscored slide image based on the correction function. In some implementations, each of the plurality of calibration images is selected to correspond to a respective score value. In some implementations, to determine the correction function, the system can perform a color calibration process. In some implementations, the system can calculate a color correction matrix to be applied to the unscored slide image. In some implementations, to determine the correction function, the system can determine one or more color thresholds corresponding to the plurality of calibration images. In some implementations, the system can annotate the unscored slide image based on the correction function. In some implementations, to determine the correction function, the system can determine one or more intensity values ​​over multiple days.

[0010] In some implementations, the system can receive a second unscored slide image generated by a second scanner during a second time period. In some implementations, the system can identify a second correction function associated with the second scanner and the second time period. In some implementations, the system can present the second unscored slide image in application with the second correction function. In some implementations, each of the multiple calibration images corresponds to a respective HER2 score.

[0011] These and other aspects and implementations are described in detail below. The foregoing information and the following detailed description, including illustrative examples of the various aspects and implementations, provide an overview or framework for understanding the nature and characteristics of the aspects and implementations as claimed. The drawings provide an explanation and further understanding of the various aspects and implementations, and are incorporated in and constitute a part of this specification. It will be readily understood that features described in the context of one aspect of the invention can be combined with other aspects. The aspects can be implemented in any convenient form, for example by a suitable computer program that can be carried on a suitable carrier medium (computer readable medium), which can be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). The aspects can also be implemented using a suitable apparatus, which can be in the form of a programmable computer executing a computer program configured to implement the aspects.

[0012] The above and other objects, aspects, features and advantages of the present disclosure will become more apparent and will be better understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1] According to one or more implementations, an example of a system that performs color calibration using a control tissue or calibration slide to automate detection is shown. [Diagram 2] FIG. 1 illustrates an example of a system architecture diagram illustrating the process of processing stained slide images using the color calibration techniques described herein, according to one or more implementations. [Diagram 3] According to one or more implementations, a comparison between different scans of the same stained slide performed on different scanners is shown. [Figure 4] 1 shows a side-by-side comparison of one or more implementations of an immunohistochemistry (IHC) calibration slide set for use in connection with the present technology. [Diagram 5] According to one or more implementations, views of a calibration slide containing various staining intensities are shown. [Figure 6] According to one or more implementations, a color map that maps staining intensity to color and a side-by-side comparison of the corresponding stained slide is shown. [Figure 7] 1 shows a diagram of estimated 3,3′-diaminobenzidine (DAB) concentration used as a threshold for slide images being scored, according to one or more implementations. [Figure 8] According to one or more implementations, a side-by-side comparison of images that have been scored for staining with different intensities is shown. [Figure 9] According to one or more implementations, a color bar (eg, color mapping) is shown displayed in a user interface along with a whole slide image (WSI) of a tissue sample. [Figure 10] According to one or more implementations, an example of calibrator results over a five day period is shown, including color mapping between DAB concentration and color intensity. [Figure 11] According to one or more implementations, an example of cell palette results over a 5-day period is shown, including color mapping between DAB concentration and color intensity. [Figure 12] According to one or more implementations, an example of assigning an intensity profile (eg, color mapping) to one or more slide images is shown. [Figure 13]1 illustrates an example of a color bar displayed on a user interface that includes an image of a WSI having different intensities, according to one or more implementations. [Figure 14] According to one or more implementations, an example of how color calibration can be performed using a control tissue or calibration slide to automate detection is shown. [Figure 15] FIG. 2 is a block diagram of a server system and a client computer system in accordance with an exemplary embodiment. [Figure 16] 1 shows an example view of a calibration slide containing various staining intensities, according to one or more implementations. [Figure 17] 1 shows an example view of a color swatch slide containing various reference colors, according to one or more implementations. [Figure 18A] FIG. 2 shows a flow diagram of an example of a process for color and intensity standardization, including scanner calibration and stain calibration, that may be implemented by the system of FIG. 1 according to one or more implementations. [Figure 18B] FIG. 2 shows a flow diagram of an example of a process for color and intensity standardization, including scanner calibration and stain calibration, that may be implemented by the system of FIG. 1 according to one or more implementations. [Figure 19] 2 shows a flow diagram of an example of a process that may be implemented by the system of FIG. 1 for calibrating a scanner, according to one or more implementations. [Figure 20A] 2 shows a flow diagram of an example process for stain calibration that may be implemented by the system of FIG. 1 according to one or more implementations. [Figure 20B] 2 shows a flow diagram of an example process for stain calibration that may be implemented by the system of FIG. 1 according to one or more implementations. [Figure 21] According to one or more implementations, slide images, such as stained breast cancer tissue images, plots, and mean and variance values, respectively, are shown. [Figure 22] According to one or more implementations, slide images, such as HER IHC stained breast cancer tissue images, plots, and mean and variance values, respectively, are shown. [Figure 23]According to one or more implementations, examples of slide images, such as IHC calibrator images, plots, and mean and variance values ​​are shown, respectively. [Figure 24] According to one or more implementations, a comparison of dE* maps with and without brown in a HER2 IHC stained breast cancer tissue image is shown. [Diagram 25] According to one or more implementations, a comparison of dE* maps with and without brown color in an IHC stained calibrator image is shown. [Figure 26] According to one or more implementations, a comparison of dE* maps with and without brown color in H&E stained breast cancer tissue images is shown. [Figure 27A] 1 shows a HER2 IHC breast cancer image scanned using a 40x objective with an intensity color bar, according to one or more implementations. [Figure 27B] 1 shows a HER2 IHC breast cancer image scanned using a 40x objective with an intensity color bar, according to one or more implementations. [Figure 28] According to one or more implementations, a comparison of a tissue image and a dE* map is shown. [Figure 29] According to one or more implementations, a HER2 IHC stained sample is shown using a 40x objective. [Diagram 30] According to one or more implementations, HER2 IHC stained breast cancer images scanned at 0.23 μm / pixel resolution (equivalent to 40x) are shown for 0, 1+, 2+, and 3+ cases, respectively. [Diagram 31] 1 illustrates an example of an IHC calibrator with HER2 IHC staining, according to one or more implementations. [Diagram 32] 1 illustrates a flow diagram of an example process implemented by the system described in connection with FIG. 1 to perform normalization between slide images, according to one or more implementations. [Diagram 33] 1 illustrates a data flow diagram of a process for calculating a stain intensity threshold from a tissue image, according to one or more implementations. [Diagram 34]FIG. 33 shows a data flow diagram of a first implementation of the process described in connection with FIG. 32, which includes correcting the reference threshold to accommodate the original tissue image, according to one or more implementations. [Diagram 35] FIG. 33 shows a data flow diagram of a second implementation of the process described in connection with FIG. 32, including color and intensity correction of tissue images, according to one or more implementations. [Diagram 36] According to one or more implementations, an example of color correction for 2+ is shown. [Figure 37] According to one or more implementations, an example of color correction for 3+ is shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The present technology allows color calibration to be performed using control tissues or calibration slides, improving accuracy when automating detection between different institutions or at the same institution over different time periods. It should be understood that the various concepts introduced above and described in more detail below can be implemented in various ways, as the disclosed concepts are not limited to any particular implementation method. Examples of specific implementations and applications are provided primarily for illustrative purposes. To better understand the present disclosure, a brief overview of the detailed description section may be helpful.

[0015] Immunohistochemistry (IHC) staining is used as a process to detect markers of various cellular abnormalities, including cancer. Generally, IHC analysis measures the intensity of staining in WSI to detect the presence (or absence) of a target protein. One such protein is the HER2 protein. IHC testing for HER2 protein measures the amount of HER2 receptor protein on the cell surface of breast cancer tissue samples. This measurement is often reflected in the form of a score that can range from 0 to 3+. If the score is 0 to 1+, the tissue sample can be called "HER2 negative." If the score is 2+, the tissue sample can be called "borderline." If the score is 3+, the tissue sample can be called "HER2 positive." HER2 negative, borderline, and HER2 positive breast cancers may each have different treatment plans. Therefore, it is important to ensure the accuracy of these scores when determining the best course of treatment for a patient.

[0016] In HER2 IHC staining, DAB stains the membranous regions brown, and hematoxylin stains the nuclear regions blue or purple for tissue counterstaining. During image analysis, a score is classified based on the intensity and percentage of membranous staining. HER2 status is usually assessed visually by pathologists using a microscope, but this tends to be subjective, and reproducibility of interpretation is a significant issue. Modern guidelines recommend image analysis using digital slides (e.g., WSI) to increase the objectivity and accuracy of HER2 IHC assessment.

[0017] However, the calculation of these scores depends on the quality of the WSI scan. When analyzing stained tissue samples, various factors can affect the accuracy of the scoring process. Generally, the HER2 score for a particular slide is calculated based on the staining intensity present in the stained WSI. However, the intensity of the WSI color can vary for various reasons. For example, the intensity can vary depending on the institution, the date and time of staining, the specific scanner used to perform the scan or the scanning protocol used in the scan, etc.

[0018] These differences can result in large differences in scores. An example of such differences is shown by a side-by-side comparison of the same tissue sample captured using different scanners shown in Figures 3A and 3B. As shown in Figure 3A, the scan result is very light and is likely to result in a score in the 0-1+ range (e.g., HER2 negative). However, the same tissue sample scanned using a different scanner shown in Figure 3B is much darker and is likely to result in a score of 2+ or 3+, indicating a borderline or HER2 positive case. Each of these different results will result in a different treatment course for the patient.

[0019] Therefore, color intensity changes are a significant issue in pathology. The variability of histochemical staining is known to affect accuracy and reproducibility in clinical practice and research. Small differences in staining intensity, especially in the case of membranous staining, can greatly affect the interpretation of IHC slides and lead to differences in diagnostic and therapeutic outcomes. Furthermore, color intensity changes are not limited to irregularities within the staining process. Digitization of slides by WSI scanners can further introduce color changes. Such changes affect image analysis for HER2 testing.

[0020] It would therefore be advantageous for the system to automatically correct these differences so that scanning the same tissue sample at different institutions, time periods, or scanning techniques does not affect HER2 scoring or other processes that rely on determining the intensity of IHC staining in WSI. The disclosed system and method solves these and other problems by utilizing control tissues or calibration slides to perform a color calibration process of tissues scanned with a particular scanner, as described herein. The control tissues can be a set of calibration slides used to calibrate the scanning device for a particular staining process (e.g., HER2 testing). The calibration slides are used to create a color mapping of a particular scanner at a particular time period (mapping various staining intensities (e.g., colors) to known HER2 scores (e.g., 0, 1+, 2+, 3+, etc.)). This color mapping is also used to score other stained slides captured at the same time period with the same scanner.

[0021] The color mapping may be created by testing the calibration slide over a period of time (e.g., 5 days, 1 week, etc.). These and other features are described in more detail herein. The calibration slide may include microbeads coated with different concentrations of peptides, for example, to achieve different colors or shades. The calibration slide may include microbeads coated with different concentrations of peptides, for example, to achieve different colors or shades. As described herein, the calibration slide may be stained along with the tissue sample slide to obtain a color intensity of staining that correlates with the color intensity of the staining of the test slide. The color standardization or normalization methods described herein may be applied to address color variation issues. In IHC color standardization, the techniques described herein may be used to calibrate the intensity of DAB.

[0022] Referring to FIG. 1, an example of a system for performing color calibration and automating detection using a control tissue or a calibration slide is shown according to one or more implementations. In brief overview, the system may include at least one data processing system 105, at least one network 110 (which may be the same as or part of the network 1526 described below in connection with FIG. 15), and at least one computing device 120. The data processing system 105 may include at least one calibration image acquisition device 130, at least one intensity profile determination device 135, at least one slide image acquisition device 140, at least one image presentation device 145, and at least one score calculation device 150. In some implementations, the data processing system 105 may include a data storage 115, and in some implementations, the data storage 115 may be external to the data processing system 105. For example, if the data storage 115 is external to the data processing system 105, the data processing system 105 (or a component thereof) may communicate with the data storage 115 via the network 110. In some implementations, the data processing system 105 may implement or perform any of the functions and operations described herein.

[0023] Each of the components of system 100 (e.g., data processing system 105, network 110, computing device 120, calibration image capture device 130, intensity profile determiner 135, slide image capture device 140, image presentation device 145, score calculation device 150, etc.) may be implemented using hardware components or a combination of software and hardware components of a computing system (e.g., server system 1500, client computing system 1514, other computing systems described herein, etc.) as detailed herein in conjunction with FIG. 15. Each of the components of data processing system 105 may perform the functions detailed herein.

[0024] More specifically, the data processing system 105 may include at least one processor and a memory (e.g., processing circuitry). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more operations described herein. The processor may include a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like, or a combination thereof. The memory may include, but is not limited to, an electronic, optical, magnetic, or other storage or transmission device capable of providing program instructions to the processor. The memory may further include a floppy disk, a CD-ROM, a DVD, a magnetic disk, a memory chip, an ASIC, an FPGA, a read only memory (ROM), a random access memory (RAM), an electrically erasable programmable ROM (EEPROM), an erasable programmable ROM (EPROM), a flash memory, an optical medium, or other suitable memory from which the processor can read instructions. The instructions may include code in any suitable computer programming language. The data processing system 105 may include one or more computing devices or servers capable of performing various functions described herein. Data processing system 105 may include some or all of the components, and perform some or all of the functionality, of server system 1500 or client computing system 1514 described below in conjunction with FIG.

[0025] Network 110 may include computer networks such as the Internet, local, wide, metro, or other area networks, intranets, satellite networks, other computer networks such as voice or data cellular telephone communications networks, and combinations thereof. In some implementations, network 110 may be, be a part of, or include one or more aspects of network 1526 described in connection with FIG. 15. Data processing system 105 of system 100 may communicate with at least one computing device 120, for example, via network 110. Network 110 may be any form of computer network capable of relaying information between data processing system 105, computing device 120, and, in some implementations, one or more external or third-party computing devices, such as a web server. In some implementations, network 110 may include the Internet and / or other types of data networks, such as a local area network (LAN), wide area network (WAN), cellular network, satellite network, or other types of data networks.

[0026] The network 110 may further include any number of computing devices (computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within the network 110. The network 110 may further include any number of wired and / or wireless connections. Some or all of the computing devices described herein (e.g., data processing system 105, computing device 120, server system 1500, client computing device 1514, etc.) may communicate wirelessly (e.g., via WiFi, cellular, radio, etc.) using transceivers wired (e.g., via fiber optic cable, CAT5 cable, etc.) to other computing devices in the network 110. Some or all of the computing devices described herein (e.g., data processing system 105, computing device 120, server system 1500, client computing device 1514, etc.) may also communicate wirelessly with computing devices of the network 110 through proxy devices (e.g., routers, network switches, gateways).

[0027] The data storage 115 may be a database configured to store and / or maintain any of the information described herein. The data storage 115 may maintain one or more data structures, which may include, index, or otherwise store each of the values, pluralities, sets, variables, vectors, thresholds, or generated or determined information described herein. The data storage 115 may be accessed using one or more memory addresses, index values, or identifiers of any items, structures, or regions held in the data storage 115. The data storage 115 may be accessed over the network 110 by components of the data processing system 105 or other computing devices described herein. In some implementations, the data storage 115 may be internal to the data processing system 105. In some implementations, the data storage 115 may be external to the data processing system 105 and accessed over the network 110. The data storage 115 may be distributed across different computer systems or storage elements and may be accessed over the network 110 or a suitable computer bus interface.

[0028] Data processing system 105 may store some or all of the results of the calculations, determinations, selections, identifications, generation, construction, or calculations in one or more areas of memory of data processing system 105, or in one or more data structures indexed or identified with appropriate values ​​in data storage 115. Some or all of the values ​​stored in data storage 115 may be accessed by any computing device described herein, such as data processing system 105, to perform any of the functions or actions described herein.

[0029] Computing device 120 may include at least one processor and memory (e.g., processing circuitry). The memory may store processor-executable instructions that, when executed by the processor, cause the processor to perform one or more operations described herein. The processor may include a microprocessor, an ASIC, an FPGA, or the like, or a combination thereof. The memory includes, but is not limited to, electronic, optical, magnetic, or other storage or transmission devices capable of providing program instructions to the processor. The memory may further include floppy disks, CD-ROMs, DVDs, magnetic disks, memory chips, ASICs, FPGAs, ROMs, RAMs, EEPROMs, EPROMs, flash memories, optical media, or other suitable memories from which the processor can read instructions. The instructions may include code in any suitable computer programming language. Computing device 120 may include one or more computing devices or servers capable of performing various functions described herein. The computing device 120 may include some or all of the components, and perform some or all of the functionality, of the server system 1500 or the client computing system 1514 described below in conjunction with FIG.

[0030] The scanner device 125 may be any type of scanner device capable of generating a whole slide image (or a portion thereof) of tissue stained in an IHC analysis process. The scanner device 125 may be used to generate any type of scan of any type of IHC stain, such as membrane stain, nuclear stain, cytoplasmic stain, etc. The scanner device 125 may be communicatively coupled to the data processing system 105 via the network 110 or another type of communication bus or input / output interface. The scanner device 125 may transmit any of the scanner images generated by scanning the stained slide of the tissue sample in response to one or more requests from the data processing system 105, for example. In some implementations, the scanner device 125 may communicate with one or more computing devices 120 (e.g., via the network 110, a communication bus, or an input / output interface, etc.). The scanner device 125 may provide any scanned image to the computing device 120, which may transmit the image to the data processing system 105 for further analysis or processing.

[0031] The scanner device 125 can associate the generated scanner image with a timestamp or time period corresponding to the date and time the image was captured by the scanner device 125. The scanner device 125 can generate images of the slide at various magnification levels (e.g., 20x, 30x, 40x, 50x, 60x, etc.). The scanner device 125 can capture images at a predetermined resolution, for example, 0.13 microns / pixel (NA*0.95) or 0.24 microns / pixel (NA 0.75). Other magnifications and resolutions are possible. The image generated by the scanner device 125 can be a full slide image or a portion of a full slide image. In some implementations, the scanner device 125 can store the generated image in the data storage 125 along with corresponding metadata (e.g., timestamp, identifier of the scanner device 125, etc.). The data processing system 105 can utilize the images generated by the scanner device 125 to perform the operations described herein.

[0032] The calibration image acquisition unit 130 can receive one or more calibration images from the scanner device 125. Each of the one or more calibration images can show a portion of a respective stained tissue sample having a known score value. Each calibration image can be selected to correspond to a respective score value and can correspond to a respective slide scanned by the scanner device 125. Some examples of such calibration slides are shown in FIG. 4. In FIG. 4, certain slide images correspond to different days of the week. In some implementations, and as described in more detail herein, the calibration process described herein can be used to generate an intensity profile of the scanner device 125 over a predetermined period of time. In such implementations, the calibration slides can be selected to correspond to particular days of the week and include tissue samples representing predetermined scores (e.g., HER2 scores of 0, 1+, 2+, or 3+, etc.).

[0033] Referring back to FIG. 1 , the data processing system 105 can be communicatively coupled to the scanner device 125, which can capture and provide images of the calibration slide for further processing. In some implementations, the scanner device 125 can store the calibration image(s) in one or more data structures in the data storage 115. The one or more data structures containing the calibration images can be indexed by various parameters or characteristics of the slide, such as attributes of the sample (such as tissue type, sample donor identifier, or other type of biomedical image identifier), the time the calibration image was captured by the scanner device 125, and an identifier of the scanner device 125. The calibration image acquisition unit 130 can obtain one or more images from the data storage 115, the scanner device 125, or the computing device 120, and provide the calibration image to any component of the data processing system 105 for further processing.

[0034] The calibration image capture device 130 can identify, receive, or otherwise obtain one or more calibration images obtained from calibration tissue samples (e.g., selected to have a predetermined score value, such as a HER2 score). The calibration tissue samples can be taken from biopsies, biological tissue samples, archival tissue, or other biological material. Briefly referring to FIG. 5, an example of a calibration slide is shown that includes different cases corresponding to different score values. As shown in FIG. 5, the slide includes tissue stained with an IHC test stain (such as DAB). Each section of the slide corresponds to a HER2 score, from left to right, of 0, 1+, 2+, and 3+, respectively. Note that the stain color of each slide is dark brown for borderline and HER2 positive tissue samples. This example slide, or slides containing similar tissue, can be used to create calibration images for use in the processes described herein. In some implementations, the calibration slide includes a nanoparticle-based calibrator (such as a Boston Cell calibrator), a cell palette, and at least four representative HER2 breast cancer cases (e.g., corresponding to scores of 0, 1+, 2+, and 3+, respectively). A calibration image can represent such a calibration slide captured by the scanner device 125.

[0035] Referring back to FIG. 1, after receiving the calibration image, the calibration image acquisition device 130 can store the calibration image in one or more data structures in the data storage 115. The calibration image obtained by the calibration image acquisition device 130 can include any type of image file (e.g., JPEG, JPEG-2000, PNG, GIF, TIFF, etc.), including, for example, an SVS image file. The calibration image acquisition device 130 can open, analyze, or otherwise load the calibration image into the working memory of the data processing system 105 (e.g., accessible from any component of the data processing system 105, etc.) using slide analysis software. The slide analysis software is any type of software capable of opening, modifying, or displaying an image captured using the scanner device 125. The calibration image can be stored in the data storage 115 in association with a timestamp corresponding to the time the calibration image was captured by the scanner device 125 and an identifier for the scanner device 125.

[0036] The intensity profile determiner 135 can determine an intensity profile of the scanner device 125. The intensity profile is stored in the data storage 115 and can include a mapping between one or more score values ​​(e.g., HER2 score values) and respective color values. The intensity profile determiner 135 can determine an intensity profile based on one or more calibration images corresponding to the scanner device 125 and associate the intensity profile with a first time period (e.g., a time period associated with the time when the calibration image was created using the scanner device 125). To generate an intensity profile for a particular time period, the intensity profile determiner 135 can access one or more calibration images obtained from the scanner device 125. As described above, each calibration image can be stored in association with a known score value (e.g., a known HER2 score value determined by a physician or another offline process). The intensity profile determiner 135 can determine the staining profile by estimating the staining intensity of each control tissue or calibration slide used to generate the calibration image (e.g., each section of the WSI corresponds to a control tissue or calibration slide with HER2 scores of 0, 1+, 2+, and 3+).

[0037] In some implementations, the intensity profile determiner 135 can determine the intensity profile of the scanner device 125 over a period of multiple days. For example, multiple different calibration slides can be utilized to generate a daily color bar corresponding to the scanner device 125. The color bar can show a mapping between the HER2 score and the intensity level in the slide image. In some implementations, the intensity profile is populated with multiple color mappings corresponding to the time period (e.g., morning or evening) and day of the week when each calibration slide was scanned. Examples of these color bars that may be calculated based on the estimated DAB concentration in the calibration slide are shown in FIG. 10. A slide image corresponding to an exemplary cell palette that may be included in the calibration slide is shown in FIG. 11 in association with the respective color bars generated from the cell palette image. Additionally, FIG. 12 shows a comparison of the intensity values ​​of the calibrator, cell palette, and tissue sample corresponding to 0, 1+, 2+, 3+, and control HER2 scores.

[0038] Referring back to FIG. 1, in some implementations, the intensity profile determiner 135 can perform a color calibration process. The color calibration process can be performed, for example, by utilizing one or more portions of the WSI dedicated to the calibrator (which may include, for example, regions with a predetermined color or intensity). The color calibration technique can be used, for example, to generate a color calibration matrix, which can be applied to a slide image, such as a calibration image obtained by the calibration image acquisition device 130. In some implementations, a stain calibration process can also be performed. The color calibration technique can be the color calibration technique described in Bautista PA, Hashimoto N, Yagi Y. Color standardization in whole slide imaging using a color calibration slide. J Pathol Inform 2014;5(1)4, the contents of which are incorporated herein by reference. The color calibration matrix M can be applied to the red, green, and blue (R, G, B, or sometimes referred to as "RGB") values ​​of each pixel in the image to which the calibration matrix is ​​applied. An example of this operation is shown below, where the R, G, B values ​​of each pixel i are modified according to a matrix M:

number

[0039] After applying the calibration matrix to the slide image (e.g., a calibrator image or an unscored image), the intensity profile determiner 135 can determine the intensity profile of the slide image using the processes described herein. The generated intensity profile can be stored in association with an identifier of the scanner device 125 used to generate the intensity profile. An example of an image showing a color mapping of an intensity profile generated from a corresponding calibration image is shown in FIG. 6. As shown in FIG. 6, the intensity profile includes a color mapping between different color intensities and corresponding HER2 score values. From left to right, each slide image shows a 0 to 1+ HER2 score, a 2+ HER2 score, and a 3+ HER2 score.

[0040] Referring back to FIG. 1 , once the intensity profile of the scanner device 125 is generated, the unscored slide images can be analyzed using the intensity profile. The slide image acquirer 140 can receive the unscored slide images showing a portion of stained tissue, e.g., a sample of a breast cancer biopsy. The unscored slide images can be captured by and received from the scanner device 125 during a first time period. The slide image acquirer 140 can obtain the unscored slide images, e.g., by sending a request to the scanner device 125. In some implementations, the scanner device 125 can store the unscored slide image(s) in one or more data structures in the data storage 115. The one or more data structures containing the unscored slide images can be indexed by various parameters or characteristics of the slide, such as attributes of the sample (e.g., type of tissue, an identifier of the sample donor, or other type of biomedical image identifier), the time the calibration image was captured by the scanner device 125, and an identifier of the scanner device 125. The slide image acquisition device 140 obtains one or more unscored slide images from the data storage 115, the scanner device 125, or the computing device 120, and can provide the unscored slide images to any component of the data processing system 105 for further processing. The unscored slide images can show a portion of a respective stained tissue sample with an unknown score value. The stained tissue sample can be stained with, for example, a DAB solution.

[0041] The image presenter 145 can present the unscored slide image with the intensity profile corresponding to the first time period in a graphical user interface of an application executing on the data processing system 105. When the image presenter 145 receives the unscored slide image, it can present the unscored slide image in a slide viewer application. The image presenter 145 can present the unscored slide image in a user interface along with a color mapping (e.g., color bar) of the intensity profile, as shown, for example, in the user interfaces shown in FIGS. 9 and 13. In FIG. 9, the intensity profile color bar is displayed in association with a single slide image. As shown, the intensity profile color bar can be displayed as an overlay in the same display window as the unscored slide image. In FIG. 13, the intensity profile color bar is displayed in association with multiple slide images (e.g., here displayed as three separate slide images). In some implementations, the user interface of FIG. 13 can be used to display one or more separate portions of the same WSI, for example, at different magnification levels. This allows a physician to accurately assess the difference between the intensity color (and associated HER2 score value) and the portion of the WSI of the unscored sample.

[0042] Referring back to FIG. 1, the techniques described herein may be used to generate an intensity profile for any scanner device 125 over any given time period. Thus, the techniques described herein may be utilized to more accurately analyze unscored slide images by displaying a color bar of the intensity profile determined along with the unscored slide image (e.g., the entire slide image or a portion thereof). To do so, the image presenter 145 may receive the unscored slide images generated by the scanner device 125 during an identified time period. The time period and an identifier of the scanner device 125 may be received along with the unscored slide image, for example, in response to a request or as part of a data package. In some implementations, slide metadata (e.g., an identifier of the scanner device 125 and an identifier of the time period) may be received separately from the unscored slide image. Using the slide metadata, the image presenter 145 may identify the intensity profile determined for the scanner device 125 during the identified time period (e.g., stored in the data storage 115) and present the unscored slide image to an application using the identified intensity profile, as shown in FIGS. 9 and 13.

[0043] The score determiner 150 can utilize the intensity profile to display a color bar generated with the unscored full slide image, as well as to score the unscored slide image based on the intensity profile. To do so, the score determiner 150 can determine one or more color thresholds corresponding to each score value (e.g., each HER2 score value). The color thresholds can be defined, for example, based on the color mapping indicated in the intensity profile. The intensity value of each color can indicate a lower threshold, and intensity values ​​(e.g., pixel intensity values) greater than the threshold (e.g., up to the next threshold) indicate the corresponding score value. The score determiner 150 scans each pixel in the slide image based on the color thresholds in the slide image and determines an average staining intensity value. The average staining intensity value can be compared to the mapping between the intensity value and the corresponding HER2 score. An appropriate HER2 score indicated in the mapping can then be selected based on the average staining intensity, and the score value can be associated with the unscored slide image in the data storage 115.

[0044] The application executed by the image presentation device 145 may allow modification of these thresholds, as shown in FIG. 7, where the user may modify these thresholds (e.g., depending on the detected staining intensity) and set the annotation color corresponding to each score value. These annotation colors may be represented as a gradient, with gradual fading in intensity (e.g., from dark yellow to light yellow) based on the intensity of the HER2 score (e.g., as shown, a relatively high intensity 1+ score is assigned light yellow and a low intensity 1+ score is assigned dark yellow). As described herein, the intensity mapping may be selected at least in part based on the estimated DAB concentration detected in the slide image.

[0045] Referring back to FIG. 1, after these colors are selected, the image presentation device 145 can scan each pixel in the slide image, obtain a staining intensity value (e.g., pixel color, etc.) for each pixel, and assign an annotation color to that pixel based on the annotation mapping. The result of this process is shown in FIG. 8. As shown in FIG. 8, areas of the unscored slide image that correspond to the intensity values ​​indicated in the annotation mapping are annotated with the corresponding annotation color. Three cases are shown, where the 1+ case is mostly not annotated with some small parts annotated in yellow (representing the 1+ case), the 2+ case is mostly annotated in yellow and orange, and the 3+ case is mostly annotated in red. In some implementations, a user of the application can turn annotations on or off using one or more interactive user interface elements (not shown).

[0046] Referring to FIG. 2, an example of a system architecture diagram 200 is shown illustrating a process for processing stained slide images using the color calibration techniques described herein, according to one or more implementations. This example architecture can be used to perform the processes and techniques described herein. As shown, first, a sample is collected (e.g., from a biopsy), and a slide is created and stained. The stained slide is then sent to various scanning devices to generate an image of the slide (e.g., a full slide image). The slide image (unscored slide image or calibration image) can be used in the techniques described herein or as part of various other processes, such as chromogenic in situ hybridization (CISH), fluorescent in situ hybridization (FISH), IHC testing, H&E testing, etc. The control tissue or calibration slide based scoring and calibration techniques described herein can be utilized in any number of these techniques or processes that include these techniques. As shown, various other processes, such as artificial intelligence based processes or segmentation based processes, can be used to aid in the detection and treatment of cancer. The scoring information for a particular slide image can be included in the patient case report associated with that slide image.

[0047] 14, a flow diagram of an exemplary method 1400 of whole slide image analysis and quality control is shown, according to an exemplary embodiment. The method 1400 may be performed by any computing device described herein, including, for example, data processing system 105. Briefly, in performing the method 1400, a data processing system (such as, for example, data processing system 105) may receive a calibration image from a scanner device (e.g., scanner device 125) (step 1405), determine an intensity profile for the scanner device (step 1410), receive unscored slide images from the scanner device (step 1415), and present the unscored slide images along with the intensity profile (step 1420).

[0048] In further detail of the method 1400, in step 1405, the data processing system can receive a calibration image from a scanner device (e.g., scanner device 125). Each of the one or more calibration images can show a portion of a respective stained tissue sample with a known score value. Each calibration image can be selected to correspond to a respective score value and can correspond to a respective slide scanned by the scanner device. In some implementations, the scanner device can store the calibration image(s) in one or more data structures in a data storage (e.g., data storage 115) of the data processing system. The one or more data structures containing the calibration images can be indexed by various parameters or characteristics of the slide, such as attributes of the sample (such as a type of tissue, an identifier of the sample donor, or other type of biomedical image identifier), a time when the calibration image was captured by the scanner device, and an identifier of the scanner device. The data processing system can obtain one or more images from the data storage, the scanner device, or other computing device(s) (e.g., computing device(s) 120) and provide the calibration images to any component of the data processing system for further processing.

[0049] The data processing system can store the calibration image in one or more data structures in the data storage. The calibration image obtained by the data processing system can include any type of image file (e.g., JPEG, JPEG-2000, PNG, GIF, TIFF, etc.), including, for example, an SVS image file. The data processing system can open, analyze, or otherwise load the calibration image into the working memory of the data processing system (e.g., accessible to any component of the data processing system, etc.) using slide analysis software. The slide analysis software is any type of software capable of opening, modifying, or displaying an image captured using a scanner device. The calibration image can be stored in the data storage in association with a timestamp corresponding to the time the calibration image was captured by the scanner device and an identifier for the scanner device.

[0050] In step 1410, the data processing system can determine a correction function (sometimes referred to as an intensity profile) for the scanner device. The intensity profile can be stored in a data storage and used to generate a mapping between one or more score values ​​(e.g., HER2 score values) and respective color values. In some implementations, the intensity profile can include one or more data structures that store a mapping or relationship between one or more score values ​​(e.g., HER2 score values) and respective color values. The data processing system can determine the intensity profile based on one or more calibration images corresponding to the scanner device and associate the intensity profile with a first time period (e.g., a time period associated with the time when the calibration image was created using the scanner device). To generate an intensity profile for a particular time period, the data processing system can access one or more calibration images obtained from the scanner device. As described above, each calibration image can be stored in association with a known score value (e.g., a known HER2 score value determined by a physician or another offline process). The data processing system can determine the staining profile by estimating the staining intensity of each control tissue of the calibration slide(s) or calibration image (e.g., each section of the WSI corresponds to a calibration slide(s) or control tissue with HER2 scores of 0, 1+, 2+ and 3+).

[0051] In some implementations, the data processing system can determine the intensity profile of the scanner device over a period of multiple days. For example, multiple different calibration slides can be utilized to generate a daily color bar corresponding to the scanner device. The color bar can show a mapping between HER2 scores and intensity levels in the slide image. In some implementations, the intensity profile is populated with multiple color mappings corresponding to the time period (e.g., morning or evening) and day of the week when each calibration slide was scanned. In some implementations, the data processing system can perform a color calibration process. The color calibration process can be performed, for example, by utilizing one or more portions of the WSI dedicated to the calibrator (e.g., which may include areas with a predetermined color or intensity). The color calibration technique can be used, for example, to generate a color calibration matrix, which can be applied to a slide image, such as a calibration image obtained by the data processing system. In some implementations, a stain calibration process can also be performed. After applying the calibration matrix to the slide image (e.g., a calibrator image or an unscored image), the data processing system can determine the intensity profile of the slide image using the process described herein. The generated intensity profile can be stored in association with an identifier of the scanner device used to generate the intensity profile. In some implementations, the data processing system can determine the scanner correction function (and any values ​​that may be included in the intensity profile) using the techniques described in connection with Figures 20A, 20B, 32, 33, 34, and 35, among other techniques described herein.

[0052] In step 1415, the data processing system may receive an unscored slide image from the scanner device (step 1415). The data processing system may receive an unscored slide image showing a portion of stained tissue, e.g., a sample of a breast cancer biopsy. The unscored slide image may be captured by and received from the scanner device during a first time period. The data processing system may obtain the unscored slide image, e.g., by sending a request to the scanner device. In some implementations, the scanner device may store the unscored slide image(s) in one or more data structures in a data storage. The one or more data structures containing the unscored slide images may be indexed by various parameters or characteristics of the slide, such as attributes of the sample (e.g., type of tissue, sample donor identifier, or other type of biomedical image identifier), the time the calibration image was captured by the scanner device, and an identifier of the scanner device. The data processing system may obtain one or more unscored slide images from a data storage, a scanner device, or other computing device(s) in communication with the data processing system, and provide the unscored slide image to any component of the data processing system for further processing. The unscored slide images can show a portion of each stained tissue sample with an unknown score value. The stained tissue samples can be stained with, for example, a DAB solution.

[0053] In step 1420, the data processing system can present the unscored slide image along with the intensity profile. Upon receiving the unscored slide image, the data processing system can display the unscored slide image in a slide viewer application. The data processing system can display the unscored slide image along with a color mapping (e.g., color bar) of the intensity profile in a user interface. The techniques described herein can be used to generate an intensity profile of any scanner device over any predetermined time period. Thus, the techniques described herein can be utilized to more accurately analyze the unscored slide image by displaying a color bar of the determined intensity profile along with the unscored slide image (e.g., the entire slide image or a portion thereof). To do so, the data processing system can receive the unscored slide image generated by the scanner device during the identified time period. The time period and the scanner device identifier can be received along with the unscored slide image, for example, in response to a request or as part of a data package. In some implementations, slide metadata (e.g., scanner device identifier and time period identifier) ​​can be received separately from the unscored slide image. Using the slide metadata, the data processing system can identify an intensity profile (e.g., including a correction function that is stored in data storage) determined for the scanner device for the identified time period and present the unscored slide image to the application using the identified intensity profile, as shown in Figures 9 and 13. In some implementations, the data processing system can execute the correction function to determine a score (e.g., a HER2 score) for the unscored slide image and display the score in a graphical user interface.

[0054] The aforementioned techniques may be utilized to perform color calibration using control tissues or calibration slides and automate detection. In some implementations, the data processing system may perform automatic correction of colors present in unscored slide images using correction functions using the techniques described in connection with Figures 20A, 20B, 32, 33, 34, and 35, and other techniques described herein. The color of the whole slide image may be highly dependent on the scanner system used to prepare the slide image. Slide preparation may include staining of the tissue. Color differences may occur even when standard staining protocols are followed, and small differences are important in many IHC tests that evaluate staining intensity. Each laboratory needs to understand the staining quality and characteristics so that color differences can be corrected to perform a proper evaluation. The present technology provides an IHC calibration tool for evaluation based on whole slide images. Experimental results are provided to evaluate the current technology and see if it can help in the process of image-based manual / automated IHC evaluation.

[0055] The experimental results were generated using the nanoparticle-based calibrator, the Cell Palette, and a dataset of six slides each containing four typical HER2 0-3+ breast cancer cases from our institution. 28 datasets were obtained in 10 weeks. The calibrator was used to assess the staining quality, and the Cell Palette was used to assist the visual assessment. Staining intensity was estimated from the WSI. To visualize the staining changes and compare the performance of each tool, an intensity score color bar was created based on the estimated staining intensity. Furthermore, a threshold value was used for the automated HER2 score assessment to check whether the correct score was obtained for typical cases.

[0056] Color difference and staining intensity were properly visualized, and the automatic score evaluation was accurate. The intensity score color bar was shared with the corresponding laboratory to indicate the staining status (stability). Pathologists can visually use the intensity score color bar for manual evaluation of HER2. The 3+ cells on the cell palette are darker than the typical 3+ cases in our hospital, so the percentage of positive cells tends to be underestimated. On the other hand, the calibrator is used for purposes other than its original purpose, so the correspondence with the tissue is not always linear, and correction taking these into account is desirable.

[0057] This technology helps to measure (or quantify) the color difference and staining intensity, allowing for accurate assessment of any WSI in IHC scores, and can be used to manually or automatically assess any differences.

[0058] Various operations described herein may be implemented on a computer system. FIG. 15 shows a simplified block diagram of a representative server system 1500, a client computer system 1514, and a network 1526 that may be used to implement certain embodiments of the present disclosure. In various embodiments, the server system 1500 or a similar system may implement the services or servers described herein, or portions thereof. The client computer system 1514 or a similar system may implement the clients described herein. The system 100 described herein may be similar to the server system 1500. The server system 1500 may have a modular design incorporating multiple modules 1502 (e.g., blades in a blade server embodiment), and although two modules 1502 are shown at the same time, any number of modules may be provided. Each module 1502 may include a processing unit(s) 1504 and local storage 1506.

[0059] The processing unit(s) 1504 may include a single processor, which may have one or more cores, or multiple processors. In some embodiments, the processing unit 1504 may include a general-purpose primary processor as well as one or more special-purpose co-processors, such as a graphics processor, digital signal processor, etc. In some embodiments, some or all of the processing unit 1504 may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, such integrated circuits execute instructions stored in the circuitry itself. In other embodiments, the processing unit(s) 1504 may execute instructions stored in local storage 1506. The processing unit(s) 1504 may include any type of processor in any combination.

[0060] The local storage 1506 may include volatile storage media (e.g., DRAM, SRAM, SDRAM, etc.) and / or non-volatile storage media (e.g., magnetic disks, optical disks, flash memory, etc.). The storage media incorporated in the local storage 1506 may be fixed, removable, or upgradeable as needed. The local storage 1506 may be physically or logically divided into various sub-units such as system memory, read only memory (ROM), and persistent storage devices. The system memory may be a volatile read / write memory such as a read / write memory device or a dynamic random access memory. The system memory may store some or all of the instructions and data that the processing unit(s) 1504 need at run time. The ROM may store static data and instructions needed by the processing unit(s) 1504. The persistent storage device is a non-volatile read / write memory device that can store instructions and data even when the module 1502 is powered off. As used herein, the term "storage medium" includes any medium on which data may be stored indefinitely (subject to overwriting, electrical disturbances, power loss, etc.), but does not include carrier waves and transitory electronic signals propagated over wireless or wired connections.

[0061] In some embodiments, local storage 1506 may store one or more software programs executed by processing unit(s) 1504 (such as an operating system and / or programs implementing various server functions, such as the functionality of system 100 of FIG. 1 or any other system described herein, or any other server(s) associated with system 100 or any other system described herein).

[0062] "Software" generally refers to a sequence of instructions that, when executed by the processing unit(s) 1504, cause the server system 1500 (or portions thereof) to perform various operations, and thus defines one or more specific machine embodiments that execute and perform the operations of a software program. The instructions may be stored as firmware in a read-only memory and / or as program code stored in a non-volatile storage medium and that can be loaded into a volatile working memory and executed by the processing unit(s) 1504. The software may be implemented as a single program or as a collection of separate programs or program modules that interact as necessary. The processing unit 1504 may obtain the program instructions to execute and the data to process from the local storage 1506 (or non-local storage, as described below) to perform the various operations described above.

[0063] In some server systems 1500, multiple modules 1502 may be interconnected via a bus or other interconnect 1508 to form a local area network that supports communication between the modules 1502 and other components of the server system 1500. The interconnect 1508 may be implemented using a variety of technologies, such as server racks, hubs, routers, etc.

[0064] A wide area network (WAN) interface 1510 may provide data communication capabilities between a local area network (interconnect 1508) and a network 1526, such as the Internet. Technologies such as wired (e.g., Ethernet, IEEE 802.3 standard) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standard) may be used.

[0065] In some embodiments, the local storage 1506 provides working memory for the processing unit(s) 1504 and is intended to provide fast access to programs and / or data being processed while reducing traffic on the interconnect 1508. One or more mass storage subsystems 1512 may be connected to the local storage 1508 to provide storage of larger amounts of data over a local area network. The mass storage subsystems 1512 may be based on magnetic, optical, semiconductor, or other data storage media. Direct attached storage, storage area networks, network attached storage, etc. may be used. Any data stores or other collections of data generated, consumed, or maintained by the services or servers described herein may be stored in the mass storage subsystem 1512. In some embodiments, additional data storage resources may be accessible via a WAN interface 1510 (which may increase latency).

[0066] The server system 1500 may operate in response to requests received via the WAN interface 1510. For example, one of the modules 1502 may implement a monitoring function and assign individual tasks to other modules 1502 in response to received requests. Work allocation techniques may be used. Once a request is processed, a result is returned to the requester via the WAN interface 1510. Such operations may typically be automated. Additionally, in some embodiments, the WAN interface 1510 may interconnect multiple server systems 1500 to provide a scalable system capable of managing large volumes of activity. Other techniques for managing server systems and server farms (collections of collaborating server systems), such as dynamic resource allocation and reallocation, may be used.

[0067] The server system 1500 can interact with various devices owned or operated by users over a wide area network such as the Internet. One example of a user operated device is shown in Figure 15 as client computing system 1514. The client computing system 1514 can be implemented as a consumer device such as, for example, a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, glasses), desktop computer, laptop computer, etc.

[0068] For example, client computing system 1514 may communicate over WAN interface 1510. Client computing system 1514 may include computer components such as processing unit(s) 1516, storage device 1518, network interface 1520, user input device 1522, user output device 1524, etc. Client computing system 1514 may be computing devices implemented in a variety of form factors, such as desktop computers, laptop computers, tablet computers, smartphones, other mobile computing devices, wearable computing devices, etc.

[0069] The processor 1516 and storage device 1518 are similar to the processing unit(s) 1504 and local storage 1506 described above. An appropriate device may be selected based on the requirements of the client computing system 1514, for example, the client computing system 1514 may be implemented as a "thin" client with limited processing power or as a high performance computing device. The client computing system 1514 is populated with program code executable by the processing unit(s) 1516 to enable various interactions with the server system 1500.

[0070] The network interface 1520 may provide a connection to a network 1526, such as a wide area network (e.g., the Internet), to which the WAN interface 1510 of the server system 1500 is also connected. In various embodiments, the network interface 1520 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0071] User input device 1522 may include any device (or devices) that allows a user to provide signals to client computing system 1514 that can be interpreted by client computing system 1514 as indicating a particular user request or information. In various embodiments, user input device 1522 may include some or all of a keyboard, touchpad, touchscreen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, etc.

[0072] The user output devices 1524 may include any device that enables the client computing system 1514 to provide information to a user. For example, the user output devices 1524 may include a display for displaying images generated by or delivered to the client computing system 1514. The display may incorporate a variety of image generating technologies, such as liquid crystal display (LCD), light emitting diode (LED) including organic light emitting diode (OLED), projection systems, cathode ray tube (CRT), and supporting electronics (e.g., digital-to-analog converters, analog-to-digital converters, signal processors). Some embodiments may include devices such as a touch screen that function as both an input device and an output device. In some embodiments, other user output devices 1524 may be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, and the like.

[0073] Some embodiments include electronic components, such as a microprocessor, storage, memory, etc., that store computer program instructions on a computer-readable storage medium. Many of the functions described in this specification may be implemented as a process specified as a series of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, the processing unit(s) perform various operations specified in the program instructions. Examples of program instructions or computer code include machine code generated by a compiler, files containing high-level code that are executed by a computer, electronic component, or microprocessor using an interpreter, etc. With appropriate programming, the processing unit(s) 1504 and 1516 can provide various functions to the server system 1500 and the client computing system 1514, including any or other functions described herein as being performed by a server or client.

[0074] It will be understood that the server system 1500 and the client computing system 1514 are exemplary and that variations and modifications are possible. Computer systems used in connection with embodiments of the present disclosure may have other features not specifically described herein. Additionally, while the server system 1500 and the client computing system 1514 are described with reference to certain blocks, it will be understood that these blocks are defined for convenience of description and do not imply any particular physical arrangement of component parts. For example, different blocks may be located in the same facility, in the same server rack, or on the same motherboard, but this is not required. Additionally, the blocks do not necessarily correspond to physically different components. The blocks may be configured to perform various operations by programming a processor or providing appropriate control circuitry, and various blocks may be reconfigurable or non-reconfigurable depending on how the initial configuration was obtained. The embodiments of the present disclosure may be realized in a variety of apparatuses, including electronic devices implemented using any combination of circuitry and software.

[0075] Further approaches to the preparation and evaluation of various calibration approaches are provided herein in the context of techniques that may be implemented by the system 100 of Figure 1. The various techniques below are described in conjunction with exemplary experimental results, which are provided to illustrate non-limiting exemplary performance of the techniques described herein using control and clinical tissue samples. The following techniques may utilize various IHC calibrator slides described herein, such as the calibrator slide shown in Figure 16 and the color reference slide shown in Figure 17.

[0076] Referring to FIG. 16, exemplary views 1605 and 1610 of a calibration slide including various staining intensities are shown, according to one or more implementations. View 1605 shows an exemplary slide image of an IHC calibrator slide for HER2. View 1610 shows an exemplary full slide image of an IHC calibrator slide for HER2 scanned at a resolution of 0.23 μm / pixel and displayed at 20x magnification, levels 1-5 (top right to left), 6-10 (bottom right to left). The IHC calibrator slide for HER2 can be utilized in combination with any of the techniques described herein, including those implemented by the data processing system 105 shown in FIG.

[0077] An IHC calibrator slide for HER2 may contain microbeads coated with different amounts of peptide concentrations. The DAB color intensity of the microbeads in the calibrator image is correlated with that of the tissue sample image. The DAB color is derived from the color of the microbeads, and the intensity characteristics are derived from the intensity of the different levels of the microbeads. The color intensity derived from the calibrator image is used to standardize the tissue image. The proposed method confirmed that it can calibrate color and intensity and classify HER2 status with low variation between datasets. To focus on the changes in staining, the evaluation was performed using a single scanner. In some implementations, the various color calibration techniques described herein may be utilized to determine the color of different parts of different types of tissues in any type of whole slide image (e.g., CISH, FISH, H&E, etc.). In some implementations, the calibrator tissue may be used to determine the color of the nucleus, CEP17, or HER2. In some implementations, the calibrator tissue may be utilized in place of an IHC calibrator slide containing microbeads, for example, in the case of techniques utilizing CISH, FISH, or H&E.

[0078] 17, exemplary views 1705 and 1710 of a color swatch slide including various reference colors are shown, according to one or more implementations. View 1705 shows an exemplary slide image of a color swatch slide. View 1710 shows a close-up view of the reference colors on the color swatch slide. The color swatch slide can be utilized in combination with any of the techniques described herein, including the techniques implemented by the data processing system 105 shown in FIG.

[0079] In one non-limiting example experiment, calibrator slides and breast cancer slides of tissue were stained together consecutively on different dates. Stained and color swatch slides were digitized by two WSI scanners. Various color calibrations implemented by the data processing system 105 were utilized to address both staining and scanning variations between the two WSI scanners. Color swatch slides (e.g., color swatch slides of FIG. 17) may be utilized to calibrate scanner variations. Calibrator slides (e.g., calibrator slides described herein, including the calibrator slide of FIG. 16) contain microbeads coated with different concentrations of peptides, thereby used to calibrate staining and scanner variations. Further details and approaches to color calibration, including these techniques, are described herein in the context of the functionality implemented by the data processing system 105 of FIG. 1.

[0080] In this non-limiting example experiment, two scanners (e.g., scanner device 125) can be used to assess both staining and scanning changes. If more than one scanner is utilized, these techniques can be used to calibrate both changes. In relation to this example experiment, the results of automated HER2 assessment between a combination of calibrator slides and swatch slides and using a calibrator alone are described in further detail herein. In a non-limiting example experiment, a swatch slide (e.g., FIG. 17) of a breast cancer resection case with known score, a calibrator slide (e.g., FIG. 16), and a HER2 IHC stained slide (e.g., as described herein) can be utilized for assessment.

[0081] In this non-limiting example experiment, four HER2 0-3+ breast cancer cases were selected for evaluation, and the formalin-fixed paraffin-embedded breast cancers were sectioned at 4 μm to produce six serial section slides per case. In this non-limiting example experiment, two types of calibration slides are used, a calibrator slide for staining changes and a color swatch slide for scanning changes. A calibrator for HER2 (e.g., shown in Figure 16) may contain two different microbeads. The larger microbeads are test microbeads coated with 10 types of different levels of peptide concentration and stained with DAB by IHC staining. The smaller microbeads are colored brown to represent the standard DAB color.

[0082] A color swatch slide (e.g., as shown in FIG. 17) may be used to calibrate, standardize, and trace color settings for image analysis. A color swatch slide may include a glass slide with 12 color patches and background areas embedded. Five slides, including HER2 0-3+ slides and a calibrator slide, were used as the data set. To obtain slides that reflect the daily changes in staining, the data set was stained consecutively on different dates using PATHWAY anti-HER2 / neu antibody. Six IHC staining data sets were prepared. The stained and color swatch slides were digitized by two WSI scanners (e.g., scanner device 125 in FIG. 1) at a resolution of 0.23 μm / pixel (equivalent to 40x) and 0.18 μm / pixel (equivalent to 20x). The exported bitmap image data was used for evaluation. The evaluation process of these example slides in this non-limiting example experiment is shown in FIGS. 18A and 18B.

[0083] 18A and 18B, in the context of the components described in connection with FIG. 1, a flow diagram of an example of a process of color and intensity standardization, including scanner calibration and stain calibration, that may be implemented by the system of FIG. 1 according to one or more implementations, is shown. FIG. 18A illustrates step 1805 of the flow diagram, where scanner devices 125 (individually shown herein as "Scanner A" and "Scanner B") may be utilized to generate a reference WSI for Scanner A and a target WSI for Scanner A and Scanner B, respectively. This process may be utilized to address both stain process-dependent and scanner-dependent sub-problems.

[0084] In step 1810, color variations dependent on the WSI scanner are calibrated using a color swatch slide. Next, in step 1815, color intensity variations dependent on the staining process are calibrated using a calibrator slide by various techniques described herein (e.g., in connection with FIG. 19, etc.). Finally, in step 1820, the standardized images are evaluated using automated image analysis software for HER2, as described herein. Even if a color swatch slide is not used, the use of a calibrator slide can mitigate variations caused by the scanner.

[0085] Referring to Figure 19, in the context of the components described in connection with Figure 1, a flow diagram of an example of a process 1900 that may be implemented by the system 100 of Figure 1 for scanner calibration, according to one or more implementations, is shown. In the process 1900, the color of each scanner image may be standardized to approximate a reference color of a sample slide. Furthermore, the stained slide and the sample slide may be scanned with the same settings (e.g., by the scanning device 125 of Figure 1).

[0086] In step 1905, the reference colors are obtained using an attached spreadsheet that contains the spectral transmission data of each patch on the slide. In step 1910, the scanned colors are obtained from the image. In step 1915, before calculating the color correction matrix, it is checked whether the gamma of the scanned image is linear. If the color correction matrix includes gamma correction (e.g., the gamma curve is not linear), gamma correction may be performed first. In step 1920, a color correction matrix is ​​derived from the reference colors and the scanned colors. In step 1925, the color correction matrix is ​​applied to the other images to obtain color-standardized images. Steps 1910-1925 may be applied to each scanner image. To evaluate the effect of color calibration, the color difference between images from different scanners is calculated in step 1930.

[0087] In step 1905 of process 1900, the data processing system 105 or one or more components thereof (e.g., intensity profile determiner 135) can calculate the reference colors of the color swatch slide from the representative spectral transmittance samples. Each color patch has a spectral transmittance T(λ) from, for example, 340 to 830 nm in 5 nm increments. The CIEXYZ color space represents colors in terms of tristimulus values ​​X, Y, and Z of the reflected object color. The tristimulus values ​​of each patch are calculated using the following formula:

number

[0088] In the above formula, S(λ) is the value of the spectral distribution of the light source at wavelength λ,

number

number

number

number

number

number

[0089] In the experiments of this non-limiting example, illumination D65 is used as the illuminant S(λ). A 3x13 matrix Gr containing the reference RGB values ​​of the 12 color patches and the background is obtained, and Figure 17 shows the calculated reference colors.

[0090] In step 1910 of process 1900, data processing system 105 or one or more of its components (e.g., intensity profile determiner 135, etc.) can obtain the scanned colors of the sample body image. The RGB values ​​of each pixel are normalized by data processing system 105 by dividing by the incident light, as shown in the following equations:

number

[0091] In the above formula,

number

number

number

[0092] In step 1915 of process 1900, data processing system 105 or one or more components thereof may utilize the grayscale colors (A1, B1, C1 and background in FIG. 17) for gamma correction. Gamma, denoted as γ, may be described as the relationship between input x and output y, as shown in the following equation:

number

[0093] In this non-limiting example experiment, the relationship between the reference color and the scanned color can be derived from the tristimulus value Y of the grayscale color. If the scanned image is linear (e.g., gamma is 1.0), the scanned values ​​will be equal to the reference values, producing a straight line. If the scanned image is non-linear, the data processing system 105 can use the resulting gamma to correct all pixel values.

[0094] In step 1920 of the process 1900, the data processing system 105 or one or more components thereof may derive a color correction matrix M using the reference color matrix Gr and the scanned color matrix Gs based on the following equation:

number

[0095] M may be determined as follows.

number

[0096] In the above formula,

number

number

[0097] In some implementations, another approach to finding M involves using white-point preserving least squares (WPPLS), which finds M that minimizes the overall residual squared error while at the same time preserving the white of the background, as shown in the following equation:

number

[0098] In the above equation, u is the background row vector, which is equal to (1,1,1)t. Linear color correction is used to find a 3x3 color correction matrix M. In some implementations, a higher order polynomial (nonlinear) transformation may be used. In this non-limiting example experiment, WPPLS and a polynomial transformation are combined to obtain M. M is obtained for each scanner (e.g., for each scanner device 125). In step 1925, the obtained M is applied to other images to standardize the colors of the images.

[0099] In step 1930 of the process 1900, to represent the color difference between the images, the data processing system 105 can correlate the CIELAB color space. L* correlates with lightness, and a* and b* correlate with the opponent color dimensions. The conversion from XYZ to L*a*b* is defined by the following equations:

number

[0100] In the above formula, X0, Y0 and Z0 correspond to the tristimulus values ​​of the reference white point. * , a1*, b1 * ) and (L2 * , a2 * , b2 * ) is proportional to the Euclidean distance and is calculated using the following formula:

number

[0101] Calculated dE * When dE is between 1.0 and 2.0, only experienced observers can notice the perceptual difference, and when dE is less than 1.0, observers cannot notice the difference.

[0102] 20A and 20B show a flow diagram of an example of a staining calibration process 2000 that may be implemented by the system of FIG. 1, according to one or more implementations. The use of automated image analysis software is envisioned for HER2 testing. The process 2000 includes a preparation step shown in FIG. 20A and an evaluation step shown in FIG. 20B. In this non-limiting example experiment, a calibrator slide is stained at each step to provide a comparable staining standard.

[0103] In the preparation stage of FIG. 20A, in step 2002, a calibrator slide and a breast cancer case with a known score are prepared as a reference data set. The reference data set is digitized using a single scanner (e.g., one of the scanner devices 125) that is considered as the reference scanner. In some implementations, if the automated image analysis software is intended to use a specific scanner, that scanner becomes the reference. In the evaluation stage shown in FIG. 20B, another calibrator slide is stained and scanned together with the clinical tissue slide to be evaluated to generate a target data set 2012. If the same scanner is used to digitize the reference data set and the target data set, the proposed process 2000 implies stain correction. If the scanners are different (e.g., the scanner devices 125 are different), the process 2000 has the effect of both scanner correction and stain correction. When combining scanner calibration, color swatch slides can be scanned together at each stage.

[0104] For standardization of IHC stained images of HER2, the color intensity of DAB can be calibrated. Color separation methods based on optical models can be used to separate color images of multi-stained specimens into components of staining intensity representing single stains. Light attenuation follows the Beer-Lambert law. The optical density (OD) measured from the images is linearly related to the amount of staining, as shown in the following formula:

number

[0105] In the above formula,

number

number

[0106] If the specimen is tissue stained with DAB and hematoxylin, the OD is expressed by the following model:

number

[0107] In the above formula, V DAB DAB staining matrix, V H are stained matrices for hematoxylin. These matrices have a staining intensity of C DAB and C H This is necessary for color separation to derive

[0108] Referring to the preliminary stage of the process 2000 shown in FIG. 20A, the data processing system 105 derives the DAB V from each level of microbeads in the calibrator image. DAB,R and C DAB,R To do so, in step 2005, the data processing system 105 extracts the staining color and intensity of hematoxylin V from the image with a score of 0 because the tissue is stained with hematoxylin only. H,R The colors are then obtained and then automated image analysis techniques are used to measure staining intensity from the calibration images in step 2010. As shown, a reference data set 2002 is utilized that includes images of reference tissues as described herein.

[0109] Referring to the evaluation stage of process 2000 shown in FIG. 20B, the data processing system 105 derives a target V from an image of the calibrator slide generated as described herein. DAB,T and C DAB,T The staining color and intensity of the target data set can be obtained. The data processing system 105 can calculate an intensity correction function f(C) to map the staining intensity of the target data set to the staining intensity of the reference data set. The data processing system 105 can use regression analysis to obtain a linear function f(C),

number

[0110] As shown, process 2000 includes selecting at least two automated evaluation methods at step 2015. Using multiple implementations, data processing system 105 (or an operator of data processing system 105 via user input) can select the appropriate correction method depending on the automated image analysis software. For example, if the image analysis software is threshold adjustable, the approach of step 2020 can be utilized. Otherwise, step 2025 can be selected.

[0111] In step 2020 of process 2000, data processing system 105 can use f(C) to adjust a threshold for classifying DAB film intensity. In step 2025 of process 2000, data processing system 105 can use f(C) and the stain matrix to correct the color and intensity of the tissue image to match the reference image. In this non-limiting example experiment, and as described above, two scanners (Scanner A and Scanner B, each of which may be one of scanner devices 125) are used, with Scanner A being considered as the reference scanner in the stain calibration step.

[0112] Referring back to FIG. 1 and to a more detailed explanation of the non-limiting example experiment described above, scanner calibration was performed for all scanned images using the techniques described herein. A dE* map was created to evaluate the perceptual color difference between the two images on a pixel-by-pixel basis. The dE* map shows the dE* between the two images calculated as a pixel value for each corresponding pixel. Since the images exported from the two scanners have different spatial dimensions and pixel resolutions, image registration may be performed for experimental evaluation. In this non-limiting example experiment, AKAZE (Accelerated-KAZE), which detects and matches keypoints on the two images, was utilized for image registration. To determine the order of the polynomial transformation, dE* was calculated for various combinations of the orders of the two scanners. The combination with the smallest color difference was selected.

[0113] Further, in this non-limiting example experiment, a simulation experiment was performed to check whether adding brown reduces the color difference of IHC staining. A color correction matrix was derived using 3x14 matrices Gr' and Gs', which were originally 3x13 matrices Gr and Gs, to which brown was added. Brown was obtained from the IHC calibrator image. The reference color was the image obtained from the large microbeads of scanner A and color calibrated using Gr and Gs. The scanned colors were obtained from the small microbeads of the original images of each scanner. To check whether the color intensity of brown affects the color calibration, two color intensity patterns were prepared: light brown and dark brown. Light brown corresponds to the intensity of the large microbeads at level 10. Dark brown corresponds to the intensity of the smaller microbeads.

[0114] In this non-limiting example experiment, further analysis was performed using automated HER2 assessment. Automated HER2 assessment was performed on pre-calibration and post-calibration images. Accuracy was evaluated after performing various combinations of scanner and stain calibration, as shown in Table 1 below. When performing stain calibration, one of the six datasets was considered as the reference dataset and the remaining five datasets were considered as the target datasets for HER2 testing. [Table 1]

[0115] The data processing system 105 can execute automatic image analysis software for HER2 score classification. A non-limiting example of such software includes QuPath. The software executed by the data processing system 105 can be semi-automatic and can enable setting color vectors for color unmixing and thresholds for classification. The data processing system 105 can calculate the histoscore (H-score), which is one of the evaluation indexes in HER2 evaluation, by solving the following formula:

number

[0116] In the above formula, I is the immune score classified according to a threshold (0, 1, 2 or 3), and P is the percentage of immune-reactive tumor cells, respectively. The data processing system 105 can calculate the standard deviation (SD) of the H-score. The SD was utilized to compare the calibration combinations in this non-limiting example experiment. A two-way analysis of variance (ANOVA) was performed to analyze the effect of scanner calibration and staining calibration on the automated HER2 assessment. As a post-hoc analysis, Tukey's honestly significant difference (HSD) test was also used.

[0117] Figures 21A, 21B, 21C, 22A, 22B, 22C, and 23A, 23B, 23C show examples of the results of calibrating a scanner using color swatch slides based on the techniques described herein. The top panels of Figures 21A, 22A, and 23A show uncalibrated images, and the bottom panels show color calibrated images. By performing calibration using the techniques described herein, the average color difference was reduced in the H&E stained tissue images (6.2 to 2.5), the IHC stained tissue images (4.1 to 1.6), and the IHC stained calibrator images (4.6 to 4.1).

[0118] 21A, 21B, and 21C show slide images, such as stained breast cancer tissue images, plots, and mean and variance values, respectively, according to one or more implementations. FIG. 21A shows a comparison of images before and after color calibration and a dE* map based on the techniques described herein. The dE* map shows the color difference between the two scanners as pixel values ​​calculated for each corresponding pixel. FIG. 21B shows an example histogram of the dE* map before and after calibration. FIG. 21C shows the mean and variance values ​​of dE* before and after calibration. Images from both Scanner A and Scanner B used in the non-limiting example experiments are shown.

[0119] Figures 22A, 22B, and 22C show slide images, plots, and mean and variance values, such as HER IHC stained breast cancer tissue images, according to one or more implementations. Figure 22A shows a comparison of HER IHC stained breast cancer tissue images and corresponding dE* maps before and after color calibration. Figure 22B shows example histograms of dE* maps before and after calibration. Figure 22C shows mean and variance values ​​of dE* before and after calibration. Images from both Scanner A and Scanner B used in the non-limiting example experiments are shown.

[0120] Figures 23A, 23B, and 23C show example slide images, such as IHC calibrator images, plots, and mean and variance values, respectively, according to one or more implementations. Figure 23A shows a comparison of an IHC calibrator image before and after color calibration with the corresponding dE* map. Figure 23B shows an example histogram of the dE* map before and after calibration. Figure 23C shows the mean and variance values ​​of dE* before and after calibration. Images are shown for both Scanner A and Scanner B used in the non-limiting example experiments.

[0121] Figures 24A, 24B and 24C show a comparison of dE* maps with and without brown in HER2 IHC stained breast cancer tissue images. Figures 25A, 25B and 25C show a comparison of dE* maps with and without brown in IHC stained calibrator images. Figures 26A, 26B and 26C show a comparison of dE* maps with and without brown in H&E stained breast cancer tissue images. Figures 24A-26C show the comparison results of color calibration with and without brown. Table 2 below shows the percentage of dE* greater than 6 and the average dE*.

[0122] In these non-limiting example results, the addition of dark brown reduced the percentage of large color differences in IHC stained tissues (0.5 to 0.2) and calibrator images (16.9 to 14.8). In H&E stained images, the addition of brown reduced dE*. Table 3 below shows the parameters obtained by analyzing the relationship of DAB intensity between scanners A and B. This parameter would ideally be 1.0 since it is the relationship of DAB intensity for the same calibrator slide. Scanner calibration brought it closer to 1.0 without the addition of brown. [Table 2] [Table 3]

[0123] Further complementing the results of the non-limiting experimental examples above, Table 4 below shows the results of the automated HER2 assessment described herein. When no stain calibration was performed (Methods A and D in Table 1 above), some slides were misclassified. However, both scanner and stain calibration (Methods B, C, E, F in Table 1 above) resulted in 100% agreement with pathologist assessment. Additionally, both calibrations resulted in smaller SDs for the H-scores.

[0124] Table 5 below shows the results of the two-way ANOVA and Table 6 below shows the results of the Tukey HSD test described herein. The two-way ANOVA revealed no statistically significant interaction between scanner and stain calibration. Simple main effects analysis showed that scanner calibration was statistically significant for 1+, F(1,54)=22.30, P=0.000, and for 3+, F(1,54)=14.09, P=0.000. Simple main effects analysis showed that stain calibration was statistically significant for 3+, F(2,54)=3.63, P=0.033. Post hoc analysis using the significance test Tukey's HSD test showed that this method was significant in the 2+ cases between A and E (P = 0.002) and between A and F (P = 0.002), and in the 3+ cases between B and D (P = 0.047), between C and D (P = 0.001), and between C and E (P = 0.035). [Table 4] [Table 5] [Table 6]

[0125] Referring to Figures 27A and 27B, HER2 IHC breast cancer images scanned using a 40x objective with intensity color bars are shown. Figures 27A and 27B compare the original and color and intensity calibrated images captured from Scanner A and Scanner B of the non-limiting example experiment above, with Figure 27A showing the 2+ case and Figure 27B showing the 3+ case. Figures 27A and 27B show examples of color and intensity calibrated images (A, C, and F in Table 4 above). The intensity color bar shown below the tissue image was generated from the color intensity of the calibrator image. The color of the target image was corrected to be closer to the reference image.

[0126] Referring to FIG. 28, a comparison of tissue images captured from Scanner A and Scanner B of the non-limiting example experiment is shown with the corresponding dE* map. The dE* map is overlaid on the tissue image in magenta. The techniques described herein can utilize the color sample slide of FIG. 17 to standardize the color of the WSI, which varies depending on the scanning device (e.g., scanner device 125). The color difference between scanners has decreased for both H&E staining and IHC staining. It is said that dE drops below 2.0, especially for IHC staining, and only experienced observers can notice the difference.

[0127] The dE* of HER2 IHC stained tissues tends to be larger in areas where DAB stained darkly (red arrow in Figure 28). Adding dark brown when determining the color correction matrix reduced the dE* in dark areas. The results of the calibrator slide shown in Figures 25A, 25B, and 25C show the effect of adding darker brown than adding the corresponding intensity. The results in Table 3 show that the effect is sufficient even without brown. There is a slight color difference around the membrane (green arrow in Figure 28), which is likely due to the difference in the spatial frequency of the scanner. This is because once the colors are standardized by the scanner calibration, other factors may cause color differences.

[0128] In this non-limiting example experiment, accuracy was evaluated after performing various combinations of scanner and stain calibration. The results of this non-limiting example experiment, as shown in part in Table 4, indicate that performing both calibrations eliminated misclassification. Furthermore, applying both calibrations reduced the SD of the H-score. Compared to methods D-F in Table 1, the SD of methods A-C in Table 1, which do not use swatch slides, tend to be smaller. For 3+ tissue samples, differences were observed between methods C and E in Table 1. This result means that method C may have improved performance compared to method E. Using automated HER2 assessment, it can be assessed with high accuracy without using swatch slides.

[0129] The technique using the calibrator slide described herein is designed to use a reference scanner. By utilizing the technique described herein, images scanned by other scanners can be analyzed using information from the reference scanner. The technique described herein can be introduced into clinical workflows, since the only additional process is to perform the same staining protocol on the calibrator slide. When performing scanner calibration using a color sample slide, gamma correction is performed before color correction. However, when only the calibrator slide is used, gamma is not corrected. In some implementations, a scanner with linear characteristics can be used as a reference.

[0130] Further non-limiting example experiments are described below, which may be implemented using the techniques described in connection with the example experiments above. These techniques are described in connection with steps of process 3200 of Figure 32, which may be similar to steps of process 2000 described below and in connection with Figures 20A and 20B.

[0131] In this further non-limiting example experiment, tissues from a calibrator slide (e.g., any of the exemplary calibrator slides described herein) and a breast cancer slide were stained together consecutively on different dates and scanned with a WSI scanner (e.g., scanner device 125 of FIG. 1). In this further non-limiting example experiment, 60 images were used to evaluate the techniques described herein. The results show that in an implementation that does not utilize the techniques described herein, the HER2 status of 7 of the 60 slides was misclassified using automated image analysis software.

[0132] However, by using the technique described herein, misclassification was nearly eliminated (0 / 60 slides in one implementation, 1 / 60 slides in another implementation). Moreover, further non-limiting example experiments showed that the technique can significantly reduce the variability of HER2 intensity scores. Analysis of variance showed that the effect of the method was significant compared to the evaluation without a calibrator, F(2,40)=11.79, P=0 for 2+ cases and F(2,40)=5.14, P=0.010 for 3+ cases. Thus, the proposed technique allows for a more consistent calibration of DAB intensity and evaluation of HER2 status, which is an improvement over conventional techniques. The technique can also be used to automate the evaluation of other IHC tests (membranous or cytoplasmic staining to evaluate staining intensity).

[0133] Referring to Figures 29A and 29B, HER2 IHC stained samples are shown using a 40x objective. Figure 29A shows microbeads on an IHC calibrator slide. Figure 29B shows negative and positive controls (left) from an IHC test on the same slide of a HER2 positive breast cancer sample. The HER2 calibrator slide shown in Figure 29A is made with microbeads coated with different amounts of peptide concentrations as described herein. The calibrator slide can be used to check whether each laboratory's immunostaining is set up correctly. Negative and positive controls can be run with every assay to ensure the quality of the staining in IHC. Including a control group on the same slide as the tissue sample helps the pathologist visually assess the intensity of the DAB, but the positive control often has a higher intensity than the clinical positive sample, as shown in Figure 29B. In some cases, the control group obtained from the tissue sample has an unknown concentration of the analyte, which changes with each sample. However, IHC stained calibrator slides have microbeads with a range of intensities appropriate for tissue samples, with different intensities depending on the concentration levels. The views of the calibrator slide shown in Figures 29A and 29B are more reproducible because the concentrations of each level are defined.

[0134] In this further non-limiting example experiment, a calibrator slide is used to process the change in WSI color intensity due to the staining process. As described herein, the process performed on the calibrator slide can be performed together with the tissue sample slide, from staining to WSI scanning. The calibrator slide can also be used for microscopy. The technology described herein utilizes the WSI of the calibrator slide and allows for automatic calibration of the DAB color intensity in the tissue image. The DAB color intensity of the microbeads in the calibrator image is correlated with that of the tissue sample image. The DAB color is derived from the color of the microbeads, and the intensity characteristics are derived from the different levels of density of the microbeads. The technology can be utilized with automated image analysis software for HER2 testing, as described herein.

[0135] As described herein in relation to FIG. 32, two experimental implementations may be selected depending on the automated image analysis software. One implementation (e.g., step 3220 of process 3200 in FIG. 32) includes setting a threshold for classifying HER2 status based on the intensity of each level of microbeads, and another embodiment (e.g., step 3225 of process 3200 in FIG. 32) includes performing color correction of the tissue image. Details of each of these steps are described in more detail below. This further non-limiting example experiment evaluates the performance of the current technology in correcting for changes created during the staining process and utilizes calibrator slides of breast cancer resection cases with known scores and HER2 IHC stained slides.

[0136] Referring to Figures 30A, 30B, 30C and 30D, HER2 IHC stained breast cancer images scanned at 0.23 μm / pixel resolution (equivalent to 40x) are shown for 0, 1+, 2+ and 3+ cases, respectively. In this further non-limiting example experiment, the morphology of the invasive breast cancer tissue used in this study had no identifiers except for the histological diagnosis. As shown in Figures 30A, 30B, 30C and 30D, four typical HER2 0-3+ breast cancer resection cases in our hospital were selected. Formalin-fixed paraffin-embedded breast cancers were sectioned at 4 μm to create 16 serial section slides per case.

[0137] With reference to Figures 31A, 31B and 31C, an example of a stained IHC calibrator for HER2 IHC is shown. Figure 31A shows an image of a calibrator slide, and Figure 31B shows a WSI scanned at 0.23 μm / pixel resolution (equivalent to 40x) and displayed at 0.5x magnification. The top and bottom lines in Figure 31B are stained at 10 intensity levels. The middle line in Figure 31B contains unstained microbeads. Figure 31C shows a WSI displayed at 20x magnification, with levels 1-5 (top left to right), 6-10 (bottom left to right). In this additional non-limiting example experiment, the HER2 IHC calibrator slide shown in Figures 31A, 31B and 31C was utilized. The IHC calibrator slide contains microbeads coated with 10 different peptide concentrations and stained with DAB by IHC staining.

[0138] In this further non-limiting example experiment, five slides including HER2 0-3+ slides and a calibrator slide were used as a data set. To obtain slides reflecting daily changes in staining, the data set was stained continuously on different dates using PATHWAY anti-HER2 / neu antibody. In this further non-limiting example experiment, 16 IHC staining data sets were prepared and WSI was acquired by a WSI scanner (e.g., scanner device 125 in FIG. 1) at a resolution of 0.23 μm / pixel (equivalent to 40x). The exported bitmap image data was used for evaluation, utilizing QuPath as an automated image analysis software for HER2 score classification (e.g., can be executed by data processing system 105).

[0139] Referring to FIG. 32 in the context of the components described in relation to FIG. 1, in this additional non-limiting example experiment, a process 3200 similar to the process 2000 in FIG. 20A and FIG. 20B and which may be implemented by the data processing system 105 of FIG. 1 was utilized to perform standardization between slide images. As described herein, the process 3200 includes a preparation stage (e.g., similar to the preparation stage shown in FIG. 20A) and an evaluation stage (e.g., similar to the evaluation stage shown in FIG. 20B). In a brief overview of the process 3200, a reference color intensity is determined in steps 3205 and 3210. A target color intensity of DAB is determined in step 3216 and used in the standardization procedure described herein. Step 3220 includes using parameters obtained from the calibrator slide to adjust a threshold value (e.g., determined in step 3215) for classifying the film intensity of DAB. Step 3225 includes using these parameters to correct the color intensity of the image.

[0140] The process 3200 can be utilized in combination with the color separation methods described above. Color separation methods based on optical models can be used to separate color images of multi-stained specimens into components of stain intensity representing single stains (DAB and hematoxylin in this case). The detected intensity of light transmitted through the specimen is described by the Beer-Lambert law, also known as the light transport through homogeneous absorbing media equation. Using the equation described in connection with the non-limiting example experiments above, the color image is transformed into optical density space and the stain matrix is ​​determined. Determination of the stain matrix can be used, for example, if the color change of the image varies from one stain batch to another, since the same stain matrix cannot be used across multiple images.

[0141] In step 3205, the data processing system uses the tissue image and the calibrator image to generate a stain matrix (e.g., V DAB and V H ) can be determined. V H To obtain a V score, use a tissue slide with a score of 0. This slide can be stained with hematoxylin only andH can be determined by calculating the unit vector from the mean optical density of the tissue image. DAB can be determined from the calibrator image by the data processing system 105. First, the microbeads are detected from the calibrator image using a Hough transform by the data processing system 105. The average value of the central area of ​​the microbeads can be used for analysis. Although the calibrator slide can be stained with DAB, microbeads with low staining intensity will have a grayish hue in the WSI. This is believed to be due to the effect of light scattering by the microbeads and the optical density O of the calibrator image, which is represented by the following model:

number

[0142] In the above formula, the scattering color vector V 散乱 can be considered as a fixed wavelength-independent value that represents a grayish hue. DAB uses non-negative matrix factorization (NMF) to 較正器 From V 散乱 can be calculated by semi-blind separation.

[0143] In step 3210, the matrix V H,R and V DAB,R may be calculated in step 3205 using the above for a reference data set 3202 (which may be determined similarly to the reference data set 2002 of process 2000 in FIG. 20A, as described herein). Additionally, the matrix C DAB,R is estimated by color separation as described above. V of the target data set 3212 (which can be obtained, for example, similar to the target data set 2012 of the process 2000 described in relation to FIG. 20B ) H,T , V DAB,T and C DAB,T A similar technique is utilized in step 3216 of the evaluation phase to calculate V. The data processing system 105 uses a negative control to calculate V H,T can be obtained.

[0144] In step 3215, the data processing system may determine a stain intensity threshold from the tissue image, as described in more detail in connection with Figure 33. Referring to Figure 33 in the context of the components of the system 100 of Figure 1, a data flow diagram of a process 3300 for calculating a stain intensity threshold from a tissue image is shown. The process 3300 may be performed by the data processing system 105. As shown in the output graph 3315, the baseline thresholds are defined as a, b, c, which divide the immune scores 0, 1, 2, 3 of the baseline dataset 3202.

[0145] In the process 3300, first, a color separation is performed to separate DAB and hematoxylin using the stain matrix obtained in step 3205 of FIG. 32. In step 3305, as shown in the figure, probability densities p0, p1, p2, and p3 are created from the DAB intensity of the membrane area. In some implementations, a threshold value can be determined using the intersection value of each probability density. For example, the intersection value of p0 and p1 can be determined as a threshold a. The intersection value can be difficult to determine because the values ​​of p0 and p1 can be close to a normal distribution and the range of values ​​of p2 and p3 can overlap widely. In some implementations, a further distribution including the intensity of one immune score can be generated by taking the difference of the probability densities. The distribution of score 0 does not include the intensity of the other scores and is represented by p0. To represent immune score 1, the value of p0 is subtracted from p1. Similarly, the values ​​of p0 and p1 can be subtracted from p2 to represent immune score 2. To represent the immune score 3, the values ​​of p0, p1, and p2 are subtracted from p3. Each distribution can be approximated using a Johnson SU distribution, and the intersection values ​​of the distributions can be set as thresholds a, b, and c.

[0146] Referring back to FIG. 32, in step 3217, the data processing system 105 can select a method for standardizing the image. As described herein, for automated evaluation, two implementations of the standardization procedure are provided. Depending on the capabilities of the automated image analysis software, the two implementations can be selected by the data processing system 105. For example, if the image analysis software is open source or has an adjustable threshold, step 3220 of the process 3200 can be performed. Otherwise, step 3225 of the process 3200 can be performed. Furthermore, the color-corrected image obtained in step 3225 can be used for review by a pathologist.

[0147] In step 3220 of process 3200, the data processing system may correct the reference threshold determined in step 3215 to the original tissue image for testing. To do so, the data processing system 105 may perform the steps of process 3400 described in connection with FIG. 34. Referring to FIG. 34 in the context of the components described in connection with FIG. 1, a data flow diagram 3400 of a first implementation of process 3200 described in connection with FIG. 32 is shown that includes correcting the reference threshold to accommodate the original tissue image. The steps of data flow diagram 3400 may be performed when step 3220 is selected in step 3217 of process 3200.

[0148] In step 3405, the data processing system DAB,R and C. DAB By analyzing the relationship between ,T, we calculate the correction function for intensity. To remove outliers, we can use the mid-intensity data in a regression analysis from the original intensity data. For this purpose, C DAB,R and C. DAB,T Both can be sorted by strength. DAB,R and C. DAB,T The median intensity, and n intensities greater than and less than the median intensity are extracted from the data. As a result, two (2n+1)×Nm pieces of data C' DAB,R and C' DAB,Twhere Nm represents the number of detected microbeads. In this study, n was empirically set to 50. The linear function f(C) was calculated by regression analysis.

number

[0149] In step 3410, the data processing system 105 can use the resulting correction function f(C) to map staining intensities of a reference dataset (e.g., the reference dataset 3202) to staining intensities of a target dataset 3212. The data processing system 105 then calculates the correction function (e.g.,

number

[0150] Referring back to Figure 32 in the context of the system 100 described in connection with Figure 1, the data processing system 105 may perform step 3225 if the corresponding method was selected in step 3217. Step 3225 includes performing the process illustrated in flow diagram 3500 of Figure 35. Referring to Figure 35, a data flow diagram 3500 of a second implementation of the process 3200 described in connection with Figure 32 is shown, which includes color and intensity correction of the tissue image. This embodiment may be performed by the data processing system 105 as described herein.

[0151] As discussed above, the process illustrated in data flow diagram 3500 may be used to correct the color and intensity of tissue images. To do so, data processing system 105 may use a calibrator to calculate an intensity correction function in the same manner as described in connection with FIG. 34 above. In step 3505, data processing system 105:

number

[0152] Because the calibrator and tissue were stained and scanned together, the same V obtained from the calibrator image described in the previous section DAB The color correction of the tissue image is performed in three steps. First, in step 3510, the data processing system 105 extracts the detected V DAB,T and V H,T Perform color separation of the tissue images using C DAB,T and C. H,T Next, in step 3515, the data processing system 105 can perform a calibration of the staining intensity. Since the DAB intensity is the subject of the HER2 score classification, the DAB intensity can be corrected without changing the hematoxylin intensity. The data processing system 105 can apply a correction function g(C) to obtain a correction function g(C) for C. DAB,T to C' DAB,T Finally, in step 3520, the data processing system 105 can perform color mixing using the following equation to obtain a corrected color image:

number

[0153] In the above formula, I 補正済are the pixel values ​​of the corrected image. Here, the matrix V obtained from the reference data set DAB,R and V H,R may be used by the data processing system 105 to correct the intensity and color of the tissue image. In step 3230, the corrected tissue image is provided to the automated image analysis software, using the reference thresholds a, b, c. The stain matrix may also be provided to a color separation technique implemented in the automated image analysis software, as described herein.

[0154] In step 3230 of process 3200, data processing system 105 can execute automated image analysis software that can be utilized in conjunction with the image normalization techniques described herein. The automated image analysis software can execute a cell detection algorithm with the input image as input and segment the cell region into nucleus, cytoplasm, and membrane. Data processing system 105 (or one or more components thereof) can execute a color separation algorithm to separate the DAB signal from the hematoxylin. Using these techniques, data processing system 105 can derive the staining intensity in the target region of the image. To obtain the HER2 status, data processing system 105 can execute an intensity classification according to a user-set intensity threshold. The DAB intensity of the membrane can be classified into an immune score (0, 1, 2, or 3, corresponding to no staining, weak staining, moderate staining, strong staining) according to the intensity threshold. Finally, data processing system 105 can determine the HER2 status by the percentage of tumor cells according to the techniques described herein.

[0155] Returning to FIG. 1, in the context of the further non-limiting example experiments described herein above, examples of experimental results of the further non-limiting example experiments generated using the techniques described herein are provided. To generate the results, one of the 16 datasets above was considered as the reference dataset, and the remaining 15 datasets were considered as the target datasets for HER2 testing. In the reference dataset, a region of interest was selected that was used for evaluation and for determining the reference threshold. The reference threshold was fixed and commonly used regardless of the dataset to be compared with the proposed method. Table 7 below shows the input image and threshold combinations for the three methods. [Table 7]

[0156] To calculate the results of the experiment of the further non-limiting example, QuPath is used as the automatic image analysis software for HER2 score classification. The software executed by the data processing system 105 can be semi-automatic and can set the color vector for color unmixing and the threshold for classification. DAB membrane staining is classified into an immune score (0, 1, 2 or 3) according to the threshold. The data processing system 105 can calculate the histoscore (H-score), which is one of the evaluation indexes in HER2 evaluation, by solving the following formula.

number

[0157] In the above formula, I is the immune score classified according to the threshold (0, 1, 2 or 3), and P is the percentage of immune-reactive tumor cells, respectively. Examples of experimental results of further non-limiting examples are as follows: Table 8 shows the results of HER2 score classification. When using an alternative approach that does not implement the present technology, there were some misclassified slides (e.g., 7 / 60 slides). By using the technology described herein, misclassification was largely eliminated (0 / 60 slides when implementing the approach described in step 3220 of FIG. 32 and 1 / 60 slides when implementing the approach described in step 3225 of FIG. 32). Both implementations resulted in smaller SD of the H-score (e.g., steps 3220 and 3225).

[0158] Table 9 shows the results of the analysis of variance, and Table 10 shows the results of the Tukey HSD test. The analysis of variance showed that the effect of this method was significant for 1+, F(2,40) = 10.75, P = 0.000, for 2+, F(2,40) = 11.79, P = 0.000, and for 3+, F(2,40) = 5.14, P = 0.010. Post-hoc analysis using significance test Tukey's HSD test showed that the method was significant at 2+ between proposed method 1 and the conventional method (P = 0.001, 95% CI [6.34, 24.36]), between proposed method 2 and the conventional method (P = 0.001, 95% CI [6.83, 24.85]), and at 3+ between proposed method 1 and the conventional method (P = 0.015, 95% CI [1.70, 18.39]), and between proposed method 2 and the conventional method (P = 0.033, 95% CI = [0.60, 17.28]). [Table 8] [Table 9] [Table 10]

[0159] Figures 36A, 36B, 36C, 36D and Figures 37A, 37B, 37C, 37D show an example of color correction resulting from performing an implementation of process 3220 in which step 3225 is selected. Figures 36A, 36B, 36C, and 36D show an example of color correction for 2+. Figure 36A shows a reference image, Figure 36B shows a target image before correction, Figure 36C shows a target image after correction, and Figure 36D shows a plot of a histogram of DAB film staining comparing the reference image and the target image. Figures 37A, 37B, 37C, and 37D show an example of color correction for 3+. Figure 37A shows a reference image, Figure 37B shows a target image before correction, Figure 37C shows a target image after correction, and Figure 37D shows a plot of a histogram of DAB film staining comparing the reference image and the target image.

[0160] The color of the target image was corrected to be closer to the reference image using the techniques described herein, and the histogram was also modified to match the reference image. Table 11 below shows a comparison of the similarity of the histograms. For correlation coefficients, values ​​closer to 1 indicate higher similarity. For chi-square analysis, lower values ​​indicate higher similarity of the results. Based on the correlation coefficients, the reference and corrected target images were highly similar even before correction. In this further non-limiting example experiment, all data sets were stained at one institution, so there was less variation in staining. The chi-square results showed that the corrected values ​​were smaller than the uncorrected values ​​for 1+ and 2+ cases. T-tests showed that the effect of this method was significant for 2+ cases, with a correlation coefficient of t=-4.90, P=0.000, and a chi-square test of t=3.58, P=0.002. [Table 11]

[0161] The technique described herein utilizing calibrator slides allows for standardization of the color and intensity of the WSI of HER2 testing, which varies depending on the staining process. In the context of this further non-limiting example experiment, the technique described herein shows that misclassification is nearly eliminated, and in most cases application of the technique results in smaller SDs of the H-scores and less variation in score classification. In 2+ and 3+ cases, significant differences were observed between alternative approaches and the technique. Furthermore, the technique can be used to obtain a threshold value from tissue images. The threshold value, in combination with our standardization method, provided the correct HER2 status.

[0162] Integrating the technique described herein into clinical workflow requires minimal modifications, such as performing the same staining protocol on the calibrator slides and the target tissue slides. By changing the antibodies in the calibrator slides, the technique described herein can be extended to HER2 testing in gastric cancer and other IHC tests in breast cancer, such as Ki67 and PD-L1. One advantage of the calibrator described herein is that it has 10 intensity levels. Compared to the usual negative and positive controls, which represent typical 0 and 3+ HER2 cases, the difference in intensity from 0 to 3+ can be observed in detail. With regard to obtaining the correction functions f(C) and g(C), thanks to the large number of levels of the calibrator, the C DAB,R and C. DAB,T The correspondence can be found in detail.

[0163] Although the present disclosure has been described in connection with specific embodiments, those skilled in the art will recognize that many variations are possible. The embodiments of the present disclosure may be implemented using a variety of computer systems and communication technologies, including but not limited to the specific examples described herein. The embodiments of the present disclosure may be implemented using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein may be implemented on any combination of the same or different processors. Where a component is described as being configured to perform a particular operation, such configuration may be realized, for example, by designing electronic circuitry to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by a combination thereof. Furthermore, while the above embodiments refer to specific hardware and software components, those skilled in the art will recognize that different combinations of hardware and / or software components may be used, and that certain operations described as being implemented in hardware may also be implemented in software, or vice versa.

[0164] A computer program incorporating various features of the present disclosure may be encoded and stored on a variety of computer-readable storage media, suitable media including magnetic disks or tapes, optical storage media such as compact disks (CDs) or DVDs (digital versatile disks), flash memory and other non-transitory media. The computer-readable medium on which the program code is encoded may be packaged in a compatible electronic device, or the program code may be provided separately from the electronic device (e.g., as a download from the Internet or as a separately packaged computer-readable storage medium).

[0165] Therefore, although the invention has been described in terms of specific embodiments, it will be understood that the invention is intended to cover all modifications and equivalents within the scope of the following claims.

[0166] It will be readily understood that several aspects can be combined and that features described in the context of one aspect can be combined with other aspects. The aspects can be implemented in any convenient form. For example, by a suitable computer program that can be carried on a suitable carrier medium (computer readable medium), which can be a tangible carrier medium (e.g., a disk) or an intangible carrier medium (e.g., a communication signal). The aspects can also be implemented using a suitable apparatus, which can take the form of one or more programmable computers that execute a computer program configured to implement the aspects. As used in this specification and claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

Claims

1. 1. A method for automating detection by performing color calibration using a calibration slide, the method comprising: receiving, by one or more processors coupled to a non-transitory memory, a plurality of calibration images from the scanner, each of the plurality of calibration images showing a portion of a respective stained tissue sample having a known score value; determining, by the one or more processors, a correction function for the scanner based on the plurality of calibration images; receiving, by the one or more processors, an unscored slide image showing a portion of stained tissue, the unscored slide image being captured by the scanner; and displaying, by the one or more processors, scores of the unscored slide images generated using the correction function of the scanner in a graphical user interface of an application running on the one or more processors.

2. The method of claim 1 , further comprising determining, by the one or more processors, the scores for the unscored slide images based on the correction function.

3. The method of claim 1 , wherein each of the plurality of calibration images is selected to correspond to a respective score value.

4. The method of claim 1 , wherein determining the correction function comprises performing a color calibration process.

5. The method of claim 4 , wherein the color calibration process includes calculating, by the one or more processors, a color correction matrix to be applied to the unscored slide images.

6. The method of claim 1 , wherein determining the correction function comprises determining, by the one or more processors, one or more color thresholds that correspond to the plurality of calibration images.

7. The method of claim 1 , further comprising annotating, by the one or more processors, the unscored slide images based on the correction function.

8. The method of claim 1 , wherein determining the correction function comprises determining one or more intensity values ​​over multiple days.

9. receiving, by the one or more processors, second unscored slide images generated by a second scanner during a second time period; identifying, by the one or more processors, a second correction function associated with the second scanner and the second time period; 2. The method of claim 1, further comprising indicating, by the one or more processors, a second score for the second unscored slide image in the application generated using the second correction function.

10. The method of claim 1 , wherein each of the plurality of calibration images corresponds to a respective HER2 score.

11. 1. A system for performing color calibration using a reference tissue to automate detection, the system comprising: one or more processors coupled to a non-transitory memory, the one or more processors comprising: receiving a plurality of calibration images from a scanner, each of the plurality of calibration images showing a portion of a respective stained tissue sample having a known score value; determining a correction function for the scanner based on the plurality of calibration images; receiving an unscored slide image showing a portion of stained tissue, said unscored slide image being captured by said scanner; and indicating scores of the unscored slide images generated using the correction function of the scanner in a graphical user interface of an application executing on the one or more processors.

12. The system of claim 11 , wherein the one or more processors are further configured to determine the score for the unscored slide image based on the correction function.

13. The system of claim 11 , wherein each of the plurality of calibration images is selected to correspond to a respective score value.

14. The system of claim 11 , wherein to determine the correction function, the one or more processors are further configured to perform a color calibration process.

15. 15. The system of claim 14, wherein the one or more processors are further configured to calculate a color correction matrix to be applied to the unscored slide images.

16. The system of claim 11 , wherein to determine the correction function, the one or more processors are further configured to determine one or more color thresholds corresponding to the plurality of calibration images.

17. The system of claim 11 , wherein the one or more processors are further configured to annotate the unscored slide images based on the correction function.

18. The system of claim 11 , wherein to determine the correction function, the one or more processors are further configured to determine one or more intensity values ​​over multiple days.

19. The one or more processors: receiving a second unscored slide image produced by a second scanner during a second time period; identifying a second correction function associated with the second scanner and the second time period; and indicating a second score for the second unscored slide image in the application generated using the second correction function.

20. The system of claim 11 , wherein each of the plurality of calibration images corresponds to a respective HER2 score.

Images