Specimen processing independent image representation learning for digital pathology

Abstract

Described herein are systems, methods, and programming for analyzing and classifying digital pathology images independent of the sample processing technology used to prepare the digital pathology images. In some embodiments, image data may be acquired, including a first set of images and a second set of images. The first set of images and the second set of images may be processed using a first slide preparer and a second slide preparer, respectively. The first set of augmented views and the second set of augmented views may be generated based on augmentations applied to the first set of images and the second set of images. For each image, a first visual transformer may generate a first representation of the augmented view of the first augmented view set and train it to enhance the similarity between the first representation and a second representation of the augmented view of the second augmented view set. The second representation may be generated via a second visual transformer.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 385,931, entitled "SAMPLE PROCESSING AGNOSTIC IMAGE REPRESENTATION LEARNING FOR DIGITAL PATHOLOGY," filed December 2, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to digital pathology and, in particular, to tools for analyzing and classifying digital pathology images. [Background technology]

[0003] With advances in digital scanning technology, many laboratories and other institutions that house tissue specimens stored on glass slides have scanned the slides to create digitized images of the tissue specimens. Pathologists or other trained professionals often evaluate the entire slide image (WSI) for evidence of abnormalities in the depicted tissues. Digitized WSIs are often very large—e.g., 100,000 pixels by 100,000 pixels in each of several color channels—making it difficult to efficiently analyze WSIs at a holistic level without applying advanced computational image analysis techniques.

[0004] Traditional image analysis techniques typically begin by obtaining a convolutional neural network (CNN) pre-trained on a common set of natural images (e.g., ImageNet) and then applying transfer training techniques. However, the value of such CNNs can be limited because common natural images typically depict objects that do not appear in WSI and may not be useful for distinguishing features of interest given the unique and specialized aspects of WSI. For example, in hematoxylin and eosin (H&E)-stained slides, most structures appear in blue, purple, and pink tones.

[0005] In digital pathology, it would be desirable to have a CNN pre-trained specifically to extract features from WSI (e.g., H&E slide images). Unfortunately, most large datasets of WSI are unlabeled, making it difficult to leverage supervised learning techniques.

[0006] Furthermore, in digital pathology, tissue samples may be processed using different techniques. These processing techniques can result in differences in the resulting digital pathology images. For example, digital pathology slides obtained from different scanners may have visible (or sometimes invisible) differences due to settings, characteristics, components, or other aspects. These different scanners may use different light sources to capture images, have different lenses, use different software or different software versions, have lenses made of different materials, or have other differences, or a combination thereof. As another example, differences may be due to different slide preparation machines (e.g., stainers) used to prepare the digital pathology slides, different batches of stains, different staining agents, or, in some cases, two different scanning machines of the same model may simply be calibrated differently. As yet another example, differences may be due to different magnifications used with the same or different scanners, different thicknesses of the prepared samples, etc.

[0007] When training a feature encoder, using images obtained through different tissue sample processing techniques (e.g., from different scanners) may preserve features specific to the tissue sample processing technique (e.g., scanner) used to obtain the corresponding image, which may impede downstream classification tasks and analyses. For example, the quality of the features obtained from the images may be reduced because the image representations are less relevant to the biological attributes depicted therein.

[0008] It is therefore desirable to develop and train machine learning models for performing digital pathology analysis that are agnostic to the tissue sample processing techniques used to prepare and capture the corresponding images. Summary of the Invention

[0009] Described herein are systems, methods, and programming for training and implementing machine-trained models for performing digital pathology analysis that are agnostic to the tissue sample processing techniques used to prepare and capture the corresponding digital pathology images.

[0010] Some embodiments include receiving image data including a first image set and a second image set. The first image set and the second image set may include digitized images of a plurality of digital pathology slides processed using a first slide preparer and a second slide preparer, respectively. The first slide preparer and the second slide preparer may each have a set of attributes, and the value of at least one of the attributes may differ between the first slide preparer and the second slide preparer. The image data may be used to generate a first augmented view set and a second augmented view set. The first augmented view and the second augmented view may be generated based on one or more augmentations applied to each image in the first image set and the second image set. A first visual transformer may be trained for each digital pathology slide. Training the first visual transformer may include using the first visual transformer to generate a first representation of the augmented view of the first augmented view set and enhancing a similarity between the first representation and a second representation of the augmented view of the second augmented view set. The second representation can be generated via a second visual transducer, and both the first and second representations can correspond to the same digital pathology slide.

[0011] Some additional embodiments include receiving training data including images of a plurality of biological samples processed using a first slide preparer or a second slide preparer. The first slide preparer and the second slide preparer may each have a set of attributes. The value of at least one of the attributes may differ between the first slide preparer and the second slide preparer. For each of the biological samples, a first representation of one of the images of the biological sample processed using the first slide preparer may be generated using a first encoder. The first representation may be provided to a discriminator to generate a prediction of whether the biological sample corresponding to one of the images was processed using the first slide preparer or the second slide preparer. One or more parameters of the discriminator may be updated based on the generated prediction and a first loss calculated based on metadata associated with one of the images. The metadata may indicate whether the biological sample was processed using the first slide preparer or the second slide preparer. For example, the metadata may indicate that the biological sample was processed using the first slide preparer. One or more parameters of the first encoder may be updated based on the first loss. The updated first encoder may generate an updated first representation of one of the images and be trained to increase the similarity between the updated first representation and a second representation of another of the images of the same biological sample generated using the second encoder.

[0012] The above-disclosed embodiments are examples only, and the scope of the present disclosure is not limited thereto. Particular embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the above-disclosed embodiments. [Brief explanation of the drawings]

[0013] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0014] [Figure 1A] FIG. 1 illustrates an exemplary system 10 used to train and use an agnostic model for processing a sample, according to various embodiments.

[0015] [Figure 1B] FIG. 1 is a diagram of an exemplary process for obtaining an image depicting a biological sample, according to various embodiments.

[0016] [Figure 2A] 1 is a diagram of exemplary image data and exemplary processing technique labels for the image data, according to various embodiments. [Figure 2B] 1 is a diagram of exemplary image data and exemplary processing technique labels for the image data, according to various embodiments.

[0017] [Figure 3] FIG. 1 is a diagram of exemplary training data used to train a model that is agnostic to the processing of a sample, according to various embodiments.

[0018] [Figure 4] FIG. 1 illustrates an exemplary framework of a model used as the basis for an agnostic model for sample processing, according to various embodiments.

[0019] [Figure 5] FIG. 1 illustrates an exemplary framework of another model used as the basis for an agnostic model for sample processing, according to various embodiments.

[0020] [Figure 6A]1A-1C are diagrams of exemplary image tiles from images captured with different tissue processing techniques included in an image set for training a model, according to various embodiments.

[0021] [Figure 6B] 1A-1C illustrate different slide preparers and different settings associated with those slide preparers for generating images representing the same biological sample, according to various embodiments.

[0022] [Figure 7] 1 is a flowchart illustrating an exemplary process for training a model agnostic to the processing of a sample using an average similarity training technique, according to various embodiments.

[0023] [Figure 8] FIG. 5 is a diagram of an exemplary adversarial model used to train an agnostic model for processing samples built on the framework of FIG. 4, in accordance with various embodiments.

[0024] [Figure 9] FIG. 9 illustrates an exemplary training process used to train the adversarial model of FIG. 8, according to various embodiments. [Figure 10] FIG. 9 illustrates an exemplary training process used to train the adversarial model of FIG. 8, according to various embodiments.

[0025] [Figure 11] FIG. 6 is a diagram of another exemplary adversarial model used to train a model agnostic to sample processing built on the framework of FIG. 5 , in accordance with various embodiments.

[0026] [Figure 12A] FIG. 12 illustrates an exemplary training process used to train the adversarial model of FIG. 11 , according to various embodiments. [Figure 12B]FIG. 12 illustrates an exemplary training process used to train the adversarial model of FIG. 11 , according to various embodiments.

[0027] [Figure 13] 1 is a flowchart illustrating an exemplary process for training a model that is agnostic to the processing of a sample using adversarial training techniques, according to various embodiments.

[0028] [Figure 14A] FIG. 2 is a diagram of an exemplary downstream classification subsystem used to train a downstream classifier, according to various embodiments.

[0029] [Figure 14B] 1 is a flowchart illustrating an exemplary process for training a downstream classifier, according to various embodiments.

[0030] [Figure 15A] FIG. 1 is a diagram of exemplary training and validation data used to train a model, according to various embodiments.

[0031] [Figure 15B] FIG. 10 illustrates exemplary downstream classifier training and validation data used to train the downstream classifier, according to various embodiments.

[0032] [Figure 16] 1 is a flowchart illustrating an exemplary process for analyzing an image of a biological sample, according to various embodiments.

[0033] [Figure 17] 10A-10C show images of different biological samples captured using different tissue processing techniques, according to various embodiments.

[0034] [Figure 18A] 1 is a plot showing embeddings generated from multi-scanner data, according to various embodiments.

[0035] [Figure 18B] 10A-10C show various plots of representations produced by an encoder for biological samples included in validation data for an encoder trained using an average similarity approach, according to various embodiments.

[0036] [Figure 19] 10 shows plots of representations produced by an encoder for biological samples included in validation data for an encoder trained using an adversarial approach, according to various embodiments.

[0037] [Figure 20A] 10A-10C show plots of the standard deviation of embeddings over training for the average embedding similarity approach and the adversarial approach, respectively, in accordance with various embodiments. [Figure 20B] 10A-10C show plots of the standard deviation of embeddings over training for the average embedding similarity approach and the adversarial approach, respectively, in accordance with various embodiments.

[0038] [Figure 21] 1 shows an image of a biological sample and a data table of sample data, according to various embodiments.

[0039] [Figure 22A] 1 shows images obtained by two different scanners of the same biological sample, according to various embodiments. [Figure 22B] 1 shows images obtained by two different scanners of the same biological sample, according to various embodiments.

[0040] [Figure 23A] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 23B] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 24A] 1A-1D show various exemplary embedding plots, according to various embodiments. [Figure 24B] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 25A] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 25B] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 26A] 10 illustrates various exemplary embedded plots according to various embodiments. [Figure 26B] 10 illustrates various exemplary embedded plots according to various embodiments.

[0041] [Figure 27] 1 illustrates an exemplary computing system in which one or more embodiments described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0042] In digital pathology, biological samples, such as tissue samples, may be processed using a slide preparation machine. The slide preparation machine may include a slide scanning machine (also referred to interchangeably herein as a "scanner"), a slide stainer, or other machines and systems. Digitized images of the biological sample may be captured using the slide scanning machine. These images can be analyzed to detect possible tissue abnormalities and other anomalies. Traditionally, these images have been analyzed by humans (e.g., physicians), but the advent of machine learning and artificial intelligence has made it possible to perform faster and more robust digital pathology analysis. Furthermore, machine learning models and artificial intelligence can detect associations and / or anomalies that would not be detectable through traditional human review.

[0043] A machine learning model can be trained to perform a task. For example, an image classifier can be trained to receive an image, generate a computer-understandable representation of the image, and output a classification result based on the generated representation. The classification result may indicate one or more classes of objects that the image is determined to depict, the likelihood (e.g., probability) that the image depicts an object belonging to a particular class or classes, or other information.

[0044] To train a machine learning model to perform classification, the model can be provided with training data including images of biological samples (e.g., whole slide images). In some embodiments, images of each biological sample can be captured. Further, in some embodiments, images of each biological sample can be captured using each slide scanner from a set of slide scanners. Slide scanners may include different attributes, such as magnification levels, lens materials, lens thicknesses, software, etc. The purpose of training a model using images captured using different slide scanners is to ensure that the model does not learn features associated with the specific slide scanner used and / or minimize the impact of specific scanner features on the classification results. Furthermore, different processing techniques can be used to process the biological samples before imaging. For example, different slide stainers can be used to prepare the biological samples for digitization. Different slide stainers can use different stains, different batches of stains, other differences, or a combination thereof. The model can be trained using images of biological samples processed using different slide preparation machines to ensure that the model does not learn features associated with the specific slide preparation machine used and / or minimize the impact of those features on downstream classification results. Also, the cut of the tissue may make a difference: for example, one biological sample may be prepared to have a first thickness, while a second biological sample may be prepared to have a second thickness.

[0045] In some embodiments, training the model backbone includes pre-training a feature encoder. The feature encoder is trained to generate meaningful representations of images. This representation describes the underlying biological sample and can be analyzed computationally to perform one or more tasks (e.g., tissue classification). As defined herein, "embedding" refers to a vector representation of features describing an image. An embedding is meaningful if it has features that are compatible with downstream classification or regression tasks. The encoder functions to "encode" image data, representing the image as an embedding in an embedding space. Thus, the encoder can be trained to focus on features that describe the morphological characteristics of the content depicted by the image, while simultaneously avoiding features that are irrelevant or of minimal value to downstream classification tasks. Regression may also be performed. For example, disease progression may be predicted, resulting in a continuous output that can be analyzed using regression techniques.

[0046] In some cases, the data used to train a feature encoder (e.g., a classifier) ​​includes images of biological samples. For example, images of tissue samples exhibiting tumors or other anatomical / biological abnormalities can be used to train a classifier to detect their presence in images of other biological samples. Tissue abnormalities that can be detected from WSI include, by way of example only and without limitation, inflammation, pigmentation, degeneration, heterotopia, hypertrophy, increased mitosis, mononuclear cell infiltration, inflammatory cell infiltration, inflammatory cell foci, decreased glycogen, glycogen accumulation (diffuse or concentrated), extramedullary myelogenesis, extramedullary hematopoiesis, extramedullary erythropoiesis, single cell necrosis, diffuse necrosis, marked necrosis, coagulative necrosis, apoptosis, karyomegaly, periductal, increased cellularity, glycogen deposition, lipid deposition, microgranulomas, congestion, Kupffer cell pigmentation, increased hemosiderin, histiocytosis, hyperplasia, or vacuolization, among others.

[0047] In some embodiments, a given biological sample is placed on a glass slide prepared using one or more slide preparation machines. The slide preparation machines may include, for example, one or more slide stainers (also referred to interchangeably herein as "stainers"), one or more slide scanning machines (e.g., scanners), or other machines. Each slide preparation machine may have a set of attributes. The values ​​of each set of attributes may be the same or similar between different slide preparation machines. Some embodiments include the value of at least one of the attributes being the same (or nearly the same) between two separate slide preparation machines. Some embodiments include the value of at least one of the attributes being the same (or nearly the same) between the same slide preparation machines used to prepare and digitize images of the biological sample. For example, a first batch of images may be captured using a scanner of biological samples using one stain, and a second batch of images may be captured using a scanner of biological samples using a different stain. The set of attributes may indicate the stain, such as hematoxylin or eosin, used to stain the slide, the thickness of the tissue used in the biological sample, or other attributes.

[0048] The slides may be imaged using one or more slide scanning devices (e.g., scanners). Attributes of the slide scanning devices may include the level of magnification, lens material, lens thickness, scanning software, or other attributes. In some embodiments, the value of at least one attribute of a first scanner may differ from the value of at least one attribute of a second scanner. Generally, the value of an attribute is independent of the morphological characteristics of the underlying biological sample. The scanners may capture an image of the biological sample, which in some embodiments may be referred to as a whole slide image (WSI).

[0049] As used herein, a WSI is a very large-format digital image (e.g., 100,000 x 100,000 pixels) that may result from digitizing a physical slide of a biological sample (e.g., tissue) into a high-resolution image file or may be output directly by a medical scanning device. WSIs are typically saved in the highest resolution format possible given the nature of the image being captured to avoid misdiagnosis of the tissue shown in the WSI due to artifacts that typically result from image compression and manipulation. WSIs often contain several orders of magnitude more pixels than typical digital images and may include resolutions of 100,000 pixels x 100,000 pixels (e.g., 10,000 megapixels) or greater.

[0050] Analysis of WSIs is a labor-intensive process requiring highly specialized individuals with the knowledge and dexterity to review the WSIs, recognize and identify anomalies, classify the anomalies, label the WSIs, and potentially diagnose the tissues indicated by the WSIs. Additionally, because WSIs are used on a wide range of tissue types, accurate analysis and diagnosis require further specialization by individuals with the knowledge and skills to identify anomalies.

[0051] Therefore, due to the labor-intensive and knowledge-intensive nature of the work, WSIs are considered candidates for automation of certain functions. However, the large size of WSIs makes typical image analysis techniques ineffective, time-consuming, and expensive. Implementing standard image recognition and deep learning techniques is impractical, requiring multiple analyses of numerous WSI samples to increase accuracy. The technology described herein aims to solve the problem of automating feature recognition in WSIs and enable the development of novel data analysis and presentation techniques that have not previously been optimized for these images (e.g., WSIs), which have different characteristics compared to natural images.

[0052] When training a classifier using images captured from different sources or using different methods and / or protocols, inherent differences can arise. For example, an image of a biological sample captured using one scanner may differ from an image of the same biological sample captured using a different scanner. As an example, referring to Figures 22A and 22B, images 2200 and 2250 show examples of the same biological sample captured using two different slide scanners. While images 2200 and 2250 have different magnifications, there are other differences in the features detected by either scanner when imaging the biological sample, which can affect the generated embeddings and introduce errors into downstream classification. As another example, an image of a biological sample prepared using one slide preparation technique may differ from an image of the same biological sample captured using a second, different slide preparation technique. These differences can affect the embeddings generated by the encoder, which can affect downstream classification. It would therefore be beneficial to have a technique for training an encoder for use in downstream biological sample image classification tasks that is agnostic to the slide preparation machine (e.g., scanner, stainer) used to capture the image of the underlying biological sample, and similarly, or alternatively, agnostic to other origin-specific characteristics of the image.

[0053] Particular embodiments may repeat one or more steps of the exemplary process, where appropriate. Although this disclosure describes and illustrates certain steps of the exemplary process as occurring in a particular order, this disclosure contemplates any suitable steps of the exemplary process occurring in any suitable order. Moreover, while this disclosure describes and illustrates an exemplary process, this disclosure contemplates any suitable process including any suitable steps, and may include all, some, or none of the steps of the exemplary process, where appropriate. Moreover, while this disclosure describes and illustrates particular components, devices, or systems that perform certain steps of the exemplary process, this disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the exemplary process.

[0054] In all exemplary embodiments described herein, appropriate options, features, and system components may be provided to enable the collection, storage, transmission, information security measures (e.g., encryption, authentication / authorization mechanisms), anonymization, pseudonymization, segregation, and aggregation of information in accordance with applicable laws, regulations, and rules. In all exemplary embodiments described herein, appropriate options, features, and system components may be provided to enable protection of the privacy of certain individuals, including, by way of example and without limitation, generating reports regarding what personal information is or has been collected and how it is or will be used, enabling the deletion or erasure of any collected personal information, and / or enabling control over the purposes for which any collected personal information is used.

[0055] 1A illustrates an exemplary system 100 used to train and use an agnostic model for sample processing, according to various embodiments. System 100 may include a computing system 102, slide preparers 120 (e.g., slide preparers 120-1 through 120-M), client devices 130 (e.g., client devices 130-1 through 130-N), databases 140 (e.g., image database 142, training data database 144, model database 146, validation data database 148), or other components. In some embodiments, the components of system 100 may communicate with each other using a network 150, such as the Internet.

[0056] In some embodiments, slide preparer 120 may prepare and digitize slides of biological specimens using one or more slide preparation techniques. These slide preparation techniques may impart different characteristics to the representation of the prepared biological specimen, which may affect downstream classification tasks. By way of example, slide preparers 120 may be the same type / model, but portions of slide preparers 120 may be different. In particular, each slide preparer 120 may have a set of attributes, and the values ​​of those attributes may be the same or different across different slide preparers, or even the same slide preparer. Slide preparers may be used to prepare slides of biological specimens for digitization, to scan slides to obtain images of the biological specimens, or to perform other tasks, or a combination thereof.

[0057] For example, the slide preparer 120 may include a slide stainer (e.g., stainer) and a slide scanner (e.g., scanner) made by the same or different manufacturers. The slide preparers 120 may have different attributes, such as settings, features, components, or other differences. For example, different scanners may use different light sources to capture images, have different lenses, use different software or different software versions, have lenses made of different materials, have different lens thicknesses, or have other differences, or combinations thereof. In some cases, two different slide scanners of the same model may be calibrated differently. As another example, different stainers may use different stains, different batches of the same stain, use different tissue thicknesses for sampling, or other differences, or combinations thereof. For example, one stain batch may be used for biological samples captured by one or more slide preparers 120. In some embodiments, one or more steps may be performed on the biological sample before it is processed by the slide preparer 120. For example, the biological sample may be cut to different thicknesses. While the cutting process is typically performed by a human, some embodiments include using an automated sample cutter. Regardless of whether the tissue is cut by a human or a machine, each cut, and even each tissue sample, may be different. Differences in tissue thickness may also affect digital pathology analysis. Therefore, the techniques described herein may be able to minimize the impact that variations in tissue sample thickness may have on downstream classification tasks.

[0058] The client devices 130 may include client devices 130-1 through 130-N. Each client device 130 may communicate with one or more components of the system 100 via the network 150 and / or via a direct connection. A client device 130 may refer to a computing device that may interface with various components of the system 100 to control one or more tasks, perform one or more actions, or perform other operations. For example, a client device 130 may be configured to receive and display images of a scanned biological sample. Exemplary computing devices to which the client device 130 may correspond include, but are not limited to, a desktop computer, a server, a mobile computer, a smart device, a wearable device, a cloud computing platform, or other client devices, and other enumerations are not meant to be limiting. In some embodiments, each client device 130 may include one or more processors, memory, communications components, display components, audio capture / output devices, image capture components, or other components, or combinations thereof. Each client device 130 may include a wearable device, a mobile terminal, a fixed terminal, or any type of other device.

[0059] It should be noted that while one or more operations are described herein as being performed by particular components of computing system 102, these operations may, in some embodiments, be performed by other components of computing system 102 or other components of system 100. By way of example, while one or more operations are described herein as being performed by components of computing system 102, these operations may, in some embodiments, be performed by components of slide preparer 120 and / or client device 130. It should be noted that while some embodiments are described herein with respect to machine learning models, in other embodiments other predictive models (e.g., statistical models or other analytical models) may be used instead of or in addition to machine learning models (e.g., in one or more embodiments, statistical models in place of machine learning models, and non-statistical models in place of non-machine learning models).

[0060] The computing system 102 may include one or more subsystems, such as, for example, a training data generation subsystem 110, a model training subsystem 112, a downstream classification subsystem 114, or other subsystems.

[0061] The training data generation subsystem 110 may be configured to generate training data used to train a model, generate validation data to validate a model after training, update the training / validation data, or perform other operations related to preparing data for use in training a model. In some embodiments, the training data generation subsystem 110 may be configured to acquire images depicting biological samples. Each image may be captured using one of the slide preparers 120 (e.g., scanners 124). For example, image 1 may be captured using a first slide preparer, while image 2 may be captured using a second slide preparer, where image 1 and image 2 are both images of the same biological sample (e.g., a tissue sample). The slide preparers 120 may include a set of attributes, and in some embodiments, the value of at least one of the attributes may be the same. For example, a first scanner A of the first slide preparer may be used to capture an image of the biological sample, and a second scanner B of the second slide preparer may be used to capture an image of the same biological sample. In some embodiments, the slide preparers 120 may employ different slide preparation processes. For example, a first stainer of a first slide preparation machine may use a first staining agent to stain a slide of a biological sample, and a second stainer of a second slide preparation machine may use a second staining agent to stain a slide of a biological sample.

[0062] In some embodiments, some or all of the captured images may include metadata indicating the slide preparation machine used for those images. The associated metadata may indicate the origin of the scanner, the origin of the stainer, slide batch information, or other information that classifies attributes of the slide preparation process used to prepare the slide and / or its captured image. An example of image database 142 may be shown in FIG. 2A.

[0063] FIG. 1B is a diagram of an exemplary process 180 for obtaining an image depicting a biological sample, according to various embodiments. As described above, the slide preparer 120 may include a stainer 122, a scanner 124, or other devices used to prepare and capture images for digital pathology analysis. While FIG. 1B depicts the slide preparer 120 including a stainer 122 and a scanner 124, each may be separate components (e.g., a stainer coupled to the slide preparer 120 that includes a scanner, or a scanner coupled to the slide preparer 120 that includes a stainer). In some embodiments, the process 180 may include providing a sample 126 to the slide preparer 120. In particular, the sample 126 may be provided to the stainer 122, which may perform one or more staining processes to prepare the slide for digitization. The prepared slide may then be scanned by the scanner 124 to obtain an image 128.

[0064] In some embodiments, the stainer 122 may apply a stain (e.g., H&E) to a slide of the sample 126. Different stains may be used by different stainers 122. Furthermore, even for the same stain, the concentration, amount, or other aspects of the application of the stain may vary from sample to sample and from stainer to stainer.

[0065] Before staining, biological samples may be cut to different thicknesses. Cutting can be performed by machine, but humans can also cut biological samples. Whether performed by machine or by humans, the cutting itself can have variability that can affect downstream analysis. For example, one biological sample may be cut to a thickness of 2 mm, while another biological sample may be cut to a thickness of 4 mm. Digital pathology images capturing these biological samples may contain different characteristics, even if they are obtained from the same source biopsy. However, the underlying morphological characteristics of the samples should remain the same because they are from the same source.

[0066] In addition to the different tissue processing techniques available for processing samples using the stainer 122, the slide preparer 120 may also provide different tissue processing capabilities. Similar to the tissue processing techniques described above, characteristics specific to the slide preparer 120 may also be encoded in the images 128, which may affect downstream classification. For example, different scanners 124 may use different lenses, magnification settings, lens thicknesses, software, or digitization techniques to capture the images 128 of the sample 126. Some embodiments include images 128 with multiple magnification levels (e.g., 5x, 10x, 20x, 40x, etc.). In this scenario, the images 128 may form a pyramidal format. In some embodiments, one of the magnification levels may be selected as a target magnification level (e.g., 40x). Tiles may be obtained from the image at that constant magnification. Some embodiments include using images from multiple magnification levels together. Additionally, the slide preparer 120 may be of different makes, models, versions, or have other differences that may be imparted to the images 128.

[0067] As used herein, a scanner refers to a computing system and imaging system capable of scanning, digitizing, compressing, storing, retrieving, and / or viewing biological specimen slides. A scanner may include a portion for loading a specimen into the machine, an image capture component, a processor, memory, a network interface, a display (or other input / output device), or other components, or a combination thereof. Some scanners may be capable of reading various numbers of slides. For example, a scanner may read 100 or more slides, 200 or more slides, 500 or more slides, or other numbers of slides. The speed at which images of the slides are captured and the magnification used to capture the images may vary depending on the scanner, software, lenses, or other considerations.

[0068] As an example, referring to FIG. 2A , image database 142 may be configured to store image data 200. Image data 200 may include N images 202 (e.g., whole slide images) depicting P biological samples. One or more images may be captured for each biological sample. For example, one or more images (e.g., 10 or more images, 100 or more images, 1,000 or more images, etc.) may be captured for each of the P biological samples. Each image may be processed and digitized using one of a predetermined set of slide preparers (e.g., slide preparer 120). For example, slide preparer 120 may include M slide preparers. Each slide preparer may include a slide stainer and a slide scanner. Thus, there may be M slide stainers and / or M slide scanners. In some embodiments, each biological sample may be prepared using some or all of the M slide stainers. For example, a first stainer may prepare a slide of a first biological sample using a first staining technique, while a second stainer may prepare a slide of the first biological sample using a second staining technique. In some embodiments, each biological sample may be scanned using some or all of the M scanners. For example, for a first biological sample, scanner 1 may be used to capture image 1 of biological sample 1, while scanner 2 may be used to capture image 2 of biological sample 1.

[0069] In some embodiments, image data 200 may be organized in a data structure based on the respective biological sample represented by the image. For example, all images captured for biological sample 1 may be stored in association with one another, while all images captured for biological sample N may be stored in association with one another. Those skilled in the art will recognize that alternative organizational schemes may be used, and the foregoing are merely exemplary.

[0070] In some embodiments, each image 202 may have metadata associated with it. For example, each image 202 may include a processing technique label (e.g., scanner label, stain label), a sample identifier, a timestamp indicating when that particular image was captured, a slide label indicating the slide used to capture the image, tissue thickness, geographic location, operator identifier, or other information, or a combination thereof. Alternatively, only a portion of the images 202 may include a processing technique label, sample identifier, or other metadata.

[0071] In some embodiments, a processing technique label may indicate the value of an attribute related to the processing of a particular image. For example, processing technique label 204-1 may indicate one or more processing techniques used to create image 202-1. While only a single processing technique label is shown in FIG. 2A, those skilled in the art will recognize that multiple processing technique labels may be associated with a given image. By way of example, with reference to FIG. 2B, processing technique label 204 may include a scanner label 252, a stainer label 254, a specimen thickness label 256, or other metadata indicating the value of an attribute related to the slide preparation machine used to prepare the slide and corresponding image. For example, scanner label 252 may indicate the scanner used to capture an image of a given biological sample, stainer label 254 may indicate the stain and / or process used to stain the biological sample slide, and specimen thickness label 256 may indicate the thickness of the biological sample. In some embodiments, a magnification label indicating a fixed magnification level used for training is also included.

[0072] The scanner label 252 may include metadata associated with an image that may indicate the scanner used to capture the corresponding image. For example, the processing technique label 204-1 associated with image 202-1 may include a first scanner label (e.g., scanner label 252) indicating that a first scanner was used to capture image 202-1. Similarly, the processing technique label 204-2 associated with image 202-2 may include a second scanner label indicating that a second scanner was used to capture image 202-2, and the processing technique label 204-M associated with image 202-N may include an Mth scanner label indicating that an Mth scanner was used to capture image 202-N.

[0073] In some embodiments, two or more images 202 may be prepared using the same slide preparer or the same components of a slide preparer. For example, two or more images 202 may be captured with the same scanner. In such cases, corresponding scanner labels may contain the same information. For example, if images 202-1 and 202-2 were both captured using scanner 1, then the corresponding scanner labels may be equivalent or otherwise indicate the same scanner (e.g., device identifier, port address, etc.).

[0074] Sample ID 206 indicates the biological sample depicted in the corresponding image. For example, image 202-1 may include sample ID 206-1, which indicates the biological sample depicted by image 202-1. Similarly, image 202-2 may include sample ID 206-2, which indicates the biological sample depicted by image 202-2, and image 202-P may include sample ID 206-P, which indicates the biological sample depicted by image 202-P. In some embodiments, two or more of images 202 may depict the same biological sample. In such cases, corresponding sample IDs may include the same information. For example, if images 202-1 and 202-2 were both images of a first biological sample, sample ID 206-1 and sample ID 206-2 may be equivalent or otherwise indicate the same biological sample (e.g., slide number, clinical trial information, etc.).

[0075] In some embodiments, the training data generation subsystem 110 may be configured to create training data and / or validation data for training a machine learning model based on the image data 200. For example, the training data generation subsystem 110 may organize the image data 200 so that images related to the same biological sample are grouped together, images captured by the same scanner are grouped together, biological samples related to the same clinical trial or treatment group are grouped together, etc. By way of example, with reference to FIG. 3 , the training data database 144 may include training data 300 generated by the training data generation subsystem 110 based on the image data 200. In some embodiments, the training data generation subsystem 110 may be configured to identify the biological sample associated with each of the images 202. For example, a sample identifier indicating the biological sample associated with a given image may be extracted from the metadata of each image. The training data generation subsystem 110 may select images with similar sample IDs and group the images together. 3, images 202-1, 202-2, 202-X, and 202-Y may each be grouped together based on each of these images having the same sample ID 206-1. Sample ID 206-1 may refer to biological sample 1, and thus, each image associated with biological sample 1 may be grouped together.

[0076] In some embodiments, the training data generation subsystem 110 may be configured to further organize images based on the slide preparation machine used to prepare and / or digitize the biological sample. For example, the training data generation subsystem 110 may organize images based on scanner origin. The training data generation subsystem 110 may detect scanner labels associated with images 202 and group together images captured by the same scanner. For example, with reference to FIG. 3 , images 202-1 and 202-2 may be grouped into a first image set 310a, and images 202-X and 202-Y may be grouped into a second image set 310b. Images 202-1, 202-2, 202-X, and 202-Y each relate to the same biological sample, indicated by sample ID 206-1; however, within this collection of images, the training data generation subsystem 110 may group images based on their corresponding processing technology labels (e.g., scanner labels). For example, the first image set 310a may include images 202-1 and 202-2 based on the fact that images 202-1 and 202-2 each have the same processing technique label, processing technique label 204-1. This may indicate that images 202-1 and 202-2 were created and processed using the same slide preparation machine. For example, images 202-1 and 202-2 may have been captured using the same scanner in addition to depicting the same biological sample (e.g., biological sample 1). As another example, the second image set 310b may include images 202-X and 202-Y based on the fact that images 202-X and 202-Y each have the same processing technique label, processing technique label 204-2. This may indicate that images 202-X and 202-Y were created and processed using the same slide preparation machine. For example, images 202-X and 202-Y may be images depicting the same biological sample (eg, biological sample 1) and may have been captured by the same scanner.

[0077] In some embodiments, training data 300 may include images associated with N biological samples. For each biological sample, one or more image sets may be generated by training data generation subsystem 110. Furthermore, each slide preparer may prepare and process the same biological sample. For example, each scanner may be used to capture one or more images of the same biological sample (but only some scanners may be used to capture some biological samples).

[0078] In some embodiments, different organizational schemes may be used by the training data generation subsystem 110. For example, in some embodiments, an image set may be generated that includes images captured by multiple scanners, such as by merging the first and second image sets 310a and 310b. Additionally, image sets may be generated that include images depicting multiple biological samples. For example, an image set may be generated that includes images of two or more biological samples.

[0079] Each image set may include one or more, 100 or more, 1,000 or more, 1,000,000,000, or other number of images. Thus, while image sets 310a and 310b are illustrated as including two images, each image set may include many more images. Furthermore, image sets may include different amounts of images. Any number of biological samples may be scanned and may be dependent on the clinical trial. For example, N biological samples may include one or more biological samples, 10 or more biological samples, 100 or more biological samples, 1,000 or more biological samples, or other amounts.

[0080] As described herein, it should be understood that processing techniques associated with different slide preparation machines may differ. For example, different scanners may have different settings, perform different tasks, be manufactured by different entities or by different methods, and different stainers may use different staining techniques, different staining agents, different tissue thicknesses, or have other differences. Thus, one skilled in the art will recognize that other differences may be used to develop a sample-agnostic model. For example, a sample-agnostic model may be agnostic to staining agents, staining techniques, tissue thickness, etc. In such a scenario, the image may include associated metadata indicating, for example, the staining agent used on the biological sample depicted by the image.

[0081] In some embodiments, the training data generation subsystem 110 can be configured to generate validation data used to validate (e.g., test) a model. The validation data can include images of biological samples captured using a known slide preparer. The validation data can be used to test the accuracy of a model in correctly inferring the image's corresponding slide preparer (e.g., scanner origin), or to test the accuracy of a model that is unaffected by the image's slide preparer and / or processing performed by it when used for downstream classification.

[0082] As noted above, inherent differences between datasets captured using different methods or protocols can impact a model's ability to perform downstream classification. A known problem with traditional digital pathology analysis is that features learned by a model may not be scanner-agnostic. For example, referring to FIG. 18A, plot 1800 shows an example of a t-distributed stochastic neighbor embedding (TSNE) plot of embeddings obtained from multi-scanner data. Plot 1800 can be created by generating embeddings from image data containing images of biological samples captured using two scanners (although other attributes of the slide preparation process may also differ). Each scanner is represented by one of the colors within plot 1800. For example, the embeddings from one scanner are represented by a "yellow" or light-colored clustering, while the embeddings from the other scanner are represented by a "purple" or dark-colored clustering. As can be seen from plot 1800, the two clusters are easily distinguishable. This indicates that even though the same biological sample was imaged with the same two scanners, the scanner can affect the embedding generation process, and the affected embeddings can affect downstream classification tasks. Thus, a technical challenge exists to generate a model configured to perform downstream digital pathology classification tasks while being agnostic to the scanner used to capture the images and / or agnostic to other non-morphological characteristics of the images (e.g., stains, magnification, tissue sample thickness, etc.).

[0083] This specification describes technical solutions to overcome the aforementioned technical problems. In some embodiments, a model is developed that is agnostic to the processing of samples to perform digital pathology classification tasks. The model may employ a student-teacher framework as its backbone. One example of this backbone is the "label-free self-distillation" or DINO framework. Another example of a backbone is the "bring your own latent bootstrap" or BYOL framework. This specification describes both frameworks, as well as improvements made to the frameworks to develop agnostic models for the processing of samples.

[0084] In some embodiments, the encoder selected for use as the model architecture is a convolutional neural network. For example, the ResNet-18 architecture can be used as the base framework for the encoder. ResNet-18 includes 18 layers organized as four residual blocks. A residual block is a block that applies identity mapping, where the input to one layer is passed directly to another layer. In some embodiments, each residual block is connected not only to the next layer in the network, but also to skipped layers further down the network. The connection between the residual block and the lower network layer is called a shortcut or skip connection, which can bypass one or more layers. Mathematically, if the input x is the input to a layer and the output is F(x), the output of the residual block can be expressed as Y = F(x) + x. In particular, ResNet-18 is a pre-trained convolutional neural network (CNN) trained on images from the ImageNet database, which classifies images into one of 1,000 categories. The input image size for ResNet-18 is 256 × 256.

[0085] DINO-based framework

[0086] In some embodiments, the base framework used to build an agnostic model for sample processing is the DINO framework. DINO is a self-supervised learning approach based on knowledge distillation, which propagates information from a small set of annotations to a large unlabeled database by having a network mimic the output of another network. This knowledge distillation process can also be extended to use cases where images do not have labels. For a full description of the DINO framework, see "Emerging Properties in Self-Supervised Vision Transformers," Caron et al., 2021, the entire contents of which are incorporated herein by reference.

[0087] As an example, referring to FIG. 4 , the DINO framework 400 includes two networks: a student network formed with a student encoder 410 and a softmax layer 412, and a teacher network including a teacher encoder 420, a softmax layer 422, and a centering layer 424. An image 402, such as an image of a biological sample, may be acquired (e.g., from the image database 142). In some embodiments, the image 402 may include metadata indicating the scanner used to scan the biological sample and generate the image 402. Additionally or alternatively, the image 402 may include metadata indicating the stainer used to apply the stain to the biological sample, the thickness of the biological sample, the level of magnification used, or other slide preparation information. In some embodiments, one or more augmentations may be performed on the image 402 to obtain augmented images 404a and 404b. Various augmentations that may be performed on the image 402 include, but are not limited to, inversion, blurring, cropping, color jittering, or other image augmentations, or combinations thereof. In some embodiments, the training data generation subsystem 110 may be configured to perform augmentation of the image 402 to obtain augmented images (also interchangeably referred to as “augmented views”) 404 a, 404 b. In some embodiments, the augmented images 404 a and 404 b may have different augmentations performed on them. For example, the augmented image 404 a may include a small crop (interchangeably referred to herein as a “local crop”), and the augmented image 404 b may include a large crop (interchangeably referred to herein as a “global crop”). By providing the local crop to the student encoder 410 while providing the global crop (and in some embodiments, also the local crop) to the teacher encoder 420, a local-to-global mapping can be obtained.

[0088] The student encoder 410 and the teacher encoder 420 may have the same architecture. For example, the student encoder 410 may use a first set of parameters θ sを The teacher encoder 420 includes a second set of parameters θ tThe student encoder 410 is trained to match the output of the teacher encoder 420, and its hyperparameters are updated based on the average (e.g., exponential moving average) of the student encoder 410. In other words, the parameters θ t is the parameter θ s In some embodiments, the parameter θ s can be learned by minimizing the loss function (Equation 1) using stochastic gradient descent:

number

[0089] The model training subsystem 112 may be configured to train a student encoder 410 and a teacher encoder 420. In some embodiments, the student encoder 410 and the teacher encoder 420 may each output an embedding that represents an input image. The model training subsystem 112 may be configured to pass the embedding from the student encoder 410 to a softmax layer 412, which outputs p1. The output p1 may include a factor indicating the likelihood that the image depicts content associated with class K. The probability may be calculated using Equation 2:

number

[0090] In Equation 2, τ represents the temperature parameter that controls the sharpness of the output distribution, and g represents the student or teacher network.

[0091] The model training subsystem 112 may be configured to pass the embedding output from the teacher encoder 420 to a centering layer 424. The centering layer 424 may be used to center the output embeddings from the teacher encoder 420 with the mean calculated over the batch of augmented views. The model training subsystem 112 may be configured to maximize the similarity of the temperature-softmax outputs of the student and teacher networks, as indicated by outputs p1 and p2. Using the notation of Equation 2, outputs p1 and p2 each represent a probability P s (x1) and P t (x2). The similarity can be measured as the cross-entropy loss shown in Equation 3:

number

[0092] In some embodiments, the technical problem of Equation 3 can accommodate more views than the local-global cropping strategy of the DINO framework. For example, different views of an image (e.g., augmented image 404a, image 404b) may be created by applying one or more image augmentations (e.g., blurring, flipping, rotating, color distortion, and / or cropping, etc.). Augmented images may also be referred to interchangeably herein as "augmented views." The augmentations may be performed by the training data generation subsystem 110 and / or the model training subsystem 112.

[0093] For one sample, the expansion result is x g1 and x g2 We obtain a set of distorted views V consisting of two global crops, denoted as , and multiple local crops. In some embodiments, all crops are fed to the student encoder 410, while only the global crop is passed to the teacher encoder 420. This allows us to learn the local-global correspondences.

[0094] The model training subsystem 112 may be further configured to solve the DINO optimization problem, which can be expressed as Equation 4:

number

[0095] In Equation 4, x and x' refer to the augmented views of the images fed to the teacher encoder 420 and the student encoder 410, respectively.

[0096] In some embodiments, the model training subsystem 112 may be configured to propagate gradients through the student encoder 410. In some cases, gradients may be propagated only through the student encoder 410. The model training subsystem 112 may apply gradient stopping to the teacher network (e.g., the teacher encoder 420, the centering layer 424, the softmax layer 422) to prevent backpropagation. In some embodiments, the model training subsystem 112 may use a momentum encoder (e.g., the teacher encoder 420) to dynamically construct the teacher network. For example, the parameter θ t is the parameter θ s It may be updated by an exponential moving average (ema) of the weights.

[0097] BYOL Framework

[0098] In some embodiments, the base framework used to build a model that is agnostic to sample processing is the BYOL framework. The BYOL model is a self-supervised image representation learning process. The BYOL architecture includes two neural networks: an "online" neural network and a "target" neural network. The online neural network and the target neural network interact and learn from each other. For example, for a given image, an augmented version of the image can be created, and the online neural network is trained using the first augmented version of the image to predict a target neural network representation for the second augmented version of the image. The rationale for this process is that a representation for an augmented view of one image should be able to predict a representation for another augmented view of the same image. Therefore, the BYOL process involves training a model to generate augmented representations by predicting the target representation using the target model.

[0099] By way of example, referring to Figure 5, a BYOL framework 500 is illustrated. In some embodiments, the framework 500 is implemented using a representation y (e.g., for classifying images). θ The framework 500 may include a first encoder 510 and a second encoder 530. The first encoder 510 and the second encoder 530 are referred to interchangeably herein as the "online encoder 510" and the "target encoder 530." In some embodiments, the first encoder 510 and / or the second encoder 530 may be a convolutional neural network.

[0100] The first encoder 510 may be defined by a set of weights θ, and the first stage: f Encoding θ, second step: g θ and the third step:q θThe second encoder 530 may be defined by another set of weights ξ and may include similar first and second stages, predicting f ξ Encode the g In some embodiments, the model training subsystem 112 may configure the second encoder 530 to train the first encoder 510 where the weights ξ are exponential moving averages of the weights θ. Thus, after each round of training, the model training subsystem 112 may update the weights ξ according to Equation 5:

number

[0101] In some embodiments, the model training subsystem 112 may be configured to use the framework 500 to take an image 502 and apply two (or more) image augmentations to the image 502 to generate two augmented versions of the image 502 (augmented view 504, v, and augmented view 506, v'). The model training subsystem 112 may randomly select the image 502 from a training set of images (e.g., a set of all slide images depicting a biological sample, etc.). In some embodiments, the image 502 comprises a sample image to be analyzed by the framework 500 or a downstream classifier trained based on the framework 500. For example, the first encoder 510 may be trained and then used to classify the image. The augmentations t and t' applied to the image 502 generate a first augmented view 504, v, and a second augmented view 506, v' of the input image 502. In some embodiments, the model training subsystem 112 may be configured to select the augmentations t, t' from a set of predefined augmentations. The extension t may be selected from a first set of predefined extensions, and the extension t' may be selected from a second set of predefined extensions. In some cases, the first and second predefined sets of extensions may share one or more common types of extensions (e.g., both sets may include rotational extensions), but the predefined sets of extensions may not share any common types of extensions.

[0102] The model training subsystem 112 may be configured to provide the first augmented view v and the second augmented view v′ to the encoder 510 and the encoder 530, respectively. The first augmented view v and the second augmented view v′ are then encoded using the encoder 510 and the encoder 530, respectively, to generate a first representation, such as an embedding 512. y θ and a second representation y′ such as embedding 532 ξ For example, the encoding by the first encoder 510 and the second encoder 530 may be defined by equations 6a and 6b, respectively:

number

[0103] The model training subsystem 112 generates a first representation y θ (e.g., embedding 512) and a second representation y' ξ (e.g., embedding 532) may be provided to a first (online) projector 514 and a second (target) projector 534. Embedding 512 and embedding 532 may be provided to a first (online) projector 514 and a second (target) projector 534, respectively. θ The first projection 516 denoted as z′ ξ , which may subsequently be used by the first projector 514 and the second projector 534 to generate a second projection 536, denoted as . For example, the projections by the first projector 514 and the second projector 534 are defined by Equations 7a and 7b, respectively:

number

[0104] In some embodiments, the online predictor 518 calculates q based on the projection 516. θ (z θ ) which is a prediction of the second projection 536. After generating the prediction 520, the model training subsystem 112 normalizes (e.g., normalizes) both the prediction 520 and the projection 536. It is configured to obtain the image (TIFF2026501098000009.tif6170-normalized).

number

[0105] loss function, TIFF2026501098000011.tif7170TIFF2026501098000012.tif7170, also known interchangeably as BYOL loss, is defined as follows:

number

[0106] After each training step, the model training subsystem 112 may update the weights according to Equation 5 and Equation 10:

number

[0107] In Equation 11, TIFF2026501098000015.tif6170TIFF2026501098000016.tif6170 is the learning rate, which may be predefined, TIFF2026501098000017.tif7170, in formula TIFF2026501098000018.tif7170 refers to a symmetrized loss function obtained by feeding the first augmented view 504 to the first encoder 510 and the second view 506 to the second encoder 530.

[0108] After training is complete, the model training subsystem 112 stores the encoding of the first encoder 510 in the model database 146. TIFF2026501098000019.tif6170 can be stored. TIFF2026501098000020.tif6170 may be used to train a classifier to classify images. In particular, the downstream classification subsystem 114 uses the data stored by the model database 146 from the first encoder 510. In some embodiments, the first encoder 510 may be configured to train a classifier to classify images of biological samples encoding TIFF2026501098000021.tif6170. TIFF2026501098000022.tif6170 may be stored in the model database 146 along with metadata. For example, the metadata may indicate when the first encoder 510 was trained and / or validated.

[0109] In some embodiments, enhancements that may be applied to the input image 502 may include, but are not limited to, random cropping, limiting, flipping about one or more axes, color distortion, adjusting the brightness, contrast, saturation, or hue of at least one image, converting to grayscale, blurring, or other enhancements, or combinations thereof, and the like, and other lists are not intended to be indicative of such enhancements.

[0110] In some embodiments, encoders 510 and 530 may be convolutional neural networks (CNNs) having multiple layers. For example, encoders 510 and 530 may include 3 or more layers, 10 or more layers, 50 or more layers, 100 or more layers, 200 or more layers, etc. An exemplary deep learning model that can be used for encoders 510 and 530 is ResNet (e.g., ResNet-18).

[0111] This paper describes two different techniques for developing agnostic models for sample processing. The first technique, the average embedding similarity approach, does not require modification of the DINO or BYOL frameworks for implementation. The second technique, the adversarial approach, results in improved accuracy when used for downstream classification, but requires modification of the DINO or BYOL frameworks.

[0112] Average embedding similarity approach.

[0113] In some embodiments, available data, such as image data 200, may include images of biological (e.g., tissue) samples processed using two (or more) slide preparers (e.g., scanned with different scanners, stained with different stainers, etc.). For example, as seen in FIG. 3, training data 300 may be organized such that each biological sample includes one or more images captured by each available slide preparer (e.g., slide preparer 120). In some embodiments, each slide preparer may be configured to stain the biological sample and / or capture images of slides containing the stained biological sample. Training data generation subsystem 110 may be configured to organize images 202-1 and 202-2 into a first image set 310a because both images 202-1 and 202-2 have the same processing technique label 204-1 (e.g., both images 202-1 and 202-2 may be captured with a first scanner using a first stain, a first magnification level, a first tissue thickness, etc.). Similarly, images 202-X and 202-Y, which also depict biological sample 1, may have the same processing label 204-2 (e.g., both images 202-X and 202-Y may be captured using a second scanner, using a second stain, using a second magnification level, a second tissue thickness, etc.).

[0114] In some embodiments, the training data generation subsystem 110 may be configured to divide each image (e.g., image 202) into multiple tiles. The tiles may each be the same size and / or shape, and in some cases may overlap one another. For example, each tile may represent a patch of 512 x 512 pixels. In some embodiments, the image from which the tiles are derived may be larger in size (e.g., 100,000 x 100,000 pixels). For example, each tile may be formed by dividing the WSI image into a set of tiles, each having the same size (which may or may not overlap). In some embodiments, the tiles may be randomly selected from a larger image.

[0115] In some embodiments, random dilation may be performed on tiles. For example, an image (e.g., an entire slide image) may be divided into tiles, and some or all of the tiles may be dilated using one or more predefined image dilations. Alternatively, an image may be dilated, and tiles may then be derived therefrom. To that end, in some embodiments, images included in training data 300 may correspond to image tiles (e.g., obtained by dividing an image into tiles), although in some cases training data 300 may include images (e.g., entire slide images) and / or image tiles. Furthermore, training data 300 may further include augmented views of images and / or image tiles. For example, image 202-1 may correspond to an image tile with one or more dilations applied.

[0116] To develop a model that is slide prep machine agnostic, the model must be trained to correctly identify outputs from two tiles of the same tissue, but prepared using different slide prep machines (e.g., scanned by different scanners). One technical solution to this technical problem is to modify the loss function associated with the model framework (e.g., Framework 400 or Framework 500) so that embeddings that stray too far from other embeddings corresponding to the same tissue sample are penalized.

[0117] One technique that can be used to perform penalization is a tile-level technique. A tissue sample scanned with a scanner can be considered a form of image augmentation that can be used to create a set of augmented views at the beginning of the framework. Similarly, a tissue sample prepared with a set of attributes using a single slide preparer can be considered a form of image augmentation. In some embodiments, a set of crops (e.g., views) V can include two tiles corresponding to tiles from images created using different slide preparers (e.g., different scanners). By way of example, referring to FIG. 6A , a process 600 includes a biological sample 602 (e.g., a tissue sample) that is provided to a first slide preparer 610 and a second slide preparer 620. As described above, the first slide preparer 610 can include a first slide stainer (e.g., a stainer) and a first slide scanner (e.g., a scanner), and the second slide preparer 620 can include a second slide stainer and a second slide scanner. The attributes of each slide preparer can be different. For example, the scanner associated with the first slide preparer 610 may be different from the scanner associated with the second slide preparer 620. As another example, the stainer associated with the first slide preparer 610 may be different from the stainer associated with the second slide preparer 620. In some embodiments, the biological sample 602 may be prepared on a glass slide and stained with hematoxylin and eosin (H&E) slide stain. The slide preparers 610 and 620 (or their associated scanners) may be configured to output a first image 612 and a second image 622, respectively, depicting the biological sample 602.

[0118] In some embodiments, image 612 may be divided (e.g., via training data generation subsystem 110) into multiple image tiles 614a-d and 624a-d, respectively. While only four image tiles are shown in FIG. 6A , one skilled in the art will recognize that other amounts of image tiles may be generated for a given image, and that the image tiles may overlap. In some embodiments, corresponding image tiles from different images captured using different slide preparers, but representing a biological specimen, may be identified and included in image set 630. For example, image tile 614d from image 612 and image tile 624d from image 622 may each be included in image set 630. In some embodiments, each image tile in image set 630 may include an identifier indicating the slide preparer used to capture and / or prepare the image, as well as an identifier indicating the biological specimen represented by the corresponding image in that tile.

[0119] In some embodiments, some or all of the image tiles included in image set 630 may be augmented using one or more image augmentations. For example, image tiles 614d and / or 624d may be rotated, flipped, blurred, their colors jittered, etc. The augmented tiles may also include several global and local crops of the image. However, because the two tiles being analyzed should contain the same portion of the biological sample (e.g., tiles 614d and 624d should depict the same portion of biological sample 602), implementing this technique may be impractical. Because registering the similarity of tissue samples depicted in different image tiles may be impractical for tracking a large number of images or image tiles, slide-level techniques may be used instead.

[0120] Instead of penalizing the distance between tile-level embeddings (e.g., differences in similarity measured as distance in latent space) in slide-level approaches, some embodiments penalize the slide-level distance. For example, a model for tile-level techniques is configured to generate embeddings based on image tiles (e.g., tiles 614a-d of image 612). The model parameters can then be adjusted to maximize the similarity of the embeddings for tiles that represent a common portion of the biological specimen (e.g., tiles 614d and 624d). Slide-level techniques may maximize the similarity between the embeddings of each image generated by each scanner. For example, the similarity between the embedding generated based on image 612 and the embedding generated based on image 622, each of which represents biological specimen 602, may be maximized. In some embodiments, the similarity that is maximized is the average tile embedding of the two corresponding full-slide images. For example, an embedding is calculated for each of image tiles 614a-d, and the embeddings are averaged together to obtain a first representation of image 612. Similar processing may be performed on image tiles 624a-624d to obtain a second representation of image 622.

[0121] In some embodiments, a loss may be calculated based on the first and second representations. In some embodiments, a batch of tiles containing image tiles may be created (and a new fragmentation of tiles within the mini-batch may be performed at each training epoch to calculate the loss). Instead of randomly creating batches of tiles, batches of tiles may be formed that correspond to the same biological sample but from different source images. For example, a batch of image tiles representing a portion of biological sample 602 but captured by a first scanner (associated with first slide preparer 610) and a second scanner (associated with second slide preparer 620) may be created. For example, the batch of image tiles may include image tile 614d and image tile 624d. In some embodiments, when there are multiple image tiles to choose from across two or more images, each corresponding to a tile from a different image, the image tiles included in the batch may be selected randomly. By way of example, image tiles 614a-614d and image tiles 624a-624d may be created, corresponding to images 612 and 622, respectively, captured and / or prepared by slide preparers 610 and 620 of biological specimen 602. Thus, to create a batch, training data generation subsystem 110 may randomly select one or more pairs of image tiles, such as tiles 614a and 624a, 614b and 624b, 614c and 624c, and 614d and 624d. Each batch may be generated by randomly selecting one or more pairs of image tiles, such as tiles 614a and 624a, 614b and 624b, 614c and 624c, and 614d and 624d. 1 ,x 2n}, and each element x i is at least one corresponding image x j (e.g., captured by the ith and jth scanners). For a given batch, we can minimize a loss function. An example loss function is Equation 12:

number

[0122] In Equation 12, g s is the student encoder, θ srepresents the parameter of the student encoder, and λ represents the scale of the summation term. DINO is given by Equation 4.

[0123] FIG. 6B illustrates different slide preparers and associated settings for generating images representing the same biological sample, according to various embodiments. In system 650, a biological sample can be provided to one or more of slide preparers 120. For example, a biological sample 652 can be prepared on a slide and provided to a first slide preparer 120-1, a second slide preparer 120-2, and an Mth slide preparer 120-M. Each of slide preparers 120-1, 120-2, and 120-M can include a slide stainer and a slide scanner. For example, slide preparer 120-1 can include stainer 122-1 and scanner 124-1, slide preparer 120-2 can include stainer 122-2 and scanner 124-2, and slide preparer 120-M can include stainer 122-M and scanner 124-M.

[0124] Each stainer 122-1, 122-2, 122-M may have a different setting 660-1, 660-2, 660-M, respectively. Additionally, each scanner 124-1, 124-2, 124-M may have a different setting 664-1, 664-2, 664-M, respectively. The settings 660-1, 660-2, 660-M may indicate a different staining agent, tissue thickness, staining level, or other characteristic of each stainer when preparing a slide of a biological specimen (e.g., biological specimen 652) that may then be imaged. The settings 664-1, 664-2, 664-M may indicate a different scanner type, model, manufacturer, software, software version, light source, lens, lens material, lens thickness, magnification level, or other characteristic of each scanner when capturing an image of the slide of the biological specimen. In some embodiments, slide preparers 120-1, 120-2, and 120-M may capture one or more images, such as images 654-1, 654-2, and 654-M, respectively. For example, a slide prepared by stainer 122-1 based on setting 660-1 may be provided to scanner 124-1, which may capture image 654-1 based on setting 664-1. Images 654-1, 654-2, and 654-M each depict biological sample 652, but may contain variations that could affect downstream classification tasks if not compensated for by making the classification model agnostic to the slide preparer and the corresponding slide preparation and processing techniques used thereby.

[0125] As examples, Tables 1 and 2 below show datasets that can be used to train machine learning models. As can be seen from Table 1, different datasets may use different stainers to stain the biological samples, have different thicknesses of tissue samples, and have different staining levels, as well as other differences. [Table 1] [Table 2]

[0126] FIG. 7 is a flowchart illustrating an exemplary process 700 for training a model agnostic to sample processing using an average similarity training technique, according to various embodiments. Process 700 may begin at operation 710, in which image data may be received. In some embodiments, the image data may include a first image set 712 and a second image set 716. The first image set 712 may include slides 714 representing images (e.g., whole slide images) captured and prepared using a first slide preparer (e.g., slide preparer 120-1). For example, slides 714 may include slides prepared using a first stainer and / or captured using a first scanner. Each image in first image set 712 may represent an image of one of multiple biological samples, such as a tissue sample. For example, if ten biological samples are imaged, slides 714 may include ten slides, each depicting one of the ten biological samples. The second image set 716 may include slides 718 representing images (e.g., whole slide images) captured using a second slide preparer (e.g., slide preparer 120-M). For example, the slides 718 may include slides prepared using a second stainer and / or captured using a second scanner. Each image in the second image set 716 may represent an image of one of multiple biological samples, such as tissue samples. Continuing with the previous example, if ten biological samples are imaged, the slides 718 may include ten slides, each depicting one of the ten biological samples. In some embodiments, each biological sample may have an image taken and / or prepared using each available slide preparer. For example, for one biological sample, the first image set 712 may include one image depicting the biological sample, and the second image set 716 may include one image depicting the biological sample. The similarities and differences between these two images may be utilized to train models that are agnostic to scanner-related and / or stainer-related embedded drivers to improve downstream biological sample classification.In some embodiments, operation 710 may be performed by a subsystem that is the same as or similar to training data generation subsystem 110.

[0127] In operation 720, a first extended viewset and a second extended viewset may be generated. The first extended viewset and the second extended viewset may, in some embodiments, correspond to the first image set 712 and the second image set 716, respectively. In some embodiments, the first and second extended viewsets may include images contained by the first and second image sets 712 and 716 (e.g., slides 714 and 718, respectively) with one or more image enhancements performed on them. For example, image enhancements that may be performed include rotation, flipping, blurring, color jittering, cropping, etc. In some embodiments, slides 714 and 718 may be divided into image tiles. The image tiles may overlap. In some embodiments, the image tiles may have enhancements performed on them, although alternatively (or additionally), enhancements may be performed before dividing the slide into image tiles. In some embodiments, each extended viewset may include images depicting a respective biological sample. For example, the first augmented viewset may include an image of the first biological sample (including image tiles of that image), and the second augmented viewset may include an image of the first biological sample (including image tiles of that image). In some embodiments, operation 720 may be performed by a subsystem that is the same as or similar to training data generation subsystem 110.

[0128] In operation 730, a biological sample may be selected. The selected biological sample may be one of the biological samples whose images were captured by the first scanner and the second scanner. For example, biological sample 1 may be selected. Based on the selection, one or more augmented views from the first augmented view set corresponding to biological sample 1 and one or more augmented views from the second augmented view set corresponding to biological sample 1 may be selected. In some embodiments, operation 730 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0129] At operation 740, a first encoder may be trained. The first encoder may be implemented by a convolutional neural network such as ResNet, a visual transformer, or other machine learning model. The first encoder may be part of the DINO framework 400, the BYOL framework 500, or other frameworks. In some embodiments, operation 740 may be performed by a subsystem that is the same as or similar to the model training subsystem 112.

[0130] Operation 740 may include one or more substeps. For example, operation 740 may include substep 742. In substep 742, a first encoder may be used to generate first representations of augmented views of the first augmented view set. In some embodiments, the first representation may be an embedding. An embedding is a mapping from variables to vectors (arrays of numbers). An embedding is a vector representation of features that describe an image. An embedding is meaningful if it has features that are suitable for downstream classification or regression tasks. As an example, an embedding may be a 2048-dimensional vector, while a corresponding medical image (e.g., an entire slide image depicting a biological specimen) contains data corresponding to a large number of pixels (e.g., tens of thousands of pixels, hundreds of thousands of pixels, millions of pixels, etc.). In some embodiments, the augmented views of the first augmented view set may include augmented views of image tiles of the corresponding image. For example, a first entire slide image from the first image set 712 may be divided into image tiles, and then the image tiles may have augmentation applied to them. Each of the image tiles for the first entire slide image may be provided to a first encoder, and an embedding representing each image tile may be generated. In some embodiments, the embeddings representing the image tiles may be combined to generate an average embedded representation for the corresponding image. As another example, a first entire slide image from the first image set 712 may have one or more augmentations applied to it and then be separated into image tiles. Each of the image tiles may be provided to a first encoder to generate an embedding, which may then be combined to generate an average embedded representation for the corresponding image. In embodiments, if multiple images of the same biological sample are captured by a single scanner, the same process may be repeated, and an overall average embedded representation may be generated based on the corresponding average embedded representation for each image.

[0131] In sub-step 744, a second encoder may be used to generate second representations of the augmented views of the second augmented view set. In some embodiments, the second representations may be embeddings. In some embodiments, the augmented views of the second augmented view set may include augmented views of image tiles of the corresponding images. For example, a second full slide image from the second image set 716 may be divided into image tiles, and then an augmentation may be applied to the image tiles. Each of the image tiles for the second full slide image may be provided to a second encoder, and an embedding representing each image tile may be generated. In some embodiments, the embeddings representing the image tiles may be combined to generate an average embedded representation of the corresponding image. As another example, a second full slide image from the second image set 712 may have one or more augmentations applied to it and then be divided into image tiles. Each of the image tiles may be provided to a second encoder to generate an embedding, and then combined to generate an average embedded representation of the corresponding image. In an embodiment, if multiple images of the same biological sample are captured by a single scanner, the same process can be repeated and an overall average embedding representation can be generated based on the corresponding average embedding representation of each image.

[0132] In substep 746, a similarity between the first representation and the second representation may be calculated. In some embodiments, the similarity may be calculated based on a loss function. For example, depending on the framework used for training, the loss function used may be a DINO loss function or a BYOL loss function. For example, the loss function for the DINO framework is expressed as Equation 12, while the loss function for the BYOL framework is expressed as Equation 10 ( TIFF2026501098000026.tif7170, in formula TIFF2026501098000027.tif7170 refers to the symmetrized loss function obtained by feeding the first view 504 to the first encoder 510 and the second view 506 to the second encoder 530).

[0133] Sub-step 748 may maximize the similarity between the first and second representations. Maximizing the similarity may include updating hyperparameters of the first and second encoders. In some embodiments, the updating may be performed using backpropagation. In some embodiments, an optimizer (e.g., see Equation 11) may be used to optimize the loss function. For example, an Adam optimizer may be used.

[0134] In some embodiments, after operation 740 is performed on a corresponding image of one biological sample (e.g., an expanded view of an image tile), parameters of the second encoder may be updated based on updates to parameters of the first encoder. For example, the weights and biases and other parameters of the second encoder may be updated using an exponential moving average (EMA) of the weights and biases of the first encoder.

[0135] In operation 750, a determination may be made as to whether to analyze any additional biological samples. For example, if there are N biological samples, the first and second extended view sets may each include N whole slide images (e.g., extended versions of each of whole slide image 714 and whole slide image 718). Thus, for a selected biological sample, extended views of the image of the selected biological sample from each of the first and second extended view sets (e.g., tiles obtained by dividing the extended views) may be used to perform one round of training of the first encoder. In some embodiments, the determination may be made after parameters of the first encoder (and possibly the second encoder) have been updated. If, at operation 750, it is determined that there are additional biological samples to be analyzed, process 700 may return to operation 730, where another biological sample may be selected, and augmented views from each of the first and second augmented view sets representing the biological sample may be used to train the first encoder (e.g., by repeating substeps 742-748 using representations of the first and second augmented views). However, if, at operation 750, it is determined that there are no additional biological samples to be analyzed, process 700 may proceed to operation 760. At operation 760, process 700 may end, and the trained model may be stored (e.g., in model database 146). In some embodiments, operations 750 and 760 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0136] In some embodiments, if it is determined in operation 750 that no additional samples need to be analyzed, the first encoder may be tested / validated using test data (also interchangeably referred to herein as validation data). For example, the first encoder may be tested using the validation data to determine how accurate the trained model is for the classification task of new data. In some embodiments, the accuracy of the trained encoder for validation may be compared to a threshold accuracy level. If the accuracy is determined to be equal to or greater than the threshold accuracy level, the model may be stored in the model database 146. For example, the trained model may be used for further downstream biological sample classification tasks.

[0137] Adversarial Approach

[0138] In some embodiments, the DINO and / or BYOL frameworks may be modified to facilitate adversarial learning. Traditionally, the DINO and BYOL frameworks are designed for unlabeled data. However, in digital pathology tasks where the dataset includes images stained using multiple stainers or captured using multiple scanners, underlying labels may still be present. For example, metadata indicating the scanner label can be included in each image (and / or image tile) indicating the scanner used to capture that image. The scanner label can be used to ensure that the embeddings encoded by the corresponding encoders in the DINO / BYOL frameworks preserve only morphological features and do not preserve or minimize the influence of scanner origin.

[0139] The DINO / BYOL framework can be modified to form an adversarial learning process in which two networks compete against each other. These networks include a "generator" and a "discriminator." The goal of adversarial learning is for the generator to produce results that can fool the discriminator, and the discriminator is designed to identify the output from the generator.

[0140] In the examples described herein, a student encoder of the DINO framework (e.g., student encoder 410) and an online encoder of the BYOL framework (e.g., online encoder 510) may be used as generators for the adversarial DINO framework and the adversarial BYOL framework, respectively. In some embodiments, the encoder of the DINO / BYOL framework may be trained asynchronously against a discriminator added for adversarial learning. For example, parameters of the discriminator may be updated, followed by updating parameters of the encoder of the DINO / BYOL framework.

[0141] Adversarial DINO Approach

[0142] 8 illustrates an exemplary adversarial framework 800 used to train a model to be scanner-type agnostic when performing digital pathology classification, according to various embodiments. In some embodiments, adversarial framework 800 may be built on the DINO framework depicted by framework 400 of FIG.

[0143] For example, the adversarial framework 800 may include two paths: one including a student encoder 410 and a softmax layer 412 (e.g., a “student network”) and one including a teacher encoder 420, a centering layer 424, and a softmax layer 422 (e.g., a “teacher network”). An image, such as image 402, may undergo one or more image augmentations to obtain augmented image 404a and image 404b. In some embodiments, augmented images 404a, 404b may represent image tiles of image 402 on which image augmentation has been performed. As mentioned above, with respect to FIG. 5 , framework 400 is a self-supervised learning approach based on knowledge distillation, which propagates information from a small annotation set to a large unlabeled database by having a network mimic the output of another network. This knowledge distillation process can also be extended to use cases where images do not have labels.

[0144] In some embodiments, the adversarial framework 800 adapts the framework 400 so that the student encoder 410 serves as the generator 820 of the adversarial learning process. The generator 820 may be configured to encode the augmented image 404a (which may be an image tile) to obtain an embedding 830. In some embodiments, the embedding 830 is the same as or similar to an embedding (for the same image) generated by the student encoder 410 using the framework 400. The adversarial framework 800 may feed the embedding 830 to a discriminator 860, which may be configured to generate a prediction 840. The prediction 840 includes a prediction by the discriminator 860 of which slide preparer was used to generate the image 402. For example, the prediction 840 may include a prediction of whether the image 402 was captured via a first scanner or a second scanner. As another example, prediction 840 may include a prediction of whether image 402 was prepared using a first stain or a second stain, a first magnification level or a second magnification level, a first tissue thickness or a second tissue thickness, or other attributes of a slide prep machine used to prepare and capture image 402. In some embodiments, discriminator 860 may be constructed as a linear layer with an input size equal to the dimensionality of the output embedding (e.g., embedding 830) from generator 820. For example, if generator 820—student encoder 410—is implemented using a ResNet-18 architecture, the dimensionality of embedding 830 will be 512, from which discriminator 860 may have an input size of 512.

[0145] In some embodiments, the model training subsystem 112 may be configured to train the discriminator 860 to determine the slide preparation type of the embedding 830 output by the generator 820. Furthermore, the model training subsystem 112 may be configured to train the generator 820 to generate meaningful representations (e.g., embeddings) of the biological sample that can deceive the discriminator 860. The generator 820 utilizes the framework 400 (e.g., the DINO framework), where the presence of an opponent (e.g., the discriminator 860) can improve the created embedding (e.g., the embedding 830) so that it has more value. However, in some embodiments, the image 402 may be fed to the student encoder 410 in addition to the first augmented view 404a. Furthermore, as described in more detail below, in some embodiments, the adversarial framework 800 may include gradient stopping (denoted by the label "sg") on the teacher network.

[0146] In contrast to the average embedding similarity approach, which simply changes the loss function, the adversarial framework 800 requires a new approach to calculating the loss. The reason behind this is the need to train an additional network, the discriminator 860. Furthermore, the training of the discriminator 860 and the student encoder 410 (as well as the teacher encoder 420) is not performed simultaneously. Therefore, at each step (each mini-batch) of the training process, two different loss functions may be created and two backpropagations may be performed. Accordingly, some embodiments include a model training subsystem 112 that performs two training steps, as described in more detail below and with respect to FIGS. 9 and 10.

[0147] In the first step of training, the discriminator 860 may be updated. In some embodiments, the model training subsystem 112 updates the parameters θ of the discriminator 860. DThe model training subsystem 112 may be configured to freeze the student encoder 410 (as well as the teacher encoder 420) so that only the gradients 404a, 404b, 404c, 404d, 404e, 404f, 404f, 404g, 404h, 404i, 404j, 404j, 404j, 404j, 404j, 404j, 404j, 404c, 404d, 404i, 404j, 404j, 404j, 404j, 404c, 404j ...

[0148] In some embodiments, the biological sample depicted by the image may include a slide preparation label (e.g., slide preparation label 204-1). For example, as detailed above, image 402, which may correspond to an image tile to which one or more image augmentations have been applied, may include a scanner label (e.g., scanner label 252) that indicates whether a first or second scanner was used to capture the image depicting the biological sample. Mathematically, for a given sample (e.g., image 402) x, which may include a scanner label y (or another label), discriminator 860 computes an n-dimensional vector D(g s830。 In some embodiments, the model training subsystem 112 may be configured to output a loss function to train the discriminator 860. The loss function to be minimized may be the cross-entropy loss defined in Equation 13:

number

[0149] In a second step of the training process, a generator (e.g., generator 820) may be updated. In some embodiments, the model training subsystem 112 may be configured to perform a second training step 1000, as seen in FIG. 10 . In particular, in the second training step 1000, the student encoder 410 (as well as the teacher encoder 420) may be updated. During the second training step 1000, the model training subsystem 112 may freeze the discriminator 860, as indicated by the dashed line in the discriminator 860, so that only the student encoder 410 and the teacher encoder 420 are updated. In some cases, the student encoder 410 may be updated, followed by an update of the teacher encoder 420. For example, the parameters of the teacher encoder 420 may be updated with an exponential moving average of the parameters of the student encoder 410. The generator 820 (e.g., student encoder 410) can be trained to generate a meaningful representation of an input image (e.g., an input image tile) while being able to fool the discriminator 860 into incorrectly predicting the scanner used to capture the image.

[0150] In some embodiments, the model training subsystem 112 may be configured to combine a traditional DINO loss function with the loss function previously used in the discriminator 860. The combined loss function is expressed in Equation 14:

number

[0151] The composite loss function described in Equation 14 may be used by the model training subsystem 112 to minimize the DINO loss (e.g., the loss function of framework 400) while also maximizing the loss of the discriminator 860. In Equation 14, μ represents a parameter used to scale the additional loss (e.g., the discriminator loss) compared to the DINO loss.

[0152] Adversarial BYOL Approach

[0153] 11 illustrates another exemplary adversarial framework 1100 used to train a model to be agnostic to attributes related to slide preparers when performing digital pathology classification, according to various embodiments. In some embodiments, adversarial framework 1100 may be built on the BYOL framework depicted by framework 500 of FIG.

[0154] For example, the adversarial framework 1100 may include two paths: a path (e.g., an “online network”) that includes an online encoder 510, an online (first) projector 514, and an online predictor 518, and a path (e.g., a “target network”) that includes a target encoder 530 and a target (second) projector 534. An image, such as image 502, may undergo one or more image augmentations to obtain augmented views 504 and 506. In some embodiments, the augmented views 504, 506 may represent image tiles of image 502 on which image augmentation has been performed.

[0155] In some embodiments, the adversarial framework 1100 adapts the framework 500 so that the online encoder 510 serves as the generator 1120 of the adversarial learning process. The generator 1120 may be configured to encode the augmented views 504 (which may be image tiles) to obtain embeddings 512. The adversarial framework 1100 may feed the embeddings 512 to a discriminator 1150, which may be configured to generate predictions 1154. The predictions 1154 may include a prediction by the discriminator 1150 of which slide preparer was used to generate the image 502. For example, the predictions 1154 may include a prediction of whether the image 502 was captured via a first scanner or a second scanner. As another example, the predictions 1154 may include a prediction of whether the image 502 was created via a first stainer or a second stainer (or via a first stainer or a second stainer, which may be applied by the same slide stainer or a different slide stainer). In some embodiments, the discriminator 1150 may be constructed as a linear layer with an input size equal to the dimensionality of the output embedding (e.g., embedding 512) from the generator 1120. For example, if the generator 1120—student encoder 410—is implemented using a visual transformer architecture, the dimensionality of the embedding 512 will be 512-dimensional, from which the discriminator 1150 may have an input size of 512-dimensional.

[0156] In some embodiments, the model training subsystem 112 may be configured to train the discriminator 1150 to determine the slide preparation type of the embeddings 512 output by the generator 1120. Additionally, the model training subsystem 112 may be configured to train the generator 1120 to generate meaningful representations (e.g., embeddings) of the biological sample that can fool the discriminator 1150. The generator 1120 utilizes the framework 500 (e.g., the BYOL framework), and the presence of an opponent (e.g., the discriminator 1150) can improve the created embeddings (e.g., embeddings 512) so that the created embeddings have more value.

[0157] Similar to the adversarial DINO approach, the adversarial BYOL approach also requires a loss to be calculated. Furthermore, training of the discriminator 1150 and the online encoder 510 (as well as the target encoder 530) may not be performed simultaneously. Thus, two different loss functions may be created at each step (each mini-batch) of the training process, and two backpropagations may be performed. Accordingly, some embodiments include a model training subsystem 112 that performs two training steps, as described in more detail below and with respect to Figures 12A and 12B.

[0158] In the first step of training, the discriminator 1150 may be updated. In some embodiments, the model training subsystem 112 updates the parameters θ of the discriminator 1150. DThe model training subsystem 112 may be configured to freeze the online encoder 510 (as well as the target encoder 530) so that only the embedding 512 is updated. The model training subsystem 112 may freeze the online encoder 510 and the target encoder 530, as indicated by the dashed lines, while allowing backward gradient propagation to be performed for the discriminator 1150, as seen by the first training step 1200 in FIG. 12A. To freeze the online encoder 510 and the target encoder 530, the model training subsystem 112 may be configured to apply gradient stopping "sg." In some embodiments, the training of the discriminator 1150 is a fully supervised classification task. The discriminator 1150 is trained using the model training subsystem 112 to predict the type of scanner used to capture the image 502 based on the embedding 512. The image 502 may be, for example, dilated and tiled to obtain the augmented image 504, which may then be fed to the online encoder 510 to be encoded as the embedding 512.

[0159] In some embodiments, the biological sample depicted by the image may include a slide preparation label (e.g., slide preparation label 204-1). For example, as detailed above, image 502, which may correspond to an image tile to which one or more image augmentations have been applied, may include a scanner label (e.g., scanner label 252) indicating whether a first or second scanner was used to capture the image depicting the biological sample. Mathematically, for a given sample (e.g., image 502) v, which may include a label 1152, the discriminator 1150 generates an n-dimensional vector D(g s(x)). In this example, n refers to the number of scanners (i.e., classes) used to capture the images of the biological sample. For example, if there are only two scanners and n=2, then the output of the discriminator 1150 may be an indicator of whether the discriminator 1150 predicted, based on the embeddings 512, whether the input image v was captured using the first scanner or the second scanner. In some embodiments, the model training subsystem 112 may be configured to minimize a loss function to train the discriminator 1150. The loss function to be minimized may be the cross-entropy loss defined in Equation 13.

[0160] In a second step of the training process, a generator (e.g., generator 1120) may be updated. In some embodiments, the model training subsystem 112 may be configured to perform a second training step 1250, as seen in FIG. 12B. In particular, in the second training step 1250, the online encoder 510 (as well as the target encoder 530) may be updated. During the second training step 1250, the model training subsystem 112 may freeze the discriminator 1150, as indicated by the dashed line in the discriminator 1150, so that only the online encoder 510 and the target encoder 530 are updated. In some cases, the online encoder 510 may be updated, followed by the target encoder 530. For example, the parameters of the target encoder 530 may be updated with an exponential moving average of the parameters of the online encoder 510. The generator 1120 (e.g., the online encoder 510) can be trained to generate a meaningful representation of an input image (e.g., an input image tile) while being able to fool the discriminator 1150 into incorrectly predicting the slide preparer used to prepare and capture that image.

[0161] In some embodiments, the model training subsystem 112 may be configured to combine a traditional BYOL loss function with the loss function previously used for the discriminator 1150. The combined loss function may be described in Equation 14, except that the DINO loss term is replaced with a BYOL loss term. In some embodiments, the BYOL framework (e.g., framework 500) may be updated using the model training subsystem 112 by backpropagation of the combined loss of the BYOL network and the discriminator.

[0162] 13 is a flowchart illustrating an exemplary process 1300 for training a model agnostic to sample processing using adversarial training techniques, according to various embodiments. Process 1300 may begin at operation 1310. In operation 1310, training data 1302 may be received. In some embodiments, training data 1302 may include a first image set 1304 and a second image set 1306. First image set 1304 may include images (e.g., whole slide images) captured using a first slide preparer (e.g., slide preparer 120-1). For example, images included in first image set 1304 may include images captured using a first scanner and / or images prepared using a first stainer. Each image in first image set 1304 may represent one image of a plurality of biological samples, such as tissue samples. For example, if 10 biological samples are imaged, image sets 1304 and 1306 may each include 10 slides, each depicting one of the 10 biological samples. The second image set 1306 may include images (e.g., whole slide images) captured using a second slide preparer (e.g., slide preparer 120-M). For example, the images included in the first image set 1306 may include images captured using a second scanner and / or images prepared using a second stainer. Each image in the second image set 1306 may represent one image of multiple biological samples, such as tissue samples.

[0163] In some embodiments, each biological sample may have its image captured using each available scanner. For example, for one biological sample, the first image set 1304 may include one image depicting the biological sample, and the second image set 1306 may include one image depicting the biological sample. Similarities and differences between these two images may be used to train a model that is agnostic to scanner-related and / or stainer-related embedded drivers to improve downstream biological sample classification. In some embodiments, operation 1310 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0164] In some embodiments, the images are augmented views of some or all of the images included in the first image set 1304 and the second image set 1306. Each image in the first image set 1304 may include a first slide preparation label indicating that a first slide preparer was used to prepare and capture the image, and each image in the second image set 1306 may include a second slide preparation label indicating that a second slide preparer was used to prepare and capture the image. In some embodiments, the images included in each of the image sets 1304 and 1306 may correspond to augmented views of the respective image. For example, the training data 1302 may include a first augmented view set and a second augmented view set. In some embodiments, the first augmented view set and the second augmented view set may correspond to the first image set 1304 and the second image set 1306, respectively. In some embodiments, the first and second augmented view sets may include images included by the first and second image sets 1304 and 1306 with one or more image augmentations performed on them. For example, image augmentations that may be performed include rotation, flipping, blurring, color jittering, cropping, etc. In some embodiments, the images in the first image set 1304 and the second image set 1306 may be divided into image tiles. The image tiles may overlap. In some embodiments, the image tiles may have augmentations performed on them, although alternatively (or additionally), augmentations may be performed before dividing the slide into image tiles. In some embodiments, each augmented viewset may include images depicting each biological sample. For example, the first augmented viewset may include an image of the first biological sample (including its image tiles), and the second augmented viewset may include an image of the first biological sample (including its image tiles). In some embodiments, the training data 1302 may include image tiles (including applied image augmentations) each having an associated label indicating one or more attributes related to the slide preparation machine used to prepare and / or capture the corresponding image of that image tile.For example, each image may be labeled with an indication of the scanner that was used to capture the corresponding image.

[0165] In some embodiments, a biological sample may be selected. The selected biological sample may be one of the biological samples prepared and / or captured via the first slide preparer or the second slide preparer. In an example, the selected biological sample may be one of the biological samples captured by the first scanner and the second scanner. For example, biological sample 1 may be selected. Based on the selection, one or more extended views (e.g., whole slide image, image tile) from the first extended view set corresponding to biological sample 1 and one or more extended views from the second extended view set corresponding to biological sample 1 may be selected.

[0166] In operation 1312, an image representation may be generated using a first encoder. Some embodiments include generating an image representation for a subset of images from the first image set 1304 and / or the second image set 1306. The first encoder may refer to, for example, the online encoder 510. The image for which the image representation is generated may correspond to an entire slide image or an image tile, which may also include applied image enhancements. In some embodiments, the image may depict a selected biological sample prepared and / or captured using a first slide preparer, the first slide preparer being associated with a first label. Alternatively or additionally, the image may depict a selected biological sample prepared and / or captured using a second slide preparer, the second slide preparer being associated with a second label. In some embodiments, the image representation may correspond to an embedding generated to represent an image depicting the selected biological sample (e.g., an image of the biological sample including the first label). By way of example, the embedding 512 may be generated by the online encoder 510. In some embodiments, the embedding may be a 2048-dimensional vector. In some embodiments, operation 1312 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0167] In operation 1314, the image representation may be provided to a discriminator configured to predict the slide preparer or its attributes used to prepare and / or capture the image of the selected biological sample. By way of example, the discriminator 1150 may receive the embedding 512 and attempt to determine whether the image 502 (represented by the embedding 512) was captured by a first scanner or a second scanner. In some embodiments, the discriminator 1150 may be constructed as a linear layer with an input size equal to the dimensionality of the output embedding (e.g., the embedding 512) from the generator 1120. For example, if the generator 1120—the online encoder 510—is implemented using a ResNet-18 architecture, the dimensionality of the embedding 512 may be 512, from which the discriminator 1150 may have an input size of 512. In some embodiments, operation 1314 may be performed by a subsystem identical to or similar to the model training subsystem 112.

[0168] In operation 1316, a discriminator loss may be calculated. The discriminator loss may be calculated based on the prediction 1154 obtained from the discriminator 1150. As described above, the prediction 1154 is a prediction by the discriminator 1150 as to which of n possible slide preparers and / or attributes associated with those slide preparers were used to prepare and / or capture the image of the biological sample from which the input embeddings were derived. For example, the prediction 1154 may indicate whether the image 502 was created using a first or second slide stainer, captured using a first or second slide scanner, etc. In some embodiments, operation 1316 may be performed by a subsystem that is the same as or similar to the model training subsystem 112.

[0169] In operation 1318, the discriminator may be updated based on the loss calculated in the previous step. Updating the discriminator may include updating parameters of the discriminator based on the calculated discriminator loss. For example, the parameter θ D may be updated based on the calculated loss. In some embodiments, the parameters may be updated using backpropagation. As mentioned above, the parameters of the online encoder 510 and the target encoder 530 are updated based on the parameters of the discriminator 1150. θ The parameter of the discriminator may remain frozen while D is updated. Updating the parameters of the discriminator refers to the operation of training the discriminator. In other words, the discriminator is trained by updating the parameters of the discriminator based on the discriminator loss calculated in operation 1316. In some embodiments, operation 1318 may be performed by a subsystem that is the same as or similar to the model training subsystem 112.

[0170] At operation 1320, an updated image representation of the image may be generated using the first encoder. For example, the same image passed to the encoder during operation 1312 may be passed to the first encoder again. The image may be passed to the encoder again because the BYOL / DINO framework was decoupled, i.e., not updated, during a previous step (e.g., updating the discriminator). Thus, now that the discriminator has been updated, the image can be encoded by the first encoder to train the previously frozen components of the BYOL / DINO framework. In some embodiments, operation 1320 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0171] At operation 1322, the updated image representation may be provided to an updated discriminator. For example, an image representation of the image input to the first encoder after the discriminator was updated (at operation 1318) may be obtained and passed to the updated discriminator. At operation 1324, an updated loss for the updated discriminator may be calculated. Operation 1324 may be substantially similar to operation 1316, except that at operation 1324, parameters of the discriminator may be updated and the updated image representation may be used. In some embodiments, operations 1322 and 1324 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0172] In operation 1326, a primal loss may be calculated. The primal loss function used may be the DINO loss function or the BYOL loss function, depending on the framework used for training. For example, the loss function for the DINO framework is expressed as Equation 12, while the loss function for the BYOL framework is expressed as Equation 10 ( TIFF2026501098000030.tif7170, in formula (TIFF2026501098000031.tif7170 refers to the symmetrized loss function obtained by feeding the first view 504 to the first encoder 510 and the second view 506 to the second encoder 530.) In some embodiments, operation 1326 may be performed by a subsystem that is the same as or similar to the model training subsystem 112.

[0173] At operation 1328, an adversarial loss may be determined based on the updated discriminator loss and the primal loss. In some embodiments, the adversarial loss may be calculated with a weight applied to the updated discriminator loss. To calculate the adversarial loss, the updated discriminator loss may be weighted and subtracted from the primal loss. For example, as seen in Equation 14, a weight μ may be applied to the discriminator loss term, which may be subtracted from the adversarial loss term. In some embodiments, operation 1328 may be performed by a subsystem that is the same as or similar to the model training subsystem 112.

[0174] In operation 1330, the first encoder may be updated based on the determined adversarial loss. For example, the parameters θ of the student encoder 410 S may be updated (e.g., using backpropagation). As another example, in an adversarial framework 1100 built on the BYOL framework, the BYOL loss is calculated and the parameters θ of the online encoder 510 are オンライン may be updated (e.g., using backpropagation). In some embodiments, operation 1330 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0175] At operation 1332, the second encoder may be updated based on the updates made to the first encoder. In some embodiments, the parameters of the second encoder may be updated based on the updated parameters of the first encoder. For example, the parameters θ of the teacher encoder 420 may be updated based on the updated parameters of the first encoder. T and the parameters θ of the target encoder 530 ターゲット are the parameters θ of the student encoder 410, respectively. S and the parameters θ of the online encoder 510 オンライン For example, the parameter θ of the teacher encoder 420 can be updated based on T To update the parameter θ of the student encoder 410, S(where the parameters θ of the student encoder 410 S may use (which has already been updated based on the discriminator loss). In some embodiments, operation 1332 may be performed by a subsystem that is the same as or similar to model training subsystem 112.

[0176] In some embodiments, process 1300 may be repeated for additional sample images. For example, additional images of biological samples included in training data 1302 may be retrieved after the second encoder is updated in operation 1332. In some embodiments, a decision may be made as to whether to analyze any additional biological samples. For example, if there are N biological samples, the first and second extended view sets may each include N whole slide images (e.g., extended versions / tiles of each of the whole slide images from the first image set 1304 and / or extended versions / tiles of the whole slide images from the second image set 1306). Thus, for a selected biological sample (whose images were used in operation 1312 and then again in operation 1320), extended views (e.g., tiles obtained by dividing the extended views) of the image of the selected biological sample from each of the first and second extended view sets may be used to perform one round of training of the first encoder. In some embodiments, the decision may be made after the parameters of the first encoder (and possibly the second encoder) have been updated. If it is determined that there are additional biological samples to analyze, the process 1300 may return to operation 1312 where another biological sample may be selected and the process 1300 may be repeated.

[0177] In some embodiments, if it is determined that no additional samples need to be analyzed, the first encoder may be tested / validated using test data (also interchangeably referred to herein as validation data). For example, the first encoder may be tested using the validation data to determine how accurate the trained model is for the classification task of new data. In some embodiments, the accuracy of the trained encoder for validation may be compared to a threshold accuracy level. If the accuracy is determined to be equal to or greater than the threshold accuracy level, the model may be stored in the model database 146. For example, the trained model may be used for further downstream biological sample classification tasks. In some embodiments, the validation step may be performed on a second encoder instead of or in addition to the first encoder.

[0178] The aforementioned training techniques (e.g., average similarity embedding approach, adversarial approach) can be used to "pre-train" the encoder for downstream classification tasks. After pre-training, each framework can be tested. In some cases, the only hyperparameter that is varied is the scale term of the global loss function (λ and μ for the average similarity embedding approach and adversarial approach, respectively). For example, a larger value of the scale term may increase the model's ability to be agnostic to the origin of the scanner. In some embodiments, other hyperparameters may remain the same. In some cases, the optimizer used is the Adam optimizer, as detailed, for example, in "Adam: A method for stochastic optimization" by Kingma et al., the entire disclosure of which is incorporated herein by reference. For example, while the learning rate in stochastic gradient descent is fixed, the Adam optimizer calculates an exponential moving average of the gradient and the squared gradient, and the decay rate of the moving average is controlled by two parameters.

[0179] Downstream tasks

[0180] In some embodiments, downstream tasks may be configured to evaluate the quality of pre-trained encoders. Pre-trained encoders may alternatively or additionally be used for many independent tasks, such as survival prediction, cell detection and segmentation, mitosis detection, and other tasks. The downstream task may be a fully supervised classification task that detects whether a biological sample exhibits a predefined characteristic associated with a condition based on an image of the biological sample prepared and captured by a given slide preparation machine. For example, the downstream task may be determining whether an image of a tissue sample contains a depiction of a tumor. In some embodiments, the downstream task is computed by taking the trained encoders from the average similarity embedding approach and the trained encoders from the adversarial approach and concatenating them into a fully concatenated classifier. For example, referring to FIG. 14A , the downstream classification subsystem 114 may be configured to perform downstream classification of images depicting biological samples to evaluate the quality of each pre-trained encoder for the average similarity embedding approach, the DINO framework, and the BYOL framework, as well as the adversarial approaches, the adversarial DINO framework, and the adversarial BYOL framework. For example, trained encoder 1410 may reference framework 400 (e.g., the DINO framework), framework 500 (e.g., the BYOL framework), framework 800 (e.g., the adversarial DINO framework), or framework 1100 (e.g., the adversarial BYOL framework).

[0181] The trained encoder 1410 may be provided with an image 1402 depicting a biological sample. For example, the image 1402 may depict a tissue sample. The tissue sample may be a tissue sample containing a tumor or other abnormality, or the tissue sample may represent "normal" tissue (e.g., no abnormality). The trained encoder 1410 may be configured to generate an embedding 1412 that represents the image 1402 in a latent space. For example, the embedding 1412 may be an n-dimensional vector. In some embodiments, the embedding 1412 may be input to a fully connected classifier 1420 configured to output a result 1422. The result 1422 may indicate a determination from the classifier 1420 as to whether the biological sample depicted by the image 1402 represents a particular abnormality or exhibits another predefined characteristic. In some embodiments, the result 1422 may be a binary flag indicating whether the tissue sample depicted by the image 1402 is a tumor.

[0182] FIG. 14B is a flowchart illustrating an exemplary process 1450 for training a classifier, according to various embodiments. In some embodiments, process 1450 may begin at operation 1452. Operation 1452 may select a model to use as a pre-trained encoder that will serve as the backbone of a downstream classifier. The model may be selected from model database 146. For example, a pre-trained encoder having one of frameworks such as frameworks 400, 500, 800, or 1100 may be selected. The selected framework may be coupled with a fully connected classifier for a downstream classification task, such as determining whether a whole slide image of a tissue sample exhibits abnormal tissue, such as a tumor. In some embodiments, operation 1452 may be performed by a subsystem that is the same as or similar to downstream classification subsystem 114.

[0183] In operation 1454, training data for training the classifier may be retrieved. For example, the training data for training the classifier may be obtained from training data database 144. In some embodiments, the training data may include images (e.g., image tiles) depicting biological samples prepared and / or captured using one slide preparer. This is to determine / confirm the agnosia of the model for different slide preparers, as described in more detail below. The images used to train the classifier may differ from the images used to train the encoder. For example, each image used to train the classifier may not include a corresponding image to an image used to train the encoder. In some embodiments, operation 1454 may be performed by subsystems the same as or similar to training data generation subsystem 110, model training subsystem 112, and / or downstream classification subsystem 114.

[0184] In operation, a classifier may be trained using the retrieved training data and a pre-trained encoder. For example, classifier 1420 may be trained using the retrieved training data and pre-trained encoder 1410. In some embodiments, training a classifier may include one or more substeps, such as substeps 1470-1474. In substep 1470, the training data may be provided to a pre-trained encoder to generate embeddings representing each image included in the training data. The training data may include, for example, images / image tiles depicting biological samples captured using a scanner and / or stainer. In some embodiments, the training data may include images captured using only one scanner, only one stainer, or only one slide preparation device (which may include a scanner and stainer). In substep 1472, the embeddings generated by the pre-trained encoder may be provided to a classifier. The classifier may be configured to generate a classification result. The classification result may include a flag indicating whether a given biological sample depicted by a given image / image tile depicts an abnormal biological sample or a normal biological sample. For example, the flag may indicate whether the image tile depicts a tissue sample containing a tumor or normal tissue. The classification result may be a vector or array indicating the likelihood that the image represents one of a predefined set of classes. For example, if there are two classes (e.g., a normal tissue class or a tumor tissue class), each value in the vector or array may represent the probability that a given image represents one of the two different classes. In substep 1474, the classifier may be updated based on the classification result. For example, the weights or biases of the classifier may be updated based on the classification result. In some embodiments, each of substeps 1470-1474 may be repeated for each image included in the training data.

[0185] In operation 1458, the accuracy of the trained classifier may be calculated. In some embodiments, the accuracy may be calculated based on validation data. The validation data may include images / tiles of images depicting biological samples with known classifications. The accuracy may be calculated by determining the number or percentage of images correctly classified by the classifier. In some embodiments, operation 1458 may be calculated by a subsystem that is the same as or similar to the model training subsystem 112 and / or the downstream classification subsystem 114.

[0186] At operation 1460, the accuracy of the classifier may be compared to a predefined threshold accuracy score to determine whether the accuracy of the classifier is greater than the threshold accuracy score. For example, the threshold accuracy score may be 75% or greater, 85% or greater, 90% or greater, 95% or greater, or some other value. If the accuracy score is determined to be less than (or equal to) the threshold accuracy score, process 1450 may return to operation 1456, where the classifier may be retrained or further trained. In some embodiments, process 1450 may return to operation 1454, where new or updated training data is selected and / or obtained and the classifier is further trained based on the new or updated training data. Some embodiments may include resetting the parameters of the classifier and / or the pre-trained encoder or selecting a different pre-trained encoder if the accuracy score is lower than the predefined threshold accuracy score. For example, if the accuracy score falls below the threshold accuracy score N or more times, the parameters of the classifier may be reset. In some embodiments, operation 1460 may be performed by a subsystem that is the same as or similar to the model training subsystem 112 and / or the downstream classification subsystem 114.

[0187] At operation 1462, the trained classifier may be stored. For example, the trained classifier may be stored in model database 146. In some embodiments, operation 1460 may be performed by a subsystem that is the same as or similar to model training subsystem 112 and / or downstream classification subsystem 114.

[0188] In some embodiments, the downstream classification subsystem 114 may be designed to perform two functions: (1) verify the baseline quality of the trained encoder and (2) test the trained encoder's ability to be agnostic to the origin of the slide preparer. To that end, the model training subsystem 112 may be configured to train the classifier 1420 on data from one slide preparer (e.g., one scanner) while testing the classifier on data from multiple (or all) slide preparers (e.g., multiple scanners). This differs from pre-training the encoder, which uses data from each slide preparer for both training and validation. For example, referring to FIG. 15A , during encoder pre-training, the training dataset 1500 may include a first training set 1502 (shown as a dashed line) containing images of biological samples captured using a first scanner and a second training set 1504 (shown as a solid line) containing images of biological samples captured using a second scanner. As another example, the first training set 1502 may include images of biological samples captured using a first slide preparer, and the second training set 1504 may include images of biological samples captured using a second slide preparer.

[0189] In some embodiments, each image included in the first training set 1502 may include the same slide preparation label (e.g., the scanner label corresponding to scanner 124-1), and each image included in the second training set 1504 may include the same slide preparation label (e.g., the scanner label corresponding to scanner 124-2). The tiled and / or augmented versions of the images may be fed to an encoder for training. For example, images 402 and 502 may be images included in the first training set 1502 or the second training set 1504, respectively. Similar to the training dataset 1500, the validation data 1510 may include a first test set 1512 including images depicting biological samples prepared and captured by a first slide preparer, and a second test set 1514 including images depicting biological samples prepared and captured by a second slide preparer. In some embodiments, each image included in the first test set 1512 may include the same slide preparation label (e.g., the scanner label corresponding to scanner 124-1), and each image included in the second test set 1514 may include the same slide preparation label (e.g., the scanner label corresponding to scanner 124-2). In some embodiments, images included in the validation data 1510 may have their labels masked. To test the accuracy of the trained encoder, tiled and / or augmented versions of the images may be provided to the trained encoder. For example, images 402 and 502 may be images included in the first test set 1512 or the second test set 1514, respectively.

[0190] 15B, downstream training of the classifier 1420 may use different data for training and validation. For example, the training data used to train the classifier may include a training data set 1522, which may include images depicting biological samples captured using a first scanner. Alternatively, the training data set 1522 may include images depicting biological samples but using a second scanner.

[0191] In some embodiments, a downstream classifier (e.g., classifier 1420) may be tested using a first test set 1524 that includes images prepared and captured by a first slide preparer, and may be tested using a second test set 1526 that includes images prepared and captured by a second slide preparer. If more slide preparers are used to prepare and / or capture images of the biological sample, test sets that include images rendered by those slide preparers may also be used during testing of the trained classifier.

[0192] Details of the data used to create the training and validation data used to pre-train the encoder and train the downstream classifier are provided below.

[0193] Encoder pre-training data

[0194] In the pre-training stage, which is self-supervised training, the images stored in image database 142 may originate from multiple slide preparers. For example, images captured by two scanners may be stored in image database 142, as shown in Table 3. [Table 3]

[0195] In some embodiments, each image may depict a biological sample, such as a tissue sample. Some or all of the biological samples may represent tumor samples from various clinical trials (e.g., various cancer research datasets). In some embodiments, the scanners of each slide preparation device may be set to the same magnification setting. For example, slides captured by Scanner 1 and Scanner 2 in Table 3 may be captured at 20x magnification. In some embodiments, tiles may be extracted for each slide. For example, training data generation subsystem 110 may be configured to generate training data and validation data used to train and validate framework 400, 500, 800, or 1100. The tiles may overlap. The tiles may then be split into training data and validation data and stored in training data database 144 and validation data database 148.

[0196] Downstream training

[0197] In some embodiments, some or all of the images stored in the image database 142 may include labels indicating whether the biological sample depicted by a given image is "normal" or "abnormal." For example, an image may include a label indicating whether the tissue sample depicted by the image is a normal tissue sample or a tumor sample, and for downstream tasks, the training data generation subsystem 110 and / or the downstream classification subsystem 114 may be configured to select, from the available images, images that include a "normal" label and images that include an "abnormal" label. Selecting images that include these labels may significantly reduce the number of samples used to train, validate, and test a scanner-agnostic model.

[0198] Table 4 below contains examples of the sizes of the training, validation, and test data after filtering out images that do not have a "normal" or "abnormal" label. [Table 4]

[0199] The training data generation subsystem 110 may be configured to divide the set of tiles into subsets grouped based on slide preparer (e.g., scanner 1 or scanner 2) and role in the training process (e.g., training, validation, test). Thus, for a two-scanner system (e.g., slide preparer 120 includes slide preparer 120-1 and slide preparer 120-2), six sample subsets are created. Furthermore, the training data generation subsystem 110 may be configured to divide various tiles among different roles in the training process such that corresponding slides belong to the same corresponding set. Thus, tiles from the test subset are not included or do not have a counterpart used in the training portion of the training process.

[0200] FIG. 16 is a flowchart illustrating an exemplary process 1600 for analyzing an image of a biological sample, according to various embodiments. In some embodiments, process 1600 may begin with operation 1602. In operation 1602, an image depicting a biological sample may be received. The biological sample may be prepared on a slide using a slide preparer, and the image may be captured using a slide scanner (the slide stainer and slide scanner may be part of the same slide preparer). In some embodiments, the type, settings, or other characteristics of the slide preparer may be unknown. For example, the scanner may be one of a set of scanners trained to ensure that the model is agnostic during classification of the biological sample. In some embodiments, the received image may be one of a series of images captured by the scanner depicting a particular biological sample. In some embodiments, operation 1602 may be performed by a subsystem that is the same as or similar to the downstream classification subsystem 114.

[0201] At operation 1604, the image may be fed to a trained classifier. The trained classifier may be trained using a backbone framework such as framework 800 (e.g., the adversarial DINO framework) or framework 1100 (e.g., the adversarial BYOL framework). For example, classifier 1420 may be fed the image received at operation 1602. In some embodiments, operation 1604 may be performed by the same or similar subsystem as downstream classification subsystem 114.

[0202] In operation 1606, one or more classifications of the image may be determined by a trained classifier. For example, the trained classifier may generate embeddings representing the image using a trained encoder (e.g., encoder 1410), and the embeddings generated by the trained encoder may be fed to the trained classifier to obtain a classification result. The classification result may include one or more classifications of the biological sample. For example, the classification result may include a classification that the biological sample represents an abnormal biological sample (e.g., tumor). In some embodiments, operation 1606 may be performed by a subsystem that is the same as or similar to the downstream classification subsystem 114.

[0203] FIG. 17 illustrates an exemplary cross-scanner benchmark dataset. Two columns of image tiles in dataset 1700 may represent images of biological samples captured via either of two different scanners. For example, the top row may represent image tiles of biological samples captured using a first scanner, and the bottom row may represent image tiles of biological samples captured using a second scanner. Image pairs 1702, 1704, and 1706 may represent various pairs of image tiles. For example, image pair 1702 may include two image tiles representing the same slide (e.g., the same slide of a biological sample) captured using two different scanners. Image pair 1706 may include two image tiles representing the same slide captured using the same scanner. Image pair 1704 may include two image tiles representing the same tile identified by two different scanners. Image pair 1704 may be a registered pair. A registered pair may include images that are supposed to be identical in content, differing only in color appearance (e.g., if the images originate from different scanners). Image pair 1702 may not be a registered pair because the content of the two images may not be the same. When embeddings are generated for each of the image pairs, the distance in latent space between the embeddings of a given pair may be different. For example, the average L2 distance of the embeddings generated for the image tiles of image pair 1702 may be greater than the average L2 distance of the embeddings generated for the image tiles of image pair 1704. However, the average L2 distance of the embedding values ​​generated for the image tiles of image pair 1704 may be less than the average L2 distance of the embedding values ​​generated for the image tiles of image pair 1706.

[0204] In some embodiments, the cross-scanner ratio may be calculated based on the average distance between a pair of registered images of the same tile from the same scanner (e.g., image pair 1704) and the average distance between a pair of unregistered images of the same slide but from different scanners (e.g., image pair 1702). For example, the cross-scanner ratio may be: [Number 14] Cross-scanner ratio = (average distance between registered pairs) / (average distance between unregistered pairs, same slide, different scanners) Equation 15

[0205] The same-scanner ratio may be calculated based on the average distance between registered pairs of images of the same tile by the same scanner (e.g., image pair 1704) and the average distance between unregistered pairs of the same slide by the same scanner. The lower these ratios, the more agnostic the trained model is to different slide preparation techniques (e.g., scanners). For example, a same-scanner ratio may be: [Number 15] Same scanner ratio = (average distance between registered pairs) / (average distance between unregistered pairs, same slide, same scanner) Equation 16.

[0206] Example Results

[0207] Results are presented below with reference to Tables 5 and 6, including a comparison of different approaches using the DINO framework (e.g., framework 400 and framework 800). Additionally, some results are presented for a ResNet-18 model pre-trained on the ImageNet dataset. In some embodiments, one or more hyperparameters are tuned. For example, weight decay may be adjusted to avoid overfitting, and the learning rate may be adjusted. In some cases, the Adam optimizer is used. [Table 5] [Table 6]

[0208] FIG. 18B shows various plots 1810-1840 of representations created by an encoder for biological samples included in validation data for an encoder trained using the average similarity approach, according to various embodiments. Each of plots 1810-1840 may be depicted as a TSNE plot, which may be used to highlight whether an encoder trained using the average embedding similarity approach is agnostic to the origin of the slide preparation machine. Plots 1810-1840 show features learned by various encoders at different scales. The scale may refer, for example, to λ from Equation 12. Plots 1810-1840 show learned features for different versions of the average similarity approach for different scales. For example, plot 1810 includes λ=1.0, plot 1820 includes λ=10.0, plot 1830 includes λ=100.0, and plot 1840 includes λ=1000.0. Some embodiments include data points for each of plots 1810-1840, each corresponding to an embedding representing an image tile, with each color corresponding to a particular slide preparer type. For example, yellow (or light) data points may represent embeddings for image tiles created and captured using a first slide preparer, while purple (or dark) data points may represent embeddings for image tiles captured using a second slide preparer.

[0209] FIG. 19 shows plots 1900 and 1950 of representations created by an encoder for biological samples included in validation data for an encoder trained using an adversarial approach, according to various embodiments. Each of plots 1900-1950 may be depicted as a TSNE plot, which can be used to highlight whether an encoder trained using an adversarial approach is agnostic to the origin of the slide preparer. Plots 1900-1950 show features learned by various encoders. For example, plots 1900-1950 show learned features for different versions of the adversarial approach for different scales, where scale refers to the weight applied to the discriminator loss (e.g., μ in Equation 14). For example, plot 1900 includes μ = 1.0, and plot 1950 includes μ = 10.0. Some embodiments include data points for each of plots 1900-1950, each corresponding to an embedding representing an image tile, with each color corresponding to a specific slide preparer type. For example, yellow (or light-colored) data points may represent embeddings of image tiles captured using a first slide preparer, and purple (or dark-colored) data points may represent embeddings of image tiles captured using a second slide preparer.

[0210] Figures 18B and 19 show that larger scales result in models that are more "slide-prep-agnostic" (or "slide-processing-agnostic"). For example, at larger scales, the TSNE plots are observed to be more mixed. This is because scale sets the importance of the slide-prep-agnostic penalty compared to traditional losses (e.g., BYOL loss, DINO loss). In particular, for adversarial frameworks, the TSNE plots show cluster fragmentation at higher scale values ​​(e.g., plot 1950), leading to the onset of significant cluster merging. For example, with the average embedding similarity approach, the mixing is less apparent.

[0211] FIG. 20A shows a plot 2000 of the standard deviation of embeddings over training for the average embedding similarity approach, according to various embodiments. Self-supervised techniques pose challenges in ensuring that models do not collapse. A collapsing model can refer to a model that produces a consistent output regardless of the input. The DINO framework (e.g., framework 400) and the BYOL framework (e.g., framework 500) are designed to avoid model collapse. For example, the embeddings generated by the teacher encoder (in the DINO framework) go through a centering layer and a sharpening step, which helps avoid model collapse. However, penalizing modifications to the DINO framework to ensure a model that is agnostic to the type of slide preparation machine increases the risk of model collapse. Model collapse can be detected based on the distribution of learned features. For example, the computing system 102 can be configured to monitor the standard deviation of the embeddings generated during training.

[0212] FIG. 20B shows a plot 2050 of the standard deviation of embeddings across training for the average embedding similarity approach and the adversarial approach, according to various embodiments. Different traces in the plot 2050 may correspond to different training approaches used. For example, one trace shows the standard deviation of embeddings generated by the average similarity approach at a scale of λ=1.0. Another trace shows the standard deviation of embeddings generated by the adversarial approach at different scales, such as μ=1.0 and μ=10.0. Another trace shows the standard deviation of embeddings generated by the DINO framework.

[0213] In some embodiments, the training process may be evaluated using several metrics. In the example dataset described above with respect to Tables 3 and 4, because the data is imbalanced, various frameworks may be ranked based on AUC-PR scores. Tables 7 and 8 below show the results of various metrics for the test data set. For example, the downstream classification subsystem 114 may be configured to evaluate the trained model using test data such as that included in Table 7. In Table 7, the test data used to validate the trained model and the training data used to train the model may include images captured using the same scanner. In Table 8, the test data used to validate the trained model and the training data used to train the model may include images captured using different scanners. [Table 7] [Table 8]

[0214] Figure 20A shows that the encoder trained with the largest scale factor appears to collapse. While this can happen in some extreme cases, other versions do not appear to collapse. For example, as seen in Figure 20B, the standard deviation of the embeddings appears to be lower than those derived from DINO and adversarial DINO. This indicates that when penalties are applied, the features can lose useful information. However, the standard deviation of the adversarial approach does not appear to indicate that collapse occurs.

[0215] As Tables 7 and 8 show, encoders trained using the average embedding similarity approach do not perform better overall than basic DINO on downstream classification tasks. Modifications to the framework do not appear to improve performance but instead appear to overly penalize training. On the other hand, the results seem to indicate that modifications to the framework for adversarial approaches (e.g., Framework 800, the adversarial DINO framework) not only result in an improved ability to be agnostic to scanner type, but also, alternatively, appear to produce improved, more meaningful embeddings. Furthermore, encoders trained using adversarial approaches appear to perform to a similar extent as ResNet-18 models trained on ImageNet, improving over the results of the DINO framework 400. Furthermore, adversarial approaches, particularly the adversarial DINO framework (e.g., Framework 800), demonstrate that added features from the framework improve the model's ability to be agnostic to scanner type.

[0216] FIG. 21 illustrates an image 2100 of a biological sample, according to various embodiments. The data used to develop training data used to train the classifier (and encoder) may include 516K images and / or image tiles of a tissue sample depicting a tumor and 42K images and / or image tiles of a normal tissue sample captured by a first slide preparer (e.g., Scanner 1), and 483K images and / or image tiles of a tissue sample depicting a tumor and 18K images and / or image tiles of a normal tissue sample captured by a second slide preparer (e.g., Scanner 2). Training and testing of the classifier may use a process similar to that described above. As seen in image 2100, a downstream classifier may determine, based on image 2100, that one or more regions of the tissue sample represent a tumor. In some embodiments, detection of a tumor or other abnormality in the biological sample may cause the computing system 102 to output a classification result to the client device 130. For example, the classification result may include an image of the tissue sample, the region in which the tissue abnormality was detected, and an indication that the image depicts an abnormality.

[0217] 23A-26B show various example embedding plots according to various embodiments. Figures 23A-23B show TSNE plots 2300 and 2350 of embeddings obtained from a validation set analyzed using adversarial DINO techniques. In TSNE plots 2300 and 2350, each color represents one unique tissue processing condition, also referred to interchangeably herein as a slide preparer or a set of slide preparer attributes (e.g., stain, magnification level, scanner, etc.). Additionally, embeddings may be generated at the tile level. TSNE plot 2300 shows the distribution of various embeddings before adversarial training is performed. In other words, without filtering sample processing-related characteristics, TSNE plot 2300 depicts how various slide preparer attributes may affect the prepared embeddings. However, TSNE plot 2350 may show the distribution of embeddings after adversarial training is performed.

[0218] 24A-24B show TSNE plots 2400 and 2450 of embeddings obtained from different datasets of a particular clinical trial analyzed using adversarial DINO techniques. In TSNE plots 2400 and 2450, each color represents a unique dataset. Furthermore, embeddings may be generated at the tile level. TSNE plot 2400 shows the distribution of embedding values ​​before adversarial training is performed. In other words, without filtering sample processing-related characteristics, TSNE plot 2400 depicts how the dataset may affect the prepared embeddings. However, TSNE plot 2450 may show the distribution of embeddings after adversarial training is performed.

[0219] 25A-25B show TSNE plots 2500 and 2550 of embeddings obtained from the validation set using the adversarial BYOL technique. In TSNE plots 2500 and 2550, each color represents one unique tissue processing condition, also referred to interchangeably herein as a slide preparer or a set of attributes of a slide preparer (e.g., stain, magnification level, scanner, etc.). Additionally, embeddings may be generated at the tile level. TSNE plot 2500 shows the distribution of embedding values ​​before adversarial training is performed. In other words, without filtering sample processing-related characteristics, TSNE plot 2500 depicts how the dataset may affect the prepared embeddings. However, TSNE plot 2550 may show the distribution of embeddings after adversarial training is performed.

[0220] Figures 26A-26B show TSNE plots 2600 and 2650 of embeddings obtained from different datasets of a particular clinical trial analyzed using the adversarial BYOL technique. In TSNE plots 2600 and 2650, each color represents a unique dataset. Furthermore, embeddings may be generated at the tile level. TSNE plot 2600 shows the distribution of embedding values ​​before adversarial training is performed. In other words, without filtering sample processing-related characteristics, TSNE plot 2600 depicts how the dataset may affect the prepared embeddings. However, TSNE plot 2650 may show the distribution of embeddings after adversarial training is performed.

[0221] 27 illustrates an exemplary computer system 2700. In particular embodiments, one or more computer systems 2700 perform one or more steps of one or more methods described or illustrated herein. In particular embodiments, one or more computer systems 2700 provide functionality described or illustrated herein. In particular embodiments, software running on one or more computer systems 2700 performs one or more steps of one or more methods described or illustrated herein, or provides functionality described or illustrated herein. Particular embodiments include one or more portions of one or more computer systems 2700. As used herein, references to a computer system may encompass computing devices, and vice versa, where appropriate. Furthermore, references to a computer system may encompass one or more computer systems, where appropriate.

[0222] This disclosure contemplates any suitable number of computing systems 2700. This disclosure contemplates computing system 2700 taking any suitable physical form. By way of example and not limitation, computing system 2700 may be an embedded computing system, a system-on-chip (SOC), a single-board computing system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computing system, a laptop or notebook computing system, an interactive kiosk, a mainframe, a mesh of computing systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computing system, or a combination of two or more of these. Where appropriate, computing system 2700 may comprise one or more computing systems 2700, may be unitary or distributed, may span multiple locations, span multiple machines, span multiple data centers, or reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computing systems 2700 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example, and not limitation, one or more computing systems 2700 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. One or more computer systems 2700 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations, where appropriate.

[0223] In a particular embodiment, computer system 2700 includes a processor 2702, memory 2704, storage 2706, an input / output (I / O) interface 2708, a communication interface 2710, and a bus 2712. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0224] In particular embodiments, processor 2702 includes hardware for executing instructions, such as those comprising a computer program. By way of example and not limitation, to execute instructions, processor 2702 may retrieve (or fetch) instructions from an internal register, an internal cache, memory 2704, or storage 2706, decode and execute them, and then write one or more results to an internal register, an internal cache, memory 2704, or storage 2706. In particular embodiments, processor 2702 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates processor 2702 including any suitable number of any suitable internal caches, where appropriate. By way of example and not limitation, processor 2702 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 2704 or storage 2706, and the instruction cache may speed up retrieval of those instructions by processor 2702. Data in the data cache may be a copy of data in memory 2704 or storage 2706 for the operation of an instruction executed in processor 2702, the results of a previous instruction executed in processor 2702 for access by a subsequent instruction executed in processor 2702 or for writing to memory 2704 or storage 2706, or other suitable data. The data cache may speed up read or write operations by processor 2702. The TLB may speed up virtual address translation for processor 2702. In particular embodiments, processor 2702 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates processor 2702 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, processor 2702 may include one or more arithmetic logic units (ALUs), may be a multi-core processor, or may include one or more processors 2702.Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0225] In particular embodiments, memory 2704 includes main memory for storing instructions to be executed by processor 2702 or data for operation of processor 2702. By way of example and not limitation, computer system 2700 may load instructions into memory 2704 from storage 2706 or another source (e.g., another computer system 2700, etc.). Processor 2702 may then load the instructions from memory 2704 into an internal register or internal cache. To execute instructions, processor 2702 may retrieve instructions from the internal register or internal cache and decode those instructions. During or after execution of an instruction, processor 2702 may write one or more results (which may be intermediate or final results) to an internal register or internal cache. Processor 2702 may then write one or more of those results to memory 2704. In particular embodiments, processor 2702 executes only instructions in one or more internal registers or internal caches or in memory 2704 (as opposed to storage 2706 or elsewhere) and operates only on data in one or more internal registers or internal caches or in memory 2704 (as opposed to storage 2706 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may connect processor 2702 to memory 2704. Bus 2712 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) reside between processor 2702 and memory 2704 to facilitate accesses to memory 2704 requested by processor 2702. In particular embodiments, memory 2704 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, where appropriate, this RAM may be single-ported or multi-ported RAM. This disclosure contemplates any suitable RAM. Memory 2704 may include one or more memories, where appropriate.Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0226] In particular embodiments, storage 2706 includes mass storage for data or instructions. By way of example and not limitation, storage 2706 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 2706 may include removable or non-removable (or fixed) media, where appropriate. Storage 2706 may be internal or external to computing system 2700, where appropriate. In particular embodiments, storage 2706 is non-volatile solid-state memory. In particular embodiments, storage 2706 includes read-only memory (ROM). Where appropriate, this ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage 2706 taking any suitable physical form. Storage 2706 may include, where appropriate, one or more storage control units that facilitate communications between processor 2702 and storage 2706. Where appropriate, storage 2706 may include one or more storages 2706. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0227] In particular embodiments, I / O interface 2708 includes hardware, software, or both and provides one or more interfaces for communication between computer system 2700 and one or more I / O devices. Computer system 2700 may include one or more of these I / O devices, where appropriate. One or more of these I / O devices may enable communication between a person and computer system 2700. By way of example and not limitation, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, other suitable I / O device, or a combination of two or more thereof. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 2708 therefor. Where appropriate, I / O interface 2708 may include one or more device or software drivers that enable processor 2702 to drive one or more of these I / O devices. I / O interface 2708 may include, where appropriate, one or more I / O interfaces 2708. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0228] In particular embodiments, communication interface 2710 includes hardware, software, or both that provide one or more interfaces for communications (e.g., packet-based communications, etc.) between computer system 2700 and one or more other computer systems 2700 or one or more networks. By way of example and not limitation, communication interface 2710 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wired-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 2710 for that network. By way of example and not limitation, computing system 2700 may communicate with an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more portions of the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. By way of example, computer system 2700 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN or an ultra-wideband WPAN), a Wi-Fi network, a Wi-MAX network, a cellular network (e.g., a Global System for Mobile Communications (GSM) network), or any other suitable wireless network, or a combination of two or more of these. Computer system 2700 may include any suitable communication interface 2710 for any of these networks, where appropriate. Communication interface 2710 may include one or more communication interfaces 2710, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0229] In particular embodiments, bus 2712 includes hardware, software, or both that connects components of computer system 2700 to one another. By way of example, and not limitation, bus 2712 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more thereof. Bus 2712 may include one or more buses 2712, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0230] As used herein, one or more computer-readable non-transitory storage media may comprise, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application-specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid-state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Computer-readable non-transitory storage media may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0231] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated otherwise by context. Moreover, "and" is both jointly and severally, unless expressly indicated otherwise or indicated otherwise by context. Thus, as used herein, "A and / or B" means "A and / or B, jointly or severally," unless expressly indicated otherwise or indicated otherwise by context.

[0232] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person skilled in the art. Moreover, any reference herein to a device or system, or a component of a device or system, being adapted to, arranged to, capable of, configured to, enabled to, operable to, or operable to perform a particular function encompasses that device, system, component, or particular function thereof, so long as it is so adapted, arranged, capable, configured, enabled, operable, or operable. Furthermore, while the present disclosure describes or illustrates particular embodiments as providing certain advantages, a particular embodiment may provide none, some, or all of these advantages.

[0233] Illustrative Embodiments Embodiments disclosed herein may include the following. 1. A computer-implemented method comprising: receiving image data including a first image set and a second image set, the first image set and the second image set comprising digitized images of a plurality of digital pathology slides processed using a first slide preparer and a second slide preparer, respectively; the first slide preparer and the second slide preparer each having a set of attributes, the value of at least one of the attributes being different between the first slide preparer and the second slide preparer; and applying a threshold applied to each image in the first image set and the second image set based on the image data. 1. A method comprising: generating a first augmented view set and a second augmented view set based on the one or more augmentations; and for each digital pathology slide, training a first visual transformer to generate, using the first visual transformer, a first representation of an augmented view of the first augmented view set; and increasing a similarity between the first representation and a second representation of an augmented view of the second augmented view set, wherein the second representation is generated via the second visual transformer, and both the first representation and the second representation correspond to the same digital pathology slide. 2. The method of embodiment 1, wherein the first slide preparer and the second slide preparer are both slide scanning machines. 3. The method of embodiment 1, wherein the first slide preparer and the second slide preparer are both slide stainers. 4. The method of embodiment 3, wherein the first slide preparator uses a first staining technique and the second slide preparator uses a second staining technique that is different from the first staining technique. 5. The method of embodiment 1, wherein the first slide preparer and the second slide preparer are the same machine, and the value of at least one of the attributes changes over time. 6. The method of any one of embodiments 1 to 5, wherein the digital pathology slides contain whole slide images of multiple tissue types. 7. The method of any one of embodiments 1 to 6, wherein the digital pathology slide is an image of tissue stained with hematoxylin and eosin. 8. The method of any one of embodiments 1 to 7, wherein the first visual transformer and the second visual transformer have the same architecture. 9. The method of any one of embodiments 1 to 8, wherein the one or more enhancements include at least one of blurring the image, flipping the image, rotating the image, distorting one or more colors of the image, or cropping the image. 10. A method according to any one of embodiments 1 to 9, further comprising dividing each image of the first extended view set into a first plurality of tiles, the first representation being generated based on the first plurality of tiles, and the second representation being generated based on the second plurality of tiles. 11. The method of embodiment 10, further comprising generating a first plurality of embeddings, each corresponding to one of the first plurality of tiles, wherein the first representation is generated based on the first plurality of embeddings. 12. The method of embodiment 11, further comprising: calculating a first average embedding based on the first plurality of embeddings, wherein the first representation includes the calculated first average embedding. 13. A method according to any one of embodiments 11 to 12, further comprising generating a second plurality of embeddings, each of which corresponds to one of the second plurality of tiles, wherein the second representation is generated based on the second plurality of embeddings. 14. The method of embodiment 13, further comprising: calculating a second average embedding based on the second plurality of embeddings, wherein the second representation includes the calculated second average embedding. 15. A method according to any one of embodiments 10 to 14, wherein the first extended view set and the second extended view set include tiles randomly selected relative to the first plurality of tiles and the second plurality of tiles. 16. A method according to any one of embodiments 1 to 15, wherein maximizing the similarity between the first representation and the second representation comprises minimizing a loss function. 17. The method of embodiment 16, wherein the loss function is:

number

number

number

Claims

1. 1. A computer-implemented method comprising: receiving image data including a first image set and a second image set, the first image set and the second image set comprising digitized images of a plurality of digital pathology slides processed using a first slide preparer and a second slide preparer, respectively, the first slide preparer and the second slide preparer each having a set of attributes, the value of at least one of the attributes differing between the first slide preparer and the second slide preparer; generating a first augmented set of views and a second augmented set of views based on the image data and one or more augmentations applied to each image of the first set of images and the second set of images; and For each said digital pathology slide: training a first visual transducer, generating a first representation of an augmented view of the first augmented view set using the first visual transformer; and Increasing a similarity between the first representation and a second representation of an augmented view of the second set of augmented views, the second representation being generated via a second visual transducer, and the first representation and the second representation both corresponding to the same digital pathology slide. training a first visual transducer to perform A method comprising:

2. 10. The method of claim 1, wherein the first slide preparer and the second slide preparer are both slide scanning machines.

3. The method of claim 1 , wherein the first slide preparer and the second slide preparer are both slide stainers.

4. 4. The method of claim 3, wherein the first slide preparer uses a first staining technique and the second slide preparer uses a second staining technique that is different from the first staining technique.

5. 2. The method of claim 1, wherein the first slide preparer and the second slide preparer are the same machine, and the value of at least one of the attributes changes over time.

6. The method of any one of claims 1 to 5, wherein the digital pathology slides contain whole slide images of multiple tissue types.

7. The method of any one of claims 1 to 6, wherein the digital pathology slide is an image of tissue stained with hematoxylin and eosin.

8. The method according to any one of claims 1 to 7, wherein the first and second visual transformers have the same architecture.

9. The method of any one of claims 1 to 8, wherein the one or more enhancements comprise at least one of blurring the image, flipping the image, rotating the image, distorting one or more colors of the image, or cropping the image.

10. Dividing each image of the first set of extended views into a first plurality of tiles; and Dividing each image of the second set of extended views into a second plurality of tiles. further comprising The method of any one of claims 1 to 9, wherein the first representation is generated based on the first plurality of tiles and the second representation is generated based on the second plurality of tiles.

11. 11. The method of claim 10, further comprising: generating a first plurality of embeddings, each corresponding to one of the first plurality of tiles, wherein the first representation is generated based on the first plurality of embeddings.

12. 12. The method of claim 11, further comprising: calculating a first average embedding based on the first plurality of embeddings, wherein the first representation includes the calculated first average embedding.

13. 13. The method of claim 11 or 12, further comprising generating a second plurality of embeddings, each corresponding to one of the second plurality of tiles, wherein the second representation is generated based on the second plurality of embeddings.

14. 14. The method of claim 13, further comprising: calculating a second average embedding based on the second plurality of embeddings, wherein the second representation includes the calculated second average embedding.

15. 15. The method of claim 10, wherein the first and second extended viewsets include tiles randomly selected relative to the first and second plurality of tiles.

16. The method of any preceding claim, wherein maximizing the similarity between the first and second representations comprises minimizing a loss function.

17. The method of any one of claims 1 to 16, further comprising training a classifier based on said first visual transducer to perform image classification of slides of biological specimens.

18. receiving an image depicting a biological sample to be classified into at least one of a plurality of tissue categories; 20. The method of claim 17, further comprising: feeding the images to the trained classifier to determine one or more tissue categories into which the biological sample is classified.

19. The method of any one of claims 1 to 18, wherein the first visual transformer has a first set of hyperparameters and the second visual transformer has a second set of hyperparameters.

20. 20. A method comprising: utilizing a trained classifier to classify digital pathology images, wherein the trained classifier has been trained using a method according to any one of claims 1 to 19.

21. 1. A computer-implemented method comprising: receiving training data including images of a plurality of biological samples processed using a first slide preparer or a second slide preparer, the first slide preparer and the second slide preparer each having a set of attributes, and a value of at least one of the attributes differing between the first slide preparer and the second slide preparer; For each of said biological samples: generating, using a first encoder, a first representation of one of the images of the biological sample processed using the first slide preparer; providing the first representation to a discriminator to generate a prediction of whether the biological sample corresponding to the one of the images was processed using the first slide preparer or the second slide preparer; updating one or more parameters of the discriminator based on a first loss calculated based on the generated prediction and metadata associated with the one of the images, the metadata indicating that the biological sample was processed using the first slide preparer; updating one or more parameters of the first encoder based on the first loss; training the updated first encoder, generating an updated first representation of the one of the images; and Increasing the similarity between the updated first representation and a second representation of another one of the images of the same biological sample generated using a second encoder. training the updated first encoder to A method comprising:

22. A non-transitory computer readable medium having stored thereon computer program instructions which, when executed, implement the method of any one of claims 1 to 20 or 21.

23. 1. A system comprising: a memory for storing computer program instructions; and one or more processors configured to execute said computer program instructions to perform the method of any one of claims 1 to 20 or 21.

24. a first slide scanner; 24. The system of claim 23, further comprising a second slide scanner.

25. 25. The system of claim 24, wherein the first slide preparer comprises the first slide scanner and the second slide preparer comprises the second slide scanner.

26. a first slide stainer; and 26. The system of claim 25, further comprising a second slide stainer.

27. 27. The system of claim 26, wherein the first slide preparer comprises the first slide stainer and the second slide preparer comprises the second slide stainer.

28. 28. The system of claim 27, wherein the first slide stainer uses a first slide staining technique and the second slide stainer uses a second slide staining technique that is different from the first slide staining technique.

Images