Medical spectroscopy and imaging analysis

Abstract

Systems and methods for medical imaging and spectroscopy are disclosed. Some embodiments relate to digital staining. Some embodiments relate to digital staining using hyperspectral imaging or multispectral imaging. Some embodiments relate to digital staining using RGB imaging. Some embodiments relate to other types of medical imaging and spectroscopy analysis. Some embodiments relate to platforms for performing analysis of medical imaging and spectroscopy data. Some embodiments relate to methods for analyzing digital stained images used in surgery.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 596,594, filed November 6, 2023, entitled “MEDICAL SPECTROSCOPY AND IMAGING ANALYSIS,” which is incorporated herein by reference in its entirety and for all purposes as fully set forth herein. Technical Field

[0002] The embodiments relate to the field of medical imaging, and more specifically, to methods, systems, and apparatus for analyzing and visualizing complex imaging data. Some embodiments relate to digital staining. Some embodiments relate to precision surgery. Background Technology

[0003] The methods described in this section are feasible, but need not be methods that have been previously conceived or implemented. Therefore, unless otherwise stated, no method described in this section should be assumed to be prior art simply because it is included in this section.

[0004] Chemical staining of tissue samples is a time-consuming and labor-intensive process. It can take more than a day to prepare a typical formalin-fixed, paraffin-embedded sample. Frozen tissue samples can be prepared more quickly, but preparation can still take a considerable amount of time. Furthermore, the chemical staining process is often destructive. That is, once a sample is stained, it is usually not possible to apply different staining agents. For example, hematoxylin and eosin (H&E) are commonly used to stain tissue samples. In many cases, pathologists or researchers may wish to identify and / or differentiate components observed in H&E-stained tissue samples. However, applying different staining agents may require preparing new tissue slides.

[0005] Chemical staining is a destructive process. Once a staining agent is applied, it is generally impossible to remove it and apply another staining agent in its place. In many cases, pathologists or researchers may want to differentiate and / or identify components observed in previously stained tissue sections. For example, a sample may initially be stained with hematoxylin and eosin, and then the pathologist may want to apply different staining agents, such as to detect microorganisms, lipids, carbohydrates, minerals, pigments, etc., in the tissue sample. With traditional chemical staining, this may require preparing new tissue samples.

[0006] Furthermore, it can be difficult to analyze stained tissue and other types of medical imaging data, such as CT scans, X-rays, MRI scans, and PET scans. Often, practitioners may struggle to interpret the images and may be unable to identify significant image features relevant to treatment or diagnosis.

[0007] In some cases, practitioners may lack the resources or knowledge to deploy advanced medical imaging and spectral analysis themselves. Summary of the Invention

[0008] Digital staining can alleviate many of the problems found in physical staining. Using digital staining, preparation time may be less, and the need to prepare multiple tissue samples when additional staining is required can be reduced or eliminated. Digital staining can, for example, involve acquiring hyperspectral images and then applying one or more transformations to the hyperspectral images to produce one or more digitally stained images. In some cases, the entire hyperspectral image can be captured before performing digital staining.

[0009] In some embodiments, the technology described herein relates to a method for generating a model for classifying digitally stained tissue images, the method comprising: receiving a first image of an unstained tissue sample, wherein the first image is a color image captured by a camera including at least one of a charge-coupled device sensor or a complementary metal-oxide-semiconductor sensor and a Bayer filter, and wherein the unstained tissue sample is a sample collected during a Moscone procedure; digitally staining the first image to produce a first digitally stained image; providing the first digitally stained image to a machine learning model; using the machine learning model to determine the presence of cancerous tissue depicted in the digitally stained image; and generating an indication of the presence of cancerous tissue in the first digitally stained image to display to a user.

[0010] In some embodiments, the techniques described herein relate to a method in which the machine learning model is a classifier model configured to classify digitally stained images as depicting cancerous tissue or not depicting cancerous tissue, wherein the machine learning model is trained using supervised learning on a training image set, wherein each training image in the training image set is a digitally stained image, and wherein each training image in the training image set is labeled as depicting cancerous tissue or not depicting cancerous tissue.

[0011] In some embodiments, the technology described herein relates to a method in which the machine learning model is an object recognition model configured to identify one or more regions of cancerous tissue depicted in a digitally stained image, wherein the machine learning model is trained using supervised learning on a training image set, wherein each training image in the training image set is a digitally stained image, wherein at least one training image in the training image set depicts at least one region of cancerous tissue, and wherein each of the at least one region of cancerous tissue is labeled as depicting cancerous tissue.

[0012] In some embodiments, the technology described herein relates to a method in which the first image depicts at least one of the following: basal cell carcinoma, melanoma in situ, dermatofibrosarcoma protuberans, keratoacanthoma, spindle cell tumor, sebaceous gland carcinoma, microcystic adnexal carcinoma, or leiomyosarcoma.

[0013] In some embodiments, the techniques described herein relate to a method in which a machine learning model is trained using a training image set, wherein each training image in the training image set depicts a frozen tissue sample, and wherein the model is trained to identify the presence of cancerous tissue in an image depicting a frozen artifact.

[0014] In some embodiments, the techniques described herein relate to a method in which the first image is captured at a magnification of about 2X to about 10X.

[0015] In some embodiments, the technology described herein relates to a method further comprising: identifying a marker in the first image, wherein the marker includes at least one of the following: a mark or dot on a slide, a mark or dot on tissue depicted in the image, a dye or stain on the slide, or a dye or stain on tissue depicted in the image.

[0016] In some embodiments, the techniques described herein relate to a method that further includes determining the orientation of the unstained tissue sample based on the location of the marker.

[0017] In some embodiments, the techniques described herein relate to a method in which the unstained tissue sample is digitally stained with digital hematoxylin and eosin staining agents.

[0018] In some embodiments, the techniques described herein relate to a method in which the unstained tissue sample includes one or more relaxed incisions.

[0019] In some embodiments, the technology described herein relates to a system for identifying cancerous tissue, comprising: at least one processor; and a non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by the at least one processor, causing the system to: receive a first image of an unstained tissue sample, wherein the first image is a color image captured by a camera including at least one of a charge-coupled device sensor or a complementary metal-oxide-semiconductor sensor and a Bayer filter, and wherein the unstained tissue sample is a sample collected during a Moscone procedure; digitally stain the first image to produce a first digitally stained image; provide the first digitally stained image to a machine learning model; use the machine learning model to determine the presence of cancerous tissue depicted in the digitally stained image; and generate an indication of the presence of cancerous tissue in the first digitally stained image to display to a user.

[0020] In some embodiments, the technology described herein relates to a system in which the machine learning model is a classifier model configured to classify digitally stained images as depicting cancerous tissue or not depicting cancerous tissue, wherein the machine learning model is trained using supervised learning on a training image set, wherein each training image in the training image set is a digitally stained image, and wherein each training image in the training image set is labeled as depicting cancerous tissue or not depicting cancerous tissue.

[0021] In some embodiments, the technology described herein relates to a system in which the machine learning model is an object recognition model configured to identify one or more regions of cancerous tissue depicted in a digitally stained image, wherein the machine learning model is trained using supervised learning on a training image set, wherein each training image in the training image set is a digitally stained image, wherein at least one training image in the training image set depicts at least one region of cancerous tissue, and wherein each of the at least one region of cancerous tissue is labeled as depicting cancerous tissue.

[0022] In some embodiments, the technology described herein relates to a system in which the first image depicts at least one of the following: basal cell carcinoma, melanoma in situ, dermatofibrosarcoma protuberans, keratoacanthoma, spindle cell tumor, sebaceous gland carcinoma, microcystic adnexal carcinoma, or leiomyosarcoma.

[0023] In some embodiments, the technology described herein relates to a system in which a machine learning model is trained using a training image set, wherein each training image in the training image set depicts a frozen tissue sample, and wherein the model is trained to identify the presence of cancerous tissue in an image depicting a frozen artifact.

[0024] In some embodiments, the technology described herein relates to a system in which the first image is captured at a magnification of about 2X to about 10X.

[0025] In some embodiments, the technology described herein relates to a system wherein the instructions are further configured to cause the system to: identify a mark in the first image, wherein the mark includes at least one of the following: a mark or dot on a slide, a mark or dot on tissue depicted in the image, a dye or stain on the slide, or a dye or stain on tissue depicted in the image.

[0026] In some embodiments, the technology described herein relates to a system in which the instructions are further configured to cause the system to: determine the orientation of the unstained tissue sample based on the location of the marker.

[0027] In some embodiments, the technology described herein relates to a system in which unstained tissue samples are digitally stained with digital hematoxylin and eosin staining agents.

[0028] In some embodiments, the technology described herein relates to a system in which the unstained tissue sample includes one or more relaxed incisions.

[0029] Importantly, digital staining accurately reflects what pathologists or other professionals would observe when using routine methods such as chemical staining, allowing them to apply existing knowledge and experience when analyzing samples. In some cases, it may be advantageous to use absorbance-based methods, which can provide improved performance compared to transmission-based methods. Attached Figure Description

[0030] These and other features, aspects, and advantages of this disclosure are described with reference to the accompanying drawings of certain embodiments, which are intended to illustrate the disclosure and not limit it. It should be understood that the drawings, which are incorporated in and constitute a part of this specification, are for illustrative purposes and may not be to scale.

[0031] Figure 1 A block diagram of a process for generating a visible spectral image based on a hyperspectral image is shown according to some embodiments.

[0032] Figure 2 A block diagram depicting the analysis process according to some embodiments is shown.

[0033] Figure 3 A block diagram illustrating a process that can run on a computer system to train a machine learning model is shown.

[0034] Figure 4A block diagram illustrating a process for generating an image feature map according to some embodiments is shown.

[0035] Figure 5 A block diagram illustrating a process for applying image transformation based on feature recognition, according to some embodiments, is shown.

[0036] Figure 6 A block diagram illustrating a process for applying image transformations based on specified conditions, according to some embodiments, is shown.

[0037] Figure 7 A block diagram illustrating a scalable platform according to some embodiments is shown.

[0038] Figure 8A and Figure 8B This is a block diagram illustrating an example of using an analytics platform in conjunction with a third-party system, according to some embodiments.

[0039] Figure 9 Examples of communication that can occur between different systems are shown in some embodiments.

[0040] Figure 10 This is a block diagram illustrating an example process for performing real-time staining according to some embodiments.

[0041] Figure 11 Examples of stained and unstained images according to some embodiments are shown.

[0042] Figure 12 An example of real-time staining according to some embodiments is shown.

[0043] Figure 13 This is a block diagram illustrating example processing for model training and testing according to some embodiments.

[0044] Figure 14 This is a block diagram illustrating an example of artificial intelligence model training processing according to some embodiments.

[0045] Figure 15 An example of a spectral waveform with hyperspectral image compression according to some embodiments is shown.

[0046] Figure 16 This is a block diagram illustrating an example process for performing digital staining according to some embodiments.

[0047] Figure 17 A graph showing illumination correction according to some embodiments is shown.

[0048] Figure 18 Examples of converting transmission spectra into absorption spectra according to some embodiments are described.

[0049] Figure 19 Example processing for training machine learning (ML) or artificial intelligence (AI) models using absorbed data, according to some embodiments, is described.

[0050] Figure 20 Example processing for training AI / ML models according to some embodiments is described.

[0051] Figure 21 Exemplary multiple images are shown according to some embodiments.

[0052] Figure 22 This is a block diagram illustrating an example process for multiple digital staining according to some embodiments.

[0053] Figure 23 Example processing for training and deploying models for digital coloring of RGB images, according to some embodiments, is shown.

[0054] Figure 24 Example processing for training a model for digitally coloring RGB images, according to some embodiments, is shown.

[0055] Figure 25 An example processing for color normalization according to some embodiments is shown.

[0056] Figure 26 An example of digital color adjustment according to some embodiments is shown.

[0057] Figure 27 An example computer system is shown that can be used to perform one or more embodiments described herein.

[0058] Figure 28 Several non-limiting examples of relaxation cutting are shown.

[0059] Figure 29 Example procedures for performing Mohs surgery or similar procedures are shown according to some embodiments.

[0060] Figure 30 This is a flowchart illustrating an example process for performing a Mozart procedure or similar procedure according to some embodiments.

[0061] Figure 31 This is a flowchart illustrating an example process for performing a Mozart procedure or similar procedure according to some embodiments.

[0062] Figure 32 An example method for registration in the depth direction z is shown according to some embodiments.

[0063] Figure 33An iterative co-registration process according to some embodiments is illustrated.

[0064] Figure 34 An example of a slice for co-registration along the depth direction z, which can be obtained from a 3D image according to some embodiments, is shown.

[0065] Figure 35 Example processing for providing histopathology platform services and charging for such services, according to some embodiments, is shown.

[0066] Figure 36 It is a block diagram depicting an embodiment of a computer hardware system configured to run software for implementing one or more embodiments of the health testing and diagnostic systems, methods and apparatuses disclosed herein. Detailed Implementation

[0067] Embodiments of the present disclosure will now be described with reference to the accompanying drawings. Because the terminology used in the description presented herein is employed in conjunction with the detailed description of embodiments of the present disclosure, it is not intended to be interpreted simply in a limiting or restrictive manner. Furthermore, embodiments of the present disclosure may include several novel features, none of which individually is responsible for its desired properties or is necessary for practicing the embodiments of the present disclosure described herein. For the purposes of this disclosure, certain aspects, advantages, and novel features of various embodiments have been described herein. It should be understood that, according to any particular embodiment, it is not necessary to achieve all of these advantages. Therefore, for example, those skilled in the art will recognize that an embodiment may be performed in a manner that achieves one or a set of advantages as taught herein without necessarily achieving other advantages as taught or suggested herein.

[0068] Various aspects of digital coloring are described in the following applications: U.S. Provisional Application No. 63 / 310014, filed February 14, 2022, entitled “HYPERSPECTRAL IMAGING ANALYSIS PLATFORM”; U.S. Provisional Application No. 63 / 315889, filed March 2, 2022, entitled “REAL TIME DIGITAL STAINING OF HYPERSPECTRAL IMAGES”; U.S. Provisional Application No. 63 / 269526, filed March 17, 2022, entitled “ABSORPTION-BASED SPECTRALMATCHING FOR DIGITAL STAINING”; and U.S. Provisional Application No. 63 / 269526, filed March 17, 2022, entitled “MULTIPLEXED DIGITAL STAINING WITH SPECTRAL MATCHING”. U.S. Provisional Application No. 18 / 168,263, filed October 11, 2023, entitled “MEDICAL SPECTROSCOPY AND IMAGING ANALYSIS”; the contents of each of these applications are incorporated herein in their entirety for all purposes, as if fully set forth herein.

[0069] Although voxels and 3D imaging techniques are mentioned throughout this specification, the systems and methods described herein are equally applicable to 2D imaging unless the context explicitly states otherwise. For example, the techniques described herein can be applied to 2D microscopic images, 2D radiographic images, and so on.

[0070] As described above and as will now be explained in more detail with reference to the accompanying drawings, this disclosure includes descriptions of systems and methods for analyzing imaging data, such as microscopic images (e.g., RGB images), multispectral images, hyperspectral images, magnetic resonance spectral images, dual-energy computed tomography (CT) scans, ultrasound images, Raman spectral data, 3D images obtained by capturing samples at multiple focal lengths, and the like. The systems and methods described herein can provide significant advantages in assisting the detection and diagnosis of medical conditions, but are not necessarily limited to medical applications.

[0071] Traditional medical imaging methods provide limited information to pathologists, radiologists, and other medical professionals. For example, in conventional tissue staining and labeling, pathologists use various stains or labels to enhance the contrast between tissue components, thereby improving visibility and enabling them to distinguish different tissues, cell populations, or organelles within individual cells. However, as briefly mentioned above, this approach is limited because a single tissue sample can typically undergo only a single staining or labeling, so multiple samples may be required to use more than one stain or label. Furthermore, conventional tissue staining and labeling techniques are limited by the availability of suitable stains and labels, and it may not be possible to find available stains or labels that can reliably identify features of interest in a tissue sample. In some embodiments, the systems and methods of this paper mitigate some or all of these limitations, as actual staining or labeling may not be necessary. In some embodiments, the systems and methods of this paper are able to identify features of interest without physically or chemically altering the sample.

[0072] Similarly, conventional CT scans typically provide limited information. In a conventional scan, X-rays pass through the patient from a source and are detected by an X-ray detector. The X-ray source can be, for example, an X-ray tube, which emits radiation across a range of wavelengths. However, each voxel in a CT scan can be assigned a single value based on the attenuation of X-ray intensity at that location. This significantly limits the application of CT scans because different tissues may have similar X-ray absorption characteristics, making it difficult or impossible to distinguish tissue types. Tissue types can be distinguished by collecting additional data. For example, photoelectric scattering is highly dependent on atomic number and is the dominant attenuation mechanism at low X-ray energies, while Compton scattering is not strongly dependent on atomic number and is the dominant mechanism at higher X-ray energies. Therefore, tissue types can be distinguished by collecting data at relatively high and relatively low energies. Data can be collected by using two X-ray sources with different energies or by switching a single source between different peak kilovolt voltages (kVp). Alternatively, a single X-ray tube can be used in conjunction with a detector capable of distinguishing X-rays of different energies.

[0073] Technological advancements have enabled the collection of significantly more data than was previously possible, and in some cases, have replaced or enhanced traditional analytical techniques. For example, hyperspectral imaging can be used to gather information from tissue samples, providing information across a much larger wavelength range than imaging in the visible spectrum, and this information can be used to enable improved analytical methods. For instance, instead of chemically or physically staining or labeling tissue samples, unstained tissue can be scanned, and staining agents can be applied virtually. Hyperspectral data can be analyzed, and images based on this data can be generated that simulate the results of physical staining of tissue samples with high accuracy suitable for use in medical diagnostics. Furthermore, because analyzing hyperspectral data is non-destructive, multiple staining agents can be simulated to identify multiple features of interest in a single tissue sample, eliminating the need to prepare multiple tissue slides. Additionally, novel staining agents without physical counterparts can be developed, allowing the identification of features that would be difficult or even impossible using traditional staining techniques.

[0074] While hyperspectral data processing enables a wide range of analyses, the cost of acquiring hyperspectral data can be a significant barrier. As discussed in this paper, technological advancements have made it possible to accurately digitally stain unstained tissue images captured using conventional cameras with red, green, and blue channels.

[0075] It should be understood that the above examples are merely illustrative, and the systems and methods described herein can be applied to other imaging and spectroscopic techniques. As an example only, the systems and methods described below can be used to analyze metabolites in tissues using data obtained from magnetic resonance spectroscopy.

[0076] Hyperspectral imaging, multispectral imaging, dual-energy CT scans, and magnetic resonance spectroscopy can contain a wealth of information that is not easily interpreted by humans. For example, hyperspectral or multispectral images can contain data beyond the visible spectrum, and even within the visible spectrum, human observers may not be able to easily distinguish narrow wavelength bands with step sizes of 1 nm or less in a hyperspectral image, nor can they easily distinguish narrow wavelength bands with step sizes of 20 nm or less in a multispectral image. For CT scans, a limited number of projections are used to reconstruct images of objects that can be observed by humans. For dual-energy CT scans, additional complexity may arise in transforming the raw data into a human-interpretable form. Even if imaging data is easily interpretable, such as visible images of unstained tissue captured using an RGB camera, it may not be presented in a way that provides practitioners with key insights. For example, RGB images of tissue slides offer practitioners limited information. For instance, practitioners may not be able to distinguish cell nuclei and extracellular matrix in images of unstained tissue samples. Computational methods can enable practitioners to make fuller use of available data.

[0077] Analysis is typically performed using human-interpretable data representations. For example, analysis of stained (or virtually stained) slides can be performed by examining only images of dummy stains and, for instance, identifying features within those images. This approach can provide useful insights, but it is rather limited because it ignores much of the available data. For instance, in the case of hyperspectral images of tissue slides, the visible representation may include only a small fraction of the total data. In some cases, 70%, 80%, 90%, or more of the data may be discarded to generate downsampled images containing visible red, green, and blue (RGB) (or cyan, magenta, yellow, and black (CMYK, etc.)) values, instead of the entire hyperspectral dataset. Similarly, when raw CT scan data is converted to a human-readable format, data may be lost or discarded.

[0078] Analysis Platform

[0079] In some embodiments, the analysis platform retains and can utilize all or at least a majority of the raw data from medical imaging and / or spectral data. For example, features within an image or digital staining of an image can be identified by examining the spectrum of each voxel in a hyperspectral image. For example, the analysis platform can digitally apply hematoxylin and eosin (H&E) staining agents to a hyperspectral image based on hyperspectral data. In addition to virtual simulations of applying conventional real-world staining agents, the platform may be able to identify other important features in the hyperspectral data. For example, features of specific types of proteins, cells, etc., may be present, such as spike proteins that may indicate infection with coronaviruses, retroviruses, etc. Similarly, the analysis platform can identify different components in dual-energy CT scan data by comparing attenuation at various wavelengths. While raw image data may be preferred for the analysis platform, it will be understood that in some cases, downsampled data can be used to improve the performance of the analysis system, reduce storage requirements, etc. For example, in some embodiments, images captured over a broad spectrum can be reduced to images containing only red, green, and blue values. In some embodiments, the resolution of the imaging or spectral data may be reduced. For example, the resolution of an image can be reduced by combining 2×2 pixel squares into a single pixel. This single pixel may have, for example, the average value of the 2×2 pixels from which it was created. In some embodiments, spectral data can be reduced, for example, by eliminating data at certain wavelengths. For example, regions in the spectrum that are not expected to contain useful information can be removed, or the step size can be increased by, for example, discarding one data point every other data point, discarding one data point every three data points, etc. In some embodiments, the original data itself may be relatively limited. For example, instead of a hyperspectral or multispectral image, the image may be an image in the visible spectrum with only a limited number of channels, such as an RGB image including red, green, and blue channels.

[0080] In some embodiments, the analysis platform can perform analysis on raw data, downsampled data, or both, and provide a human-interpretable data representation to the user of the analysis platform. In some embodiments, the analysis platform can perform relatively simple transformations on the raw data, such as selecting a subset of wavelengths from a hyperspectral image and converting that subset to visible RGB values ​​for display (e.g., by applying a constant scaling factor to each subset), or determining the reconstruction of a CT scan based on a single attenuation value at each voxel. In some embodiments, the analysis platform can perform more complex transformations, such as applying a virtual stain to a hyperspectral or RGB image, or generating color-coded CT scan images that distinguish different tissue types.

[0081] Advantageously, the analysis platform can be used to accelerate professional review by highlighting regions of interest, identifying substances, recognizing features in images, etc. For example, where spectral data has already been captured, professionals can use the platform to confirm cell types by analyzing spectral data associated with cells. In some cases, the analysis platform can help professionals identify components, cell types, structures, medical conditions, etc. In some embodiments, the analysis can be based on spectral data. In some embodiments, the analysis can be based on, for example, edge detection, shape, size, etc.

[0082] In some embodiments, the analytics platform may run locally, such as on a practitioner's computer. Alternatively or additionally, the analytics platform may run on a remote server, which may be operated by a healthcare provider, laboratory provider, imaging service provider, etc. For example, an imaging service provider may provide a platform that can be used to process and / or analyze data, for example, via web applications, electronic applications, local applications, etc.

[0083] In some embodiments, an artificial intelligence / machine learning (AI / ML) model can be trained to identify features in raw imaging data, features in downsampled imaging data, features in spectral data (e.g., Raman spectral data), etc. For example, an AI / ML model can be trained to identify features in spectral data associated with a specific voxel, which may indicate the composition of the material in the voxel. For example, an AI / ML system can identify lipids, cell nuclei, carbohydrates, etc., based on captured spectral data. In some embodiments, an AI / ML system can be trained to identify features based on shape in an image. In some embodiments, an AI / ML model can be configured to recommend transformations to be applied to an image. For example, based on features from raw or downsampled data, an AI / ML model may recommend applying certain virtual staining agents to hyperspectral images, recommend transformations to be applied to dual-energy CT scans to highlight certain features, recommend transformations to highlight certain metabolites in magnetic resonance spectroscopy scans, etc. Information about various medical conditions can be used to train an AI / ML system to determine features observed in or in a visual projection of spectral data that indicate certain medical conditions.

[0084] In some embodiments, an AI / ML model can be trained to determine a downsampling transformation, which can be used to reduce the size of image or spectral files without compromising the ability to perform analysis. For example, an AI / ML model can be trained to identify regions in hyperspectral imaging data that do not contain information that can be used for image analysis (such as wavelength ranges that do not show peaks in transmission or absorption). Thus, for example, the storage space required to store images, spectral data, or both can be reduced. The computational resources required to process images and other data can be reduced without losing important information.

[0085] In some embodiments, a user of the analysis platform may request an initial transformation (such as a virtual stain), and the platform may recommend additional transformations to the user, or in some cases, apply additional transformations automatically. In some embodiments, a user may specify a suspicious condition, and the analysis platform may determine the transformations to be applied. In some embodiments, a user may provide only the raw imaging data, and the system may identify one or more features of interest (e.g., cell walls, spike proteins, regions with spectral features indicating anomalousness, etc.) based on the raw imaging data, and determine one or more transformations to be applied to the raw imaging data.

[0086] For example, AI / ML models can be trained to identify structural features, spectral features, etc., and the dyes or other transformations to be applied can be determined based on the identified structural features, spectral features, or both.

[0087] In some embodiments, third parties may wish to extend the analytics platform. For example, a third party may develop new virtual stains, feature detection algorithms, visualization tools, etc. A third party may wish to display data in different ways or visualize data from imaging methods not originally supported by the analytics platform. In some embodiments, the analytics platform may provide application programming interfaces (APIs), libraries, etc., to allow third parties to extend the analytics platform. In some embodiments, a third party may create extensions for its own use or make extensions available to other users of the analytics platform. In some embodiments, the analytics platform may include features that allow third parties to charge for access to extensions. For example, the platform may include an online store that can be used to purchase access to extensions. In some embodiments, the platform provides access to extensions on a subscription basis. In some embodiments, the platform provides access to extensions on a perpetual license basis. In some embodiments, access to the platform itself is provided on a subscription basis, a perpetual basis, or both. In some embodiments, the platform includes a payment system. In some embodiments, the payment system may directly collect payment information. In some embodiments, the payment system interacts with one or more third-party payment processors to facilitate payments.

[0088] Figure 1Example processing for transforming a hyperspectral image into a visible image, according to some embodiments, is illustrated and can be run on a system configured to run an analysis platform. At box 101, the system may receive a hyperspectral image. For example, a user may upload an image to the system. At boxes 102, 103, 104, and 105, the system may extract spectral data for each voxel, select a subset of the spectral data, and map the voxels to colors in the visible spectrum based on the subset of spectral data (e.g., by determining RGB values). In some embodiments, the system may store one or more image transformation matrices indicating how the received image can be transformed. At decision point 106, the system may exit the loop after mapping the last voxel of the hyperspectral image, and at box 107, construct a visible image from the mapped voxels. In some embodiments, the subset selected at box 104 may be determined, for example, based on user selection (such as a dropdown menu allowing the user to select a predefined range of wavelengths (or frequencies, energy, or wavenumbers). For example, the user may select an option to display a representation of data in the infrared or ultraviolet region. For example, in some embodiments, the user may select from multiple predetermined wavelength regions. In some embodiments, a user can specify start and stop wavelengths, wavenumbers, frequencies, or energies. In some embodiments, a user can save configuration data (e.g., wavelength ranges) for future reuse. In some embodiments, a user can share configuration data with other users, such as with other users in the user's organization, or more broadly with other users. In some embodiments, the system may have additional features. For example, the color mapping can be modified based on a user's request to change contrast, brightness, or apply different mappings. As just one example, a user might want to modify the mapping so that a specific wavelength range of interest stands out in the constructed visible spectrum image.

[0089] Although Figure 1 The illustration shows the transformation of a hyperspectral image into a visible image based on a subset of hyperspectral data; however, those skilled in the art will recognize that similar methods can be used to visualize other types of data. For example, instead of receiving a hyperspectral image at box 101, the system could receive dual-energy CT scan data and map said data into a visible representation, or it could receive an RGB image and manipulate the red, green, and blue channels of the image. In some embodiments, the system could receive magnetic resonance spectral data and map said data into a visible representation. The subset of spectral data and color mapping can be selected based, for example, on a specific metabolite of interest.

[0090] Figure 2 This is a block diagram illustrating analysis processing according to some embodiments. The computer system can be configured to perform... Figure 2The processing described in [the document] is as follows. At box 201, the system receives an analysis request packet from the user of the analysis platform. At box 202, the system extracts an image from the analysis request packet, which may be, for example, a hyperspectral image, an RGB image, a CT scan, an MRI, a PET scan, etc. At box 203, the system extracts a request from the analysis request packet to apply a transformation to the image. In some embodiments, the request may be, for example, applying a virtual stain to a hyperspectral image or applying another transformation that would be useful to pathologists, radiologists, etc. At box 204, the system may apply the requested transformation to each voxel of the received image. In some embodiments, [the system may]... Figure 1 The processing shown completes the transformation requested by the application. At box 205, the system can generate a new transformed image based on the result of transforming each voxel in the received image. At box 206, the system can make the transformed image available to the user.

[0091] In some embodiments, the system may receive images from a user and may receive instructions or requests to perform transformations on the received images independently, rather than receiving analysis request packets from the user. For example, a user may upload images to the system and may select one or more transformations to apply via a user interface, such as a computer program communicating with the system or a website or web application controlled by a provider of the analysis platform. In some embodiments, the imaging system (e.g., a computer system associated with an imaging device) may be configured to upload imaging data to the analysis platform automatically, manually, or both. In some embodiments, a user may select one or more images already uploaded to the analysis platform for analysis.

[0092] In some embodiments, the analytics platform may make additional functionalities available to the platform's users. For example, the analytics platform may use artificial intelligence or machine learning (AI / ML) models to detect features, identify potential conditions, recommend additional imaging analyses, etc. Figure 3This is a block diagram illustrating a process that can run on a computer system to train a machine learning model. At box 301, the system can receive a dataset, which can be, for example, a set of hyperspectral tissue images, a set of CT scans, a set of RGB images, etc. At box 302, one or more steps can be performed to prepare the dataset, such as, for example, removing duplicates, adding or modifying metadata (e.g., images can be labeled with diagnoses corresponding to cancer types), etc. At box 303, the system can receive one or more features of interest. For example, the system can be configured to identify image regions with specific spectral characteristics, shapes, etc. At box 304, the system can create training, tuning, and testing datasets from the received dataset. In training loop 315, the system trains the model using the training dataset box 305 at box 308. Training can be performed in a supervised, unsupervised, or partially supervised manner. At box 309, the system can evaluate the model according to one or more evaluation criteria. For example, the evaluation may include false positive rate, false negative rate, true positive rate, true negative rate, etc. At decision point 310, the system can determine whether the model meets one or more evaluation criteria. If the model evaluation fails, at box 311, the system can tune the model using the tuning dataset 306, repeating the training box 308 and the evaluation box 309. Once the model passes the evaluation at 310, the system can exit the model training loop 315. The test dataset 307 can then be run with the trained model 312, and the system can evaluate the results at box 313. If the evaluation fails (e.g., due to an unacceptable false positive or false negative rate), at decision point 314, the system can re-enter the training loop 315 for additional training and tuning. If the model passes, the system can stop the training process, resulting in the trained model 312.

[0093] In some embodiments, the analysis platform may use an AI / ML model or another suitable algorithm to identify and label features in an image. Figure 4This is a block diagram illustrating a process for generating an image feature map that can be implemented on a computer system according to some embodiments. At block 401, the system receives an image analysis request packet from a user of the system. At block 402, the system extracts image data from the analysis request packet. The image may be, for example, a hyperspectral image, a conventional CT scan with only a single value per voxel, a dual-energy CT scan, a multispectral image, an RGB image, a magnetic resonance spectral image, etc. At block 403, the system extracts a request from the analysis request packet to apply a transformation to the received image. At block 404, the system may use an artificial intelligence model or other methods to identify one or more features (e.g., composition, shape, etc.) in the received image data based on the received image data. At block 405, the system generates a map of the features in the received image data. At block 406, the system may apply the requested transformation to each voxel of the received image data to generate a new transformed image at block 407 for the user to view. At block 408, the transformed image, along with features determined from the received image data, can be... Figure 1 It is available to the user. In some embodiments, the feature map may be presented as an overlay on the generated image, which can be turned on or off by the user. For example, the feature map may include labels, circles, rectangles, colored areas, etc., indicating regions of interest in the generated image.

[0094] In some embodiments, not Figure 4 All steps indicated in the instructions can be performed. For example, in some embodiments, a user can submit an image for feature recognition without requesting a transformation. As just one example, a user can submit a conventional CT scan containing a single value at each voxel, and the system can recognize features in the CT scan without applying a transformation to the received data (or only applying a minimal transformation, such as converting the absorption values ​​to a grayscale representation).

[0095] The above references an image analysis request package. In some embodiments, an image analysis request package may include image data and information indicating how the image should be processed. In some embodiments, the contents of an image analysis request package may be received separately. In some embodiments, a request package may include one or more images. In some embodiments, a request package may include a request to perform analysis on one or more images that have previously been uploaded to the analysis platform. In some embodiments, instead of images or in addition to images, spectral or other data may be provided to the platform. In some embodiments, a request package may include multiple images or spectra and may include instructions applied to each image or spectrum, which may be different or the same.

[0096] In some embodiments, the system may detect features based on a processed representation of the image rather than the original image data. While this may not be preferred, having only a subset of the original data or a representation of that subset limits the information available to the system for performing feature detection, potentially preventing the system from recognizing features present in the original data but not in the processed representation. Although more data is generally preferred, it should be understood that useful analyses can be performed on a variety of images, including RGB images, which can enable better access to imaging analysis, especially where the cost of high-end imaging equipment, such as hyperspectral imaging systems, is a significant barrier to access.

[0097] In some embodiments, the analysis platform may be configured to determine the transformation to be applied based on the analysis of the received raw data (however, as mentioned above, in some embodiments, the raw data may not be used). Figure 5 This is a block diagram illustrating a process for applying image transformations based on feature recognition, which can run on a computer system according to some embodiments. At block 501, the system receives an image analysis request packet from a user of the system. At block 502, the system extracts an image from the analysis request packet. The image may be, for example, a hyperspectral image, a dual-energy CT scan, a multispectral image, a magnetic resonance spectral image, etc. At block 503, the system extracts a request from the analysis request packet to apply a transformation to the received image. At block 504, the system applies the requested transformation to the image to generate a new transformed image at block 505. At block 506, the system may use an artificial intelligence model, such as a trained model 312, or by using other suitable techniques to identify features in the received image data. At block 507, the system may determine the transformation to be applied to the received image data based on the identified features in the received image data. For example, if the received image data indicates cancer, the system may recommend additional transformations, which may, for example, help identify the type of cancer, help distinguish between cancerous tissue and healthy tissue, etc. Alternatively or additionally, the system may recommend additional transformations based on a transformation request received from the user. For example, if a user requests that a specific virtual staining agent be applied to a hyperspectral image, the system can recommend additional virtual staining agents based on the requested virtual staining agent. At box 508, the system can apply the recommended transformation to each voxel of the received image, either in response to a request from the user or automatically. At box 509, the transformed voxels can be used to generate a new transformed image. At box 510, the generated new image can be made available to the system's user. In some embodiments, not... Figure 5All steps in the process can be performed. For example, the user can omit the request to apply a transformation. In this case, the system can determine the recommended transformation based on the identified features in the received raw image data, but the steps in boxes 503, 504, and 505 will not be performed.

[0098] In some embodiments, instead of specifying (or in addition to) specifying the transformations to be applied, users of the analysis platform can specify conditions or questions of interest. For example, a user might be looking for a particular type of cancer, trying to identify specific metabolic pathways, or searching for infections with a specific type of virus. Figure 6 This is a block diagram illustrating a process that can run on a computer system to apply image transformations based on specified conditions, according to some embodiments. At block 601, the system receives an image analysis request packet from a user of the system. At block 602, the system extracts raw image data, such as hyperspectral images, dual-energy CT scans, etc., from the analysis request packet. At block 603, the system extracts a question of interest from the analysis request packet, which may be a request to find, for example, cancer, viruses, metabolic problems, etc. At block 604, the system determines at least one transformation to be applied to the image data based on the question of interest. At block 605, the system applies at least one transformation to each voxel of the image data. At block 606, the system generates at least one transformed image based on the at least one transformation applied to the image data. At block 607, the system makes at least one new transformed image available to the user.

[0099] In some embodiments, a scalable platform may be advantageous. For example, the platform provider may enable core functionalities to work with imaging data such as visible spectral images, hyperspectral images, magnetic resonance spectral images, dual-energy CT scans, etc. For example, the platform provider may make tools available for common tasks, such as virtually staining images with common staining agents, or for transforming CT data to differentiate tissue types. The platform provider may also provide a set of analytical tools, such as feature recognition, tagging, autonavigation (e.g., automatically guiding the user to image regions containing features of interest), etc. In some embodiments, the platform provider may make the platform open to third parties. Third parties may, for example, develop new virtual staining agents that may or may not exist in nature, develop transformations for other imaging techniques, build new analytical tools, etc. For example, a third party may wish to develop new transformations or tools for research, or wish to help detect emerging conditions. In some cases, third parties may wish to make transformation and analytical capabilities available only to themselves, or they may wish to make transformation and analytical capabilities available to other users of the analytical platform, either free of charge or for a fee, which may be a one-time fee or a continuous subscription fee. In some implementations, third parties may configure whether to allow others to access their tools, libraries, functions, references, etc.

[0100] Figure 7 A scalable platform according to one embodiment is illustrated. Image data 701, such as hyperspectral image data, CT data, etc., can be manipulated using local functionality 702. This local functionality may include analysis tools 704, image transformations 705, and AI models 706 (such as a trained model 312), but local functionality 702 may also include other features. For example, the local functionality may also include a reference tool 707, which may be configured, for example, to provide information to users of the platform based on analysis. For example, reference tool 707 may show example images, provide information about the condition, explain how different staining agents can be used to identify the condition, etc. In some embodiments, the platform may include a payment system 713.

[0101] Native functionality 702 includes an application programming interface 708 or other means that allow third parties to interact with the platform. Third parties can build third-party functionality 703, which may include third-party analysis tools 709, third-party transformation libraries 710, third-party analysis functions 711, and third-party reference tools 712. In some embodiments, only a subset of the third-party functionality is available, or additional third-party functionality may be made available. Third-party transformation libraries 710 may include, for example, new virtual staining agents for hyperspectral images, new transformations for magnetic resonance spectral images, etc. Third-party analysis functions 711 may be integrated with native analysis tools available to the platform provider.

[0102] In some cases, a third party may provide a third-party analytics tool 709 that operates at least partially independently of the tools provided by the platform provider. For example, analytics functionality may function as a plug-in to an existing tool (e.g., analytics tool 704 provided by the platform provider), while the third-party analytics tool 709 may be a standalone application.

[0103] Figure 8A and Figure 8B Examples of using an analytics platform in conjunction with third-party systems, according to some embodiments, are shown. Figure 8A and Figure 8B As shown, there are various methods for utilizing third-party systems. Figure 8A In this approach, the client interacts directly with a third-party system. This method can be used, for example, when a third party has already developed software that the client will use directly (e.g., web applications, native applications, etc.). Figure 8AIn the diagram, at circle 1, the client computing system interacts directly with the third-party system, for example, using an application provided by a third party. In some embodiments, the third-party system can perform processing on data received from the client computing system. In some embodiments, processing can be performed according to processing instructions provided by the client computing system. In some embodiments, processing can be performed by the third-party system without requiring specific instructions from the client computing system. At circle 2, the third-party system uses an API to construct relevant calls to the analytics platform's functions. At circle 3, the analytics platform receives API calls. API calls may include, for example, calls to perform specific operations on data provided by the third-party system. At circle 4, the analytics platform can prepare results and use the API to pass them to the third-party system. At circle 5, the third-party system can use the API to receive results. In some embodiments, results can be automatically pushed to the third-party system. In some embodiments, the third-party system can execute a second API call or set of API calls to retrieve results from the analytics platform. At circle 6, the third-party system can provide results to the client computing system. In some embodiments, the third-party system can automatically provide results to the client computing system. For example, the third-party system can push results to the client computing system. In some embodiments, the third-party system can be configured to provide results to the client computing system after receiving a request for results from the client computing system. In some embodiments, a third-party system may process the results before providing them to the client computing system.

[0104] Although Figure 8A This illustration depicts a scenario where a client computing system interacts directly with a third-party system; however, other implementations are possible. For example, in some cases, the third party may not implement a user interface, or may wish to provide only a limited user interface, such as a configuration panel, that can be integrated into an interface provided by the analytics platform. Figure 8BIn the diagram, at circle 1, the client computing system directly interfaces with the analytics platform. For example, the client computing system can provide images or other data to the third analytics platform. In some embodiments, the client computing system can provide the analytics platform with a request to perform specific processing. In some embodiments, some or all of the processing can be performed by a third party. In some embodiments, the analytics platform can perform processing before sending a processing request to the third-party system. At circle 2, the analytics platform can use an API to send a processing request to the third-party system. At circle 3, the third-party system can receive the processing request. In some embodiments, the processing request can specify the data to be processed and instructions for the processing steps to be performed by the third-party system. The third-party system can perform processing on the data using the API according to the instructions received from the analytics platform. At circle 4, the third-party system can use the API to make the processing results available to the analytics platform. In some embodiments, the third-party system can push the results to the analytics platform. In some embodiments, the analytics platform can request the results from the third-party system via the API. At circle 5, the analytics platform can receive results from the third party. In some embodiments, the analytics platform can perform additional processing. At circle 6, the analytics platform can provide the results to the client computing system. In some embodiments, the analytics platform can push the results to the client computing system. In some embodiments, the client computing system can request the results from the analytics platform. In some embodiments, the analytics platform may provide client systems with notifications that results are available.

[0105] Figure 9 Examples of communication that can occur between different systems are shown. Systems may include client computing systems, third-party systems, imaging systems, and analysis platforms. These are merely non-limiting examples. In some embodiments, not all systems may be present. In some embodiments, additional systems, not shown, may be present. Figure 9 As shown, the systems can communicate directly with each other or via an analysis platform. For example, an imaging system such as a microscope can be networked, enabling it to communicate directly with the analysis platform, a third-party system, or a client computing system. In some embodiments, the imaging system may not be connected to a network and therefore may not be able to communicate directly with other systems. For example, the imaging system can store data on a removable storage medium that can later be connected to a client computing system, which can then communicate with the analysis platform, a third-party system, or both.

[0106] In some embodiments, the analysis platform may be integrated with an electronic medical record (EMR) system, such as an EMR system operated or used by a medical facility, hospital, etc. In some embodiments, the analysis platform may communicate with the EMR system to receive images for digital staining, receive requests for digital staining, provide digitally stained images to a provider, provide annotations or reports to a provider, etc. For example, in some embodiments, the analysis platform may include features that allow input of annotations. In some embodiments, the analysis platform may provide image annotation. For example, the analysis platform may provide functionality for manual image annotation, automatic image annotation, or both.

[0107] Real-time digital staining

[0108] The examples above enable powerful processing and analysis of imaging and other medical data. However, these examples are often (but not always) used after the imaging data has been fully captured. In some cases, such as due to a lack of relevant features in the image, the image may not provide useful information. Therefore, providing some degree of real-time processing (e.g., real-time or substantially real-time digital staining) can be advantageous. This allows practitioners to quickly identify whether image capture will be useful or is unlikely to produce useful results. In such cases, practitioners can abandon capture processing and adjust their procedures accordingly, for example, by adjusting image capture parameters (e.g., exposure time), by moving to different areas on the slide, by preparing samples (e.g., new tissue slides), etc.

[0109] Therefore, some embodiments of this document relate to systems and methods capable of real-time digital coloring. As used herein, real-time or substantially real-time means within a short time period, such as within half a second, within one second, within two seconds, within three seconds, within four seconds, within five seconds, etc. Some delay is expected and anticipated, provided that the delay is short enough to allow practitioners to review and make decisions based on the digitally colored image before the uncolored image is fully captured.

[0110] Real-time digital staining can be particularly useful in multispectral or hyperspectral imaging, which can take a significant amount of time to acquire. For example, image acquisition processing can take several minutes (e.g., 5 minutes, 10 minutes, 15 minutes, etc.) depending on various factors such as the size of the field being captured, the exposure time, the range of wavelengths captured, the wavelength step size, the number of bands, etc.

[0111] Figure 10The process for real-time digital staining according to some embodiments is described. In the setup phase, at operation 1002, the user configures the microscope for automatic image capture. At operation 1004, the user selects multiple blocks to scan from a grid. For example, the user can select a subset of blocks from the microscope's full field of view. At operation 1006, the user places an unstained slide under the microscope to prepare for imaging. At operation 1008, the user selects the area to be imaged, and at operation 1010, the user adjusts the image focus. During the capture and real-time staining phases, a counter i can be initialized and used to count the multiple blocks that have been scanned. At operation 1012, the system can capture the data cube of the i-th block. At operation 1014, the system can append the data of the i-th block to the previously captured data cube. At operation 1016, the system can calculate the digitally stained image and display it to the user. The counter i can be incremented to indicate that the appended block has been captured. At operation 1018, the system can check to see if all blocks have been captured. If there are still blocks to capture, the system can repeat operations 1012, 1014, and 1016 to capture the next block, attach the next block to the previously captured data cube, and calculate and display the digitally stained image. In some embodiments, the system can recalculate the digital staining agent for all captured blocks. In some embodiments, the system can calculate the digital staining agent only for the most recently captured block, and can attach the digitally stained block to any other block that has already been digitally stained. If, at operation 1018, the system determines that there are no more blocks to capture, the capture and real-time staining phase can be completed. During the post-processing phase, the user can apply additional staining agent at operation 1020. In some embodiments, the system can use additional staining agents. The staining agent applied at operation 1016 can be, for example, a common staining agent such as H&E, and the additional staining agent applied at operation 1020 can be a specialized staining agent for distinguishing lipids, muscle, organisms, minerals, etc. At operation 1022, one or more final images can be saved. The final images can be, for example, an unstained image, an image digitally stained with H&E, an image digitally stained with a special staining agent, etc.

[0112] In some embodiments, the number of grids (or fields) can be defined based on the active tissue field to be imaged. For example, for a full slide image scan at 40× magnification, the field size could be 1024×1024 pixels, representing a physical region of approximately 166 micrometers × approximately 166 micrometers. There can be tens or hundreds of fields. In some cases, it may be desirable to reduce the number of fields. For example, for hyperspectral imaging, it may be desirable to reduce the number of fields to reduce acquisition time. In some embodiments, a region of interest can be identified, and fields in or near the region of interest (e.g., adjacent to the region of interest) can be selected.

[0113] Figure 10 The processing using a hyperspectral line scanner is illustrated. However, it will be understood that other imaging modalities are possible. For example, a snapshot scanner can capture the entire field at once (like an optical RGB camera). For snapshot scanners, optical RGB cameras, etc., a block can be considered as the entire field. Instead of a block comprising one or more rows of the image, each field can be processed to generate a digitally stained image.

[0114] Figure 11 Example interfaces for displaying images, according to some embodiments, are depicted. Figure 11 In the image display area 1100, there is a digitally tinted view 1102 corresponding to the untinted view 1104. For example... Figure 11 As shown, digitally tinted and untinted views can depict incomplete captures. The image display area can be updated when untinted image data or digitally tinted image data becomes available. For example, in some embodiments, the system can be configured to perform digital tinting of each voxel in the image as the image is captured. In other embodiments, the system can be configured to perform digital tinting periodically (e.g., after capturing each row, after capturing 10 rows, after capturing 20 rows, etc.). As described above, in some embodiments, the system can be configured to perform digital tinting after capturing each block. Reducing the number of voxels captured before performing digital tinting can provide digitally tinted image data to the user more quickly, but trade-offs can be made in terms of, for example, the computational power required to compute the digitally tinted image. In some embodiments, digital tinting can produce better images if more voxels are captured. For example, if more voxels are used, the system can consider more spatial information. In some embodiments, the digital tinting algorithm can anticipate more than a single voxel. For example, the digital tinting algorithm can use information about neighboring voxels (e.g., nearest neighbor, second nearest neighbor, etc.) to determine how the voxels for digital tinting should appear.

[0115] Figure 12 A real-time digital staining process, which can be implemented on a computer system according to some embodiments, is described. Figure 12 In this process, the image is divided into twelve horizontal slices, and as slicing is completed, the system applies digital staining to each slice. The system adds each new slice to the previous slices, ultimately forming a complete digitally stained image. (As mentioned above...) Figure 11 The number of slices discussed can vary depending on the user's needs, the capabilities of the computing system, and / or the algorithm used to perform the digital staining process.

[0116] Model training and testing

[0117] Hyperspectral microscopy provides data in the form of data cubes containing two spatial dimensions and one spectral dimension (wavelength). Typically, a high-resolution hyperspectral image data cube can contain a large number of wavelengths, for example, more than 300, 400, or 500. In some cases, hyperspectral imaging devices can capture data in the visible and near-infrared wavelength range from about 400 nm to about 1000 nm with a wavelength resolution of, for example, about 1 nm. Short-wave infrared devices can capture data from about 900 nm to about 1700 nm in increments of about 5 nm. In the mid-infrared range from about 2500 nm to about 13000 nm, commercially available detectors can provide measurements with a spectral resolution of about 13 nm. Therefore, hyperspectral images can be very large and may require significant computational resources for manipulation and analysis. Furthermore, because pathological tissues typically contain densely packed, spatially distributed microstructures, adjacent pixels can contain different spectra. Therefore, a method is needed to perform digital staining on simplified datasets without losing crucial information.

[0118] Digital staining agents can be created by training artificial intelligence (AI) or machine learning (ML) models to convert between unstained and chemically stained images. For real-time or near-real-time digital staining, it is advantageous to reduce the amount of data and processing time used to perform the digital staining function. This can be achieved, for example, by working with a compressed or simplified dataset that transforms a hyperspectral data cube into a multiband data cube. In some embodiments, an AI / ML model can be trained to convert hyperspectral images into digitally stained images using multiband data.

[0119] Figure 13 This is an overview of a process for training and testing AI / ML models that can run on a computer system, according to some embodiments. At box 1302, the system can generate co-registered, lamp-normalized (e.g., considering the emission properties of the light source) data cube pairs from unstained and stained captures of the same tissue sample. At box 1304, the system can compress the data cube pairs to generate multi-band data cube pairs. At box 1306, the multi-band data cube pairs can then be normalized (e.g., scaled so that the values ​​range from 0 to 1 or from -1 to 1) to prepare for training the AI ​​model. At box 1308, the system can train the AI ​​model to obtain digitally stained multi-band data cubes from the unstained multi-band data cubes. That is, the system can train the AI / ML model to map unstained multi-band images to stained multi-band images. In some embodiments, a generative adversarial network can be utilized to generate the digital stain.

[0120] Although the uncolored and colored images were co-registered prior to training at box 1308, this co-registration may be inaccurate. Errors in co-registration can significantly impact the performance of AI / ML models. For example, even small differences in feature locations within the colored and uncolored images can cause significant difficulties for AI / ML models, potentially leading to poor performance. This is particularly problematic in the case of images of tissue samples, as, as briefly discussed above, tissue samples tend to have high-density small structures, which can cause significant spectral variations over very short distances that may correspond to only a few pixels. Therefore, in some embodiments, undergoing a second co-registration and training process may be advantageous. Thus, at box 1310, the digitally colored multiband data cube obtained at box 1308 can be co-registered with the colored multiband data cube to improve feature alignment between the two data cubes. The following will refer to... Figure 14 The co-registration is described in more detail. At box 1312, the AI / ML model undergoes a second training to obtain a multi-band data cube with digital coloring.

[0121] At box 1314, the training results can be evaluated by decompressing the digitally stained multiband data cube to obtain a digitally stained hyperspectral data cube. Decompressing the data cube may include generating spectra from DCT coefficients using DCT basis functions. At box 1316, the system can denormalize the digitally stained hyperspectral data cube and transform the data into an RGB representation. At box 1318, the system can transform the RGB representation to conform the images to approved histopathological standards.

[0122] Figure 14AI training processing 1400 according to some embodiments is illustrated. AI training processing 1400 can run on a computing system. In a first training block 1402, colored data cubes may be compressed at box 1406 and normalized at box 1408. At box 1410, uncolored data cubes may be compressed and normalized at box 1412. The compressed and normalized colored data cubes, as well as the compressed and normalized uncolored data cubes, may be fed into a first AI model for training and feedback at box 1414. In some embodiments, AI training and feedback 1414 may utilize a custom loss function. At circle 1, forward propagation and / or backpropagation may be used to adjust model parameters. The results of the first training block 1402 (e.g., digitally colored, compressed, normalized data cubes) may be used together with the compressed, normalized colored data cubes as input to a subpixel registration algorithm at box 1416. The output of subpixel registration box 1416 can be fed into a second training block 1404, which includes a second AI training and feedback box 1418. At circle 2, forward and / or backward propagation can be used to adjust the model parameters. In some embodiments, the AI ​​model at AI training and feedback box 1418 can be the same as the AI ​​model at AI training and feedback box 1414. For example, the subpixels learned in box 1416 can be used to re-register the normalized uncolored data cube generated at block 1412 with the normalized colored data cube generated at block 1408, or two different models can exist. See below. Figure 24 A more detailed explanation of similar training methods follows. In some embodiments, the following can be used: Figure 24 The training process handles some or all of the features.

[0123] In some embodiments, a training set can be collected by measuring pre-selected unstained tissue samples (e.g., from frozen sections or paraffin-embedded samples) under a microscope. The samples can then be chemically stained using multiple chemical baths based on the desired staining (e.g., pink staining with eosin for cells and tissue background, and deep blue-violet staining with hematoxylin for cell nuclei). The stained and unstained tissue samples can then be co-registered with high precision. For example, the co-registration error can be less than 25 μm, less than 10 μm, or less than 1 μm. Preferably, the co-registration error can be less than about 1 μm (e.g., about 10% of the size of a single nucleus). In some embodiments, mechanical co-registration (e.g., alignment of a slide on a microscope) may be sufficient. However, it may sometimes be advantageous to manually register images using algorithm-based automated registration software or by visual observation of tissue features. In some embodiments, reference markers can be introduced onto the slide substrate or another location and used for co-registration. For example, the reference may be etched onto the glass slide substrate or formed onto the glass slide substrate in other ways to ensure that the reference does not move or get washed away during the glass slide preparation step.

[0124] In some embodiments, lamp normalization can be performed on stained and unstained images. For example, minimum and maximum intensities can be normalized between the unstained and stained images. In some embodiments, instead of performing a single normalization applied to all wavelengths, normalization can be performed for each wavelength. Lamp normalization may not always be performed. However, lamp normalization can be particularly beneficial when capturing stained and unstained images using lamps with different emission spectra. For example, the emission spectrum of a lamp may change over time due to aging, variations in ambient temperature, the amount of time the lamp has been powered on, etc. In some embodiments, different lamps may be used for both unstained and stained images, for example, because the bulb is replaced.

[0125] In some embodiments, a multi-band data cube can be prepared from hyperspectral data by reducing the number of frequency bands. Depending on the desired quality, lossless or lossy compression algorithms can be used. Lossless compression produces an exact copy of the original image after decompression and reconstruction, while lossy compression can remove redundant spectral and spatial components with some information loss. Spatial redundancy may occur due to the similar intensity of adjacent pixels. Spectral redundancy may occur because nearby pixels have similar spectra. Temporal redundancy can also exist in hyperspectral image data. For example, this may occur if the same region is scanned more than once. Compression can be performed using transform-based algorithms, prediction-based algorithms, learning-based algorithms, vector quantization-based algorithms, compressed sensing-based algorithms, tensor decomposition-based algorithms, sparse representation-based algorithms, or multi-time-based algorithms. In some embodiments, principal component analysis (PCA) or maximum noise fraction (MNF) transform can be used. In some embodiments, discrete cosine transform can be used.

[0126] Compression can utilize spatial relationships, spectral relationships, or both present in the data. Therefore, spectral and spatial data can be reduced. In some embodiments, it may be preferable to reduce the number of spectral bands without compressing spatial information. For example, preserving spatial information may be important because the composition (and spectral properties) of a tissue sample can change rapidly. In some embodiments, a discrete cosine transform can be applied to the spectral bands of each pixel. The system can apply the discrete cosine transform to compress the original spectrum (which may have hundreds or more different wavelengths) using, for example, fewer than about 5, fewer than about 10, fewer than about 15, fewer than about 20, fewer than about 25, fewer than about 30, or (if desired) even more basis functions. Each basis function can have associated weighting coefficients. The spectrum can then be represented by the weighting coefficients of each basis function. Storage requirements can then be reduced by a factor of approximately N / L, where N is the number of basis functions and L is the number of data points in the spectrum. For example, Figure 15 Discrete cosine transform (DCT) fittings using 5, 10, and 25 basis functions are depicted. A larger number of basis functions can yield a better fit to the original data, but may require correspondingly more storage and computational resources. In some embodiments, the mean squared error when using DCT can be greater than the mean squared error when using other techniques such as principal component analysis. However, in some embodiments, using DCT remains preferred. Advantageously, DCT can use image-independent basis functions, and fast computation algorithms are known. By applying DCT (or another suitable transform) to each pixel in the hyperspectral data cube, the system can generate a multiband data cube comprising tensors with a reduced number of frequency bands.

[0127] Multiband data cubes can be normalized according to a defined scale to represent minimum and maximum intensity values. For example, the minimum value can be defined as 0 or -1, and the maximum value can be defined as 1. Applying this normalization to multiband data cubes can have many advantages for training AI / ML models. For example, many AI / ML algorithms prefer or require normalized input data. Linear regression, nonlinear regression, logistic regression, k-nearest neighbor, neural networks, clustering algorithms, and support vector machines can anticipate or require normalized input data.

[0128] Normalized data can be fed into an AI model for training to perform digit coloring. The AI / ML model can be, for example, a Conditional Generative Adversarial Network (GAN). In a GAN framework, a generator model learns mappings from training data cubes to perform digit coloring. A second discriminator network learns to distinguish between the generated image and ground-based data cubes (e.g., physically colored data cubes). Both networks can be trained simultaneously and conditioned on the colored data cubes. This training can be a first training iteration using data cube pairs corrected by a first registration. During training, the system can use forward and backward propagation, and the network parameters of the model can be updated during each iteration.

[0129] In some embodiments, the system can perform a second co-registration process using the results from a first training iteration and physically stained data cubes. Multiband data cube pairs (which may include, for example, stained data cubes and digitally stained data cubes) can be converted into RGB images by performing decompression, denormalization, and color transformation steps. The system can then compare the images, for example, by determining a Structural Similarity Index (SSIM) metric. Alternatively or alternatively, other measurements, such as mean squared error (MSE), peak signal-to-noise ratio (PSNR), and feature similarity indexing (FSIM), can also be used. The SSIM index (e.g., a quality assessment index) can be based on various terms, such as brightness, contrast, and structure. The total index can be a multiplicative combination of the terms. If the index is not within a threshold, the system can transform the pixels in the digitally stained RGB image by, for example, constructing a homography matrix for translating and / or rotating the image with small steps. In some embodiments, the system can also be configured to transform the digitally stained image by scaling the image, for example, based on the dilation or contraction of tissue samples as a result of physical staining.

[0130] In some embodiments, a digitally stained multiband data cube with co-registration can be used to perform additional training of a second AI model. In some embodiments, the second model can be similar to the first model. For example, in some embodiments, the second model can be identical to the first model except for one or more parameter values. For example, both can use a deep conditional generative adversarial network. In some embodiments, instead of training the second model, or in addition to training the second model, the co-registration information of the digitally stained multiband data cube can be applied to an unstained data cube, and the first AI model can be retrained using the corrected co-registration pairs. It should be understood that multiple models can be used in a cascaded configuration, or a single model can be used, or other configurations can be used during training or use.

[0131] Figure 16 A process 1600 for digital staining according to some embodiments is illustrated. Process 1600 can run on a computing system. At operation 1602, the raw hyperspectral data from an unstained tissue sample can be compressed, for example, by using discrete cosine transform or another suitable compression algorithm to form a multiband data cube. At operation 1604, the multiband data can be prepared for input into a machine learning system, for example by normalizing the data. At operation 1606, the system can apply a trained algorithm (e.g., according to...) Figure 3 The trained algorithm can generate a digitally stained image. At operation 1608, the system can denormalize the digitally stained image. At block 1610, the system can decompress the digitally stained image to form a hyperspectral digitally stained image, for example, by performing an inverse discrete cosine transform or other suitable inverse transform corresponding to the transform used to compress the unstained hyperspectral data cube at operation 1602. At operation 1612, the hyperspectral data cube can be transformed into an RGB image, etc. At operation 1614, one or more transforms can be applied to the RGB image. For example, in some embodiments, at operation 1614, the color, brightness, contrast, etc., of the image can be adjusted so that the digitally stained image looks like a physically stained image.

[0132] In some embodiments, denormalizing and transforming a hyperspectral image into an RGB image is complex. The conversion between hyperspectral and RGB data may not be direct, especially when the hyperspectral data contains information from outside the visible light range. Color matching functions can be applied to convert hyperspectral data into RGB data in, for example, the sRGB color space. In some embodiments, color matching functions may be provided by an external organization, such as the International Commission on Illumination (ICI). In some embodiments, color matching functions are linearly stretched or otherwise transformed such that they cover all wavelengths present in the hyperspectral data cube. In some embodiments, some wavelengths present in the hyperspectral data cube may be disregarded when generating an RGB representation of the data.

[0133] In some cases, additional transformations can be applied to adjust the colors of an image. For example, RGB images generated from hyperspectral data may deviate from "true" colors (e.g., those expected when performing physical staining) due to different processing steps, imaging conditions, operational inconsistencies, imperfections in spectral matching algorithms, distortions caused by using stretched color matching functions, etc. In some embodiments, histogram normalization can be used to correct the image. For example, a digitally stained image can be compared to an sRGB (or other color space) reference image of the chemical staining agent of interest by examining the histogram of a reference image and the histogram of the digitally stained image. However, in some embodiments, matching the global histogram of the reference image with the global histogram of the digitally stained image can be problematic because compositional variations may exist between the images, colors may be mismatched, and histological information may be lost.

[0134] To preserve histological information, more sophisticated color matching methods can be used. For example, the intensity of the background or blank areas in a histopathological image does not contain any staining agent. Therefore, spectral variations in blank areas are associated with luminescent differences (e.g., the difference in emission spectra of the lamps used for the reference image and those used for the digital staining image). For example, luminescent differences can be addressed by applying a correction factor involving the ratio of the two background spectra. For example, if... It is the spectral waveform of the background region of the reference image. If the spectral waveform of the background region of a digitally stained image is given, then it can be determined according to the relationship... To correct the pixels of a digitally stained image, where It is the corrected spectrum, and This is the uncorrected spectrum (e.g., the raw, captured waveform of the background region in a digitally stained image). It should be understood that this transformation can be applied to each wavelength of each pixel in the digitally stained image. Figure 17Example spectra without luminescent correction (left) and example spectra with luminescent correction (right) are depicted compared to chemically stained images.

[0135] Digital staining based on absorption

[0136] The embodiments described above typically refer to transmission spectroscopy. However, transmission spectroscopy is not always necessary. For example, absorption, reflection, or both can be used instead of transmission or in addition to transmission. In some embodiments, it may be advantageous to use absorption spectroscopy to perform digital staining or other processing of medical imaging or other spectral data. According to the Beer-Lambert-Bouguer law (hereinafter referred to as Beer's Law), the transmission and absorption of light can be correlated with each other. In particular, Beer's Law defines the relationship between absorbed and transmitted light as... Where A is the absorption of the sample, and T is the transmission through the sample. Both A and T depend on the wavelength. Absorption A in relation to sample thickness d and absorption coefficient Related, the relationship is The relationship between absorption and transmission ignores any deviations in light scattering or absorption, refractive index, etc., that may become significant under certain conditions. This relationship also ignores any inhomogeneities, such as those found in glass slides or coverslips used in preparing tissue samples. In some embodiments, despite these limitations, the absorption spectrum determined from transmission data can be used for digital staining.

[0137] Typically, a sample can contain multiple absorbers. For example, in a tissue sample, many different compounds may be present at varying concentrations throughout the sample. When N absorbers are present, absorption A can be described as... Absorption coefficient It can depend on the concentration of the absorber and the wavelength-dependent absorptivity (or extinction coefficient) of the absorber.

[0138] In some cases, it may be advantageous to convert the collected transmission data into absorption data and use the absorption data to perform machine learning training or other analyses. Figure 18 An example of the transformation between transmission and absorption is shown. Curve 1802 shows the transmission spectrum of a single pixel from a stained tissue sample. Curve 1804 shows the corresponding absorption spectrum of the same pixel.

[0139] Model training

[0140] Figure 19Example processing for training machine learning (ML) or artificial intelligence (AI) models using absorbance data, according to some embodiments, is described. This processing can run on a computing system. At operation 1902, the system can generate a co-registered, lamp-normalized data cube pair from unstained and stained images of the same tissue sample, which may contain transmission data. At operation 1904, the system can convert the transmission data into absorbance data to produce absorbance data cube pairs. At operation 1906, the system can optionally compress the absorbance data cube pairs, for example, by applying a discrete cosine transform or other suitable transform, to generate multi-band data cubes with a reduced number of frequency bands. In some embodiments, the raw data (e.g., uncompressed hyperspectral data) can be used to train the AI / ML model. However, in some embodiments, data compression may be advantageous, for example, to reduce computational requirements.

[0141] At operation 1908, the system can normalize the multi-band data cube pairs to prepare them for training the AI / ML model. For example, the system can scale the values ​​within the data cube pairs such that the maximum value is 1 and the minimum value is 0 or -1. At operation 1910, the system can perform a first pass of training the AI / ML model to obtain digitally colored multi-band data cubes from uncolored multi-band data cubes. At operation 1912, the system can perform an additional registration step. For example, the system can apply an automatic registration algorithm to co-register the digitally colored multi-band data cubes and the colored multi-band data cubes to further improve the registration between the digitally colored and colored data cubes. As mentioned above, co-registration can include translation, rotation, stretching, compression, etc. In some embodiments, constraints can be imposed on the co-registration process, such as limiting the loss of active tissue fields near the boundaries that may occur when translation and / or rotation are relatively large. At operation 1914, the system can perform a second pass of training the AI / ML model to improve the digitally colored multi-band data cubes. In some embodiments, a third, fourth, or more passes can be performed. In some embodiments, instead of a second training pass of the same model, a different model can be trained at operation 1914.

[0142] In some embodiments, the absorption data can be used to perform registration at operation 1912. For example, the output of operation 1910 may be a digital staining absorption image, which can be co-registered with an absorption spectrum derived from, for example, transmission data of a staining image produced at operation 1904 and / or an absorption spectrum derived from compressed absorption data produced at operation 1906. For example, if the digital staining image is decompressed, the digital staining image can be compared with the original absorption data, or if the digital staining image is compressed, the digital staining image can be compared with the compressed absorption data.

[0143] In some embodiments, the additional registration at operation 1912 includes converting the digitally stained multiband absorption data cube into a transmission data cube. The system can decompress the digitally stained multiband transmission data cube, denormalize the digitally stained multiband transmission data cube, and transform the digitally stained multiband transmission data cube and the stained transmission data cube into corresponding RGB images (e.g., instead of having a spectrum associated with each pixel, each pixel may have red, green, and blue values). A structural similarity index metric (SSIM) can be used to compare the RGB images. SSIM can be used to measure the similarity between digitally stained RGB images and chemically stained RGB images. If the structural similarity index is not within a threshold, the system can transform the digitally stained RGB image by constructing a homography matrix for transforming and rotating the digitally stained RGB image. Small translations and / or rotations of the digitally stained RGB image can be performed until the desired similarity index value is achieved, until the similarity index converges to a value (e.g., when additional iterations do not change the similarity index by more than a threshold amount), until the maximum number of iterations has been performed, etc. The correction resulting from the registration at operation 1912 can be used to co-register the digitally stained absorption multiband data cube with the stained multiband data absorption cube. Alternatively or additionally, the correction resulting from the registration at operation 1912 can be applied to the unstained absorption multiband data cube. In some embodiments, co-registration can be performed using only luminance or intensity data. In some embodiments, co-registration can be performed using data from all channels of the image.

[0144] The AI ​​models trained at operation 1910 and operation 1914 (which may be the same model or different models) may be deep conditional generative adversarial networks. In some embodiments, if multiple models are trained, they may operate in a cascaded configuration when used. The above processing is described for a single data cube pair. However, it should be understood that multiple data cube pairs may be used during AI / ML training processing.

[0145] Various methods can be used to train AI / ML models. For example, see the reference above. Figure 3 The AI / ML model is trained as described. In some embodiments, the AI / ML model can be trained using pairs of stained and unstained images.

[0146] Model Deployment

[0147] Figure 20An example processing step for digitally colorizing an image, according to some embodiments, is described, which can be implemented on a computing system. At operation 2002, the system may receive a raw transmission data cube, which may be, for example, a hyperspectral image, multispectral image, fluorescence image, or other image or similar data. At operation 2004, the system may convert the received raw transmission data cube into an absorption data cube. At operation 2006, the system may compress the absorption data cube, for example, by applying a discrete cosine transform or other transform. At operation 2008, the system may normalize the absorption data cube to prepare it for digital colorizing the compressed absorption data cube by applying a trained AI / ML algorithm at step 2010. At operation 2012, the system may denormalize the digitally colorized absorption data cube. At operation 2014, the system may decompress the compressed digitally colorized absorption data cube. At operation 2016, the system may convert the absorption data cube into a digitally colorized transmission data cube. At operation 2018, the system may transform the digitally colorized transmission data into an RGB representation. At operation 2020, the system can perform one or more color matching operations. For example, the system can match colors in a digitally stained RGB image with a standard for a specific stain (e.g., a histopathologically recognized standard or a reference RGB image).

[0148] Although Figure 20 The process shown is illustrated as converting a single, complete uncolored data cube into a colored data cube; however, in some embodiments, it can be applied in real-time, near real-time, or before the uncolored data cube is fully captured. Figure 20 The process is illustrated. For example, the system can be connected to a microscope and configured to digitally stain the image during capture by, for example, digital staining upon receiving each voxel, digital staining upon receiving each row in the image, digital staining after receiving a set of rows (e.g., performing digital staining after every five rows, after every ten rows, etc.). Using this process allows pathologists and others to make decisions before image capture is complete. For example, a pathologist might decide to apply different digital stains to prepare additional tissue sections based on digitally stained portions of the image, etc.

[0149] Multiple staining

[0150] As briefly discussed above, in some cases, pathologists may need to apply multiple staining agents to tissue samples. For routine chemical staining, this may require preparing multiple tissue samples. Digital staining allows multiple staining agents to be applied to a single tissue sample, thereby increasing efficiency and potentially leading to improved diagnosis. In addition to this increased efficiency, digital staining enables imaging methods that are difficult or impossible to achieve with chemical staining. Digital staining allows multiple staining agents to be applied simultaneously to the same image. That is, digital staining makes it possible to produce synthetic images by using a mixture of multiple staining agent combinations in different regions. For example, within a single image, one region may be treated with one digital staining agent, while another region may be treated with a different digital staining agent. This multiplexing enables more accurate diagnosis.

[0151] Figure 21 An example of multiple images 2100 according to some embodiments is shown. Figure 21 The image shows a tissue field stained with hematoxylin and eosin (H&E). The first region 2102 can be digitally stained with different staining agents (such as Jones-Urotropine-Silver (JMS)), and the second region 2104 can be stained with another dye (such as, for example, Masson's trichrome stain (MT)). H&E staining facilitates the differentiation of cell nuclei and cytoplasm, while JMS provides contrast with the basement membrane, and MT can be used to differentiate collagen and smooth muscle in tumors. By simultaneously applying multiple staining agents to different regions of the same image, pathologists are better able to make diagnoses.

[0152] Figure 22 This is a block diagram illustrating a process 2200 for multiple digital staining according to some embodiments. Process 2200 can be executed on, for example, a computing system such as a desktop computer, laptop computer, tablet computer, smartphone, or other suitable device. At block 2202, the system can load an unstained image, which can be, for example, a hyperspectral image, a multispectral image, etc. At block 2204, the system can receive selections of one or more (N) regions of interest for digital staining. In some embodiments, a user can manually select regions for staining, for example, by drawing the regions using an input device such as a mouse. In some embodiments, the system can automatically select regions. For example, the system can detect edges or changes in spectral data in the image, which can allow the system to automatically detect regions or assist the user in selecting regions. In some embodiments, staining agents can be applied to any unselected regions. For example, the system can be configured to apply hematoxylin and eosin staining agents to any unselected regions.

[0153] At box 2206, the system can receive one or more (Mi) staining agent selections for each of N selected regions of interest. Different staining agents can be selected for different regions, or the same staining agent can be selected for different regions. In some embodiments, the user can select more than one staining agent for a selected region. At box 2208, the system can extract the pixel coordinates of the i-th region of interest, and at box 2210, the system can extract the unstained spectral values ​​of the i-th region of interest. At box 2212, the system can apply the selected staining agent to the selected region. The system can repeat box 2212 to apply Mi different digital staining agents to the i-th region of interest. The system can repeat boxes 2208, 2210, and 2212 for each i-th region of interest among the N regions of interest. At box 2214, the digitally stained regions of interest and unselected regions (which can be unstained or digitally stained) can be combined by the system to form one or more final composite images.

[0154] In some embodiments, the system may allow a user to visualize multiple stains for a single region of interest. For example, the system may present an interface to the user, allowing them to select a region in the final composite image and choose different digital stains to be applied to that region. For instance, a pathologist may click, flip, scroll, or otherwise navigate through multiple stains for a specific region of interest and select the stain that provides the most diagnostic information, which can lead to improved diagnosis, reduced analysis time, and so on.

[0155] Digital coloring using RGB image data

[0156] Some embodiments described herein depict systems and methods for processing complex medical imaging and spectral data. In some embodiments, these methods can be used for less complex data, such as RGB images. However, it is beneficial to provide systems and methods optimized for processing specific types of data. For example, an RGB image may contain only three channels (e.g., red, green, and blue channels), while a hyperspectral or multispectral image may contain many channels. Working with RGB images, or in addition to hyperspectral or multispectral images, can have many advantages. For example, not all practitioners have access to specialized equipment for capturing hyperspectral or multispectral images. Capturing complex imaging data can be time-consuming compared to capturing RGB microscopic images, as complex imaging data is typically captured by collecting information across different wavelength ranges at different times.

[0157] The use of RGB images can alleviate some of the problems associated with other forms of imaging because it eliminates the need for equipment that is not easily accessible to practitioners and allows for rapid image capture using common microscopy equipment. Furthermore, the use of RGB images simplifies image processing steps performed as part of digital staining. For example, RGB images can be relatively small compared to hyperspectral or multispectral images, and they can include only three channels instead of tens, hundreds, or even thousands. Therefore, image compression or downsampling may not be necessary to achieve acceptable performance in digital staining systems, saving storage space, etc.

[0158] In some cases, digital staining can be performed using only a limited number of channels from hyperspectral or multispectral images. For example, digital staining can be performed using ten channels, five channels, or three channels. Those skilled in the art will understand that while the number of channels can be reduced, specific channels (e.g., wavelength ranges) generally cannot be arbitrarily chosen because sufficient information (e.g., transmission peaks / valleys) is required for the digital staining model to function. The number and location of channels on the electromagnetic spectrum (e.g., wavelength range) can depend on the specific digital staining agent to be applied. For some staining agents, sufficient information is available in the visible spectrum for an RGB camera to capture the image. For example, for hematoxylin and eosin, information in the visible region of the electromagnetic spectrum provides sufficient information for accurate digital staining. Other digital staining agents, whether based on physical or chemical staining agents or lacking physical world analogues, can be applied to RGB images.

[0159] In some embodiments, a color camera using a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor can be used to capture images suitable for digital staining. The sensor can be divided into multiple pixels and / or subpixels. Bayer filters can be used to select the wavelength range to be captured by each subpixel. For example, a Bayer filter may include an array of red, green, and blue filters. The filters allow red, green, or blue light to pass through the filters to reach the sensor while blocking other light. Advantageously, this approach enables simultaneous capture of all three channels (red, green, and blue), thereby reducing capture time. In some embodiments, an RGB camera can be configured to simultaneously capture an image of the entire tissue sample (e.g., compared to using a scanning process). For example, the camera may have lenses and apertures configured to immediately receive light from the entire region of interest and project the light onto the sensor. In some embodiments, image distortion may be less when using an RGB camera because the shorter capture time limits the effects of any vibrations that could cause image distortion. Furthermore, hyperspectral and multispectral imaging systems typically include moving parts (e.g., for filter replacement), and vibrations of these parts can also cause image distortion.

[0160] In some embodiments, the camera may be a hyperspectral camera or a multispectral camera that provides only RGB output. In such a camera, red, green, and blue channels can be constructed by combining several hyperspectral bands with appropriate red, green, and blue filters. In some embodiments, the output of the hyperspectral camera or other camera may be standard RGB values ​​or values ​​in another known color space, such as XYZ, LUV, LCh, etc. RGB, YCbCr, YUV, and CMY are color spaces that can be obtained by applying appropriate transformations. The following describes an example of conversion between RGB and YCbCr.

[0161] Although the following discussion pertains to RGB image data, it will be understood that the systems and methods presented herein are readily applicable to other three-parameter color spaces (e.g., XYZ, LUV, LCh, ...). (YCbCr, YUV, CMY). The system and method described in this paper can also be applied to four-parameter color spaces, such as CMYK, RGBD (where D represents the fourth channel or parameter), or any other arbitrary ABCD color space.

[0162] Artificial intelligence or machine learning (AI / ML) models can be trained to digitally stain unstained RGB images of tissue samples. In some embodiments, a single-level model can be used. However, in some embodiments, a multi-level model can be advantageous for digital staining. As briefly mentioned above and described in more detail below, tissue sample images can contain densely packed features (e.g., cells, nuclei, lipids, extracellular matrix, cytoplasm, etc.). Therefore, it is important to accurately represent small features, perhaps spanning only a few pixels to about 100 pixels, in the digitally stained image. This can be challenging when using a single-level model because such a model can be tuned to recognize low-frequency features (e.g., gradual changes in color or brightness) but cannot be tuned to recognize high-frequency features (e.g., the edges of cell nuclei). Conversely, a model can be tuned to recognize high-frequency features but may perform relatively poorly in recognizing low-frequency features. Therefore, in some embodiments, using a multi-level model can be advantageous. For example, a first level can be tuned to recognize low-frequency features, and a second level can be tuned to recognize high-frequency features. In some embodiments, the first stage can be tuned to identify high-frequency features, and the second stage can be tuned to identify low-frequency features. In this configuration, in some embodiments, the second stage may benefit from the first stage's recognition of edges and other features. In some embodiments, as briefly described above, a GAN network can be used. For example, the generative part of the network can digitally colorize an uncolored image, and the adversarial part of the network can attempt to distinguish between colored and digitally colored images. Ideally, the adversarial network cannot accurately guess which image has been digitally colored. For example, the correct guessing rate should preferably be close to random (e.g., close to 50%, such as within 1%, within 2%, within 3%, within 4%, within 5%, within 6%, within 7%, within 8%, within 9%, within 10%, etc.).

[0163] As described in this paper, the features in tissue sample images can be small compared to the overall image size. For example, in some cases, cell nuclei may only be tens of pixels to about 100 pixels. When training an AI / ML model, image pairs of stained and unstained tissue can be used as training data. For example, tissue slides can be prepared, and images of unstained tissue can be taken using an RGB camera. The tissue can then be stained (e.g., using H&E or trichrome staining, or using specialized staining agents such as high molecular weight keratin (HMWK), Grocott hexamethylenetetramine silver (GMS), etc.) and imaged again using an RGB camera. These two images can form an image pair, and the goal of the AI / ML model can be to manipulate the unstained image to make it very similar to the stained image. If the images are not properly aligned with each other (co-registration), training may fail, causing the model to be unable to accurately perform digital staining on new unstained images that were not used during training.

[0164] For example, registration problems can be partially mitigated by aligning the edges of the tissue sample. However, variations in the tissue sample caused by staining can make co-registration difficult. For example, some areas of the tissue sample may swell due to staining, while other areas may shrink. Therefore, in some embodiments, the stained image, the unstained image, or both may be translated, rotated, scaled, or otherwise deformed to co-register the images. In some embodiments, the deformation may be uniform across the image. However, in some embodiments, the registration adjustment may vary across the entire image. For example, some areas may be made smaller, while other areas may be made larger, for example, to accommodate elastic deformation caused by staining. For example, within a tissue sample, some areas may swell due to staining, while other areas may shrink.

[0165] As described in more detail below, in some embodiments, generating an AI / ML model may include multiple registration and training steps. For example, after performing digital staining, co-registration may be performed between the digitally stained image and the stained image, and the model may undergo additional training.

[0166] Figure 23 Example processing for training and generation according to some embodiments is shown. The preprocessed RGB image pair 2302 may undergo multiple passes of multi-donor AI processing at box 2304. The output of the multiple passes of multi-donor AI processing may be a digitally tinted image. At box 2306, image processing such as normalization and color correction may be performed to produce a normalized RGB image 2308, which may be, for example, a digitally tinted image. This will be explained in more detail below. Figure 23 The steps shown are as follows.

[0167] When preparing image pairs for processing (e.g., unstained and stained images of the same tissue), the image pairs can be co-registered to ensure that features in the unstained image match features in the stained image. Registration can be particularly important because, as mentioned above, features in, for example, tissue sample slides can be very small. For example, without a registration step, the error can typically be in the range of approximately 100 pixels. A cell nucleus can be about 100 pixels in size, depending on the magnification of the microscope-captured image. Therefore, a typical registration error can include a significant portion (in some cases, essentially the entire feature) of the feature in the image. In some embodiments, the system can be configured to perform registration by, for example, positioning one or more references present on the microscope slide. Although in Figure 23 The image shows a single preprocessed image pair, but it will be understood that multiple images can actually be provided to the AI / ML model. In some embodiments, image pairs from multiple donors can be provided to the AI ​​model. Providing training images from multiple donors can be important because, for example, while similar tissues (e.g., liver tissue) may be similar between donors, there may be variations, some of which may be unexplained, and some of which may be due to various donor factors such as sex, age, ethnicity, etc. Training the AI / ML model using images from a variety of donors makes the AI / ML model more generalizable.

[0168] Figure 23 The use of a multi-pass algorithm is illustrated. While this approach, which includes multiple passes and / or multiple registration steps, can be advantageous, in some embodiments, single-pass AI / ML training can be used. For example, if the input training image pairs are well registered, single-pass training can be used to produce appropriate digital coloring results.

[0169] like Figure 23 As shown, after image generation, the resulting digitally stained image can undergo post-processing, as described in more detail below. This can be important because, for example, while AI / ML models can produce digitally stained images of sufficiently high quality in terms of accurately representing features, the colors in digitally stained images may differ from those in chemically or physically stained images. For example, the pink hue may be too bright or too dark, or the color may appear faded. This can make digitally stained images difficult to interpret because practitioners are accustomed to observing chemically or physically stained tissues and may therefore expect specific colors when observing digitally stained images.

[0170] Model structure and training

[0171] In some embodiments, the AI / ML model (also referred to herein as a digital coloring model) may include a convolutional neural network (CNN). In some embodiments, the CNN may include multiple neurons. In some embodiments, each neuron may be responsible for a local receptive field encompassing a defined region of the image. The receptive field may be a defining portion of the image, for example, defined by the filter size of a layer within the CNN. The receptive field may indicate the selection of input data exposed to the neuron (or other units within a CNN layer). In some embodiments, the AI / ML model may include multiple layers. In some embodiments, earlier layers in the CNN network (e.g., the first layer in the CNN) may be associated with relatively small receptive fields. This approach can help enable the learning of features such as lines, edges, and other details that constitute the image. In some embodiments, higher layers in the CNN may be exposed to larger receptive fields because smaller receptive fields in lower layers combine to form a larger receptive field.

[0172] In some embodiments, single-layer, multi-layer, and / or multi-level AI / ML networks can be used to monitor the training process. For example, the AI / ML network can compare the features of the generated digital staining image with the features of the chemical staining image. The compared features can be used to further adjust the weights in the digital staining model.

[0173] In some embodiments, an image can be divided into multiple fields. For example, an image can be divided into subfields comprising multiple pixels. Larger subfields can provide improved results, but may require more computational resources than using smaller fields. Advantageously, if the image is divided into multiple subfields, there can be overlap between the subfields. For example, the first and second subfields may both contain some pixels from the entire image, and each subfield may also contain pixels not present in the other. In some embodiments, this method can improve the performance and / or training speed of AI / ML models. For example, it may be beneficial to train the model based on a mixture of previously seen information and new information. In some embodiments, the subfields can be square. For example, subfields can be 16×16 pixels, 32×32 pixels, 64×64 pixels, 96×96 pixels, 128×128 pixels, 256×256 pixels, 512×512 pixels, etc. In some embodiments, the subfields do not have to be square.

[0174] In some embodiments, a CNN may include multiple levels. In some embodiments, each level may include one or more layers. In some embodiments, the first level may be trained to detect relatively low-density structures. In some applications, a single level may be suitable for image processing, but may not produce suitable results in other applications. For example, an image of a park may be characterized by green lawns, blue skies, and some trees. The colors of the sky, lawn, trees, etc., may change slightly throughout the image, but such changes are likely to be relatively slow and smooth. For example, the color of the sky may change from a darker blue to a lighter blue, but it is unlikely to change from one color rapidly to another. Conversely, for example, due to the cellular nature of tissues, medical images of tissue samples often vary significantly even within a few pixels. A single level may recognize the general structure of a medical image, but may not be able to recognize enough structural detail to produce a convincing, accurate digital staining image.

[0175] As described above, in some embodiments, the AI / ML model may include, for example, a Conditional Generative Adversarial Network (GAN). In a GAN framework, a generator model learns mappings from training data cubes to perform digital coloring. A second discriminator network can learn to distinguish between the generated image and ground-based data cubes (e.g., physically colored RGB images). Both networks can be trained simultaneously. This training may be a first training using a pair of data cubes with a first common registration. During training, the system may use forward and / or backward propagation, and the network parameters of the model may be updated during each iteration.

[0176] Figure 24 This is a schematic diagram illustrating example embodiments for training an AI / ML model to digitally stain RGB images, according to some embodiments. Reference is made to the image pair below. It should be understood that the training model can be designed using many images of stained and unstained tissue samples. Therefore, the following description is not limited to training using a single stained tissue image and a corresponding unstained tissue image. As described above, a training dataset can be created by imaging a collection of unstained tissue samples, staining the samples, and then imaging the samples again. The stained and unstained images can then be co-registered using manual alignment and / or algorithm-based automatic registration as described above. In some embodiments, light normalization can be performed on the stained and unstained images as described above. However, in some embodiments, such light normalization may not be included.

[0177] Training can be performed on a computer system. For example... Figure 24As shown, at box 2404, the system can scale and normalize the uncolored RGB image 2402. At box 2410, the system can scale and normalize the colored RGB image 2408. Together, the scaled and normalized input uncolored image and the scaled and normalized input colored image can be a co-registered image pair. In some embodiments, scaling and normalization may include applying a denoising operation to either the colored or uncolored image. At box 2406, the system can digitally colored the scaled and normalized uncolored RGB image 2402. After digital coloring at box 2406, the system can compare the output of box 2406 (e.g., the digitally colored image) with the scaled and normalized colored RGB image. At box 2412, the system can compute a loss function. As described herein, in some embodiments, the loss function may take into account local and spatial information. Using spatial information to compute the loss function can help reduce the likelihood that the training process reaches a local rather than a global minimum. For example, the loss function may include L1 loss (e.g., the absolute value of the prediction (e.g., the pixel intensity and / or features of the digitally stained image obtained from the individual layers / levels of the AI / ML model) compared to the baseline facts (e.g., pixel intensities and / or features of the physically or chemically stained image obtained from the individual layers / levels of the AI / ML model)) and / or L2 loss (e.g., the squared difference between the prediction and the baseline facts). As described in more detail below, the training data may be divided into multiple subfields. There may be many subfields in the training dataset. For example, there may be tens, hundreds, thousands, or millions of subfields. In some embodiments, the loss function may be computed using a subset of unstained image data and digitally stained image data. The output of the loss function at box 2412 may be processed at circle 1 and used to adjust one or more weights of the model. Training may continue using the input co-registered images, and the loss function may be computed again at box 2412. For each iteration, the same subfield may be used or, advantageously, a different set of subfields may be used to compute the loss function. Processing may continue to find the minimum of the loss function. Using this feedback mechanism, training can continue until the loss function output indicates that the difference between the digitally stained and unstained images is within a threshold amount. As described in this paper, forward propagation and / or backward propagation can be used to train AI / ML models.

[0178] After exiting the first training loop, a second registration process can be performed at box 2414. In some embodiments, the second registration process can compare the digitally tinted image with the tinted image. In some embodiments, this comparison can be used (e.g., using one or more of translation, rotation, distortion, deformation, or scaling) to adjust one or more of the untinted and tinted images. The output of the subpixel registration at box 2414 can be an updated co-registered image pair created from the untinted RGB image data and the tinted RGB image data. Adjustments can be performed as described elsewhere in this application, for example using single-level and / or multi-level structural similarity index measures. In some embodiments, the structural similarity index measure can improve from about 0% to about 25% or more, for example, about 8%. In some embodiments, the translation error can be from about 0 pixels to about 20 pixels or more, for example, from about 2 pixels to about 5 pixels. In some embodiments, the rotation error can be from about 0 pixels to about 10 pixels or more. In some embodiments, the dilation / shrinkage error can be from about 0 pixels to about 10 pixels or more, for example, from about 1 pixel to about 5 pixels. The training process can then proceed to the second training loop. At box 2416, the system can perform a second training process using the updated co-registration pair generated at box 2414. The system can perform additional training on the model trained at box 2406, or it can train a model with a different AI / ML network architecture. At box 2418, a loss function can be computed by comparing the output of box 2416 (e.g., a digitally tinted image) with tinted image data (e.g., a scaled / distorted and normalized tinted RGB image, to which one or more transformations can be applied in some embodiments, e.g., depending on which tinted or untinted image pair was used for subpixel registration at box 2414). At circle 2, the output of the loss function can be used to adjust the weights of one or more models. Taking into account portions of the tinted image data and the digitally tinted image data, the training process can operate as described above. The training process can continue until the output of the loss function is within a threshold range, which can be the same as or different from the threshold used to exit the first training loop.

[0179] Although Figure 24 The first and second training cycles are shown, but it should be understood that additional cycles can be used. For example, after minimizing the loss function in the second training cycle, a third co-registration pair can be created and used in the third training cycle.

[0180] As described above, a single model can be generated that takes an uncolored image as input and produces a digitally colored image as output. However, in some implementations, instead of using colored and uncolored image pairs for all training steps, the training process can use digitally colored and colored images for some steps. For example, Figure 24 The second training cycle can be trained using stained image data and digitized stained image data, instead of stained and unstained image data. This approach can produce cascaded composite models comprising multiple models, where each model at least partially operates on the output of the preceding model in the cascade.

[0181] The preceding description discussed using models to color RGB images. However, it will be understood that any three-parameter or four-parameter color space can be used. The above description can also be readily applied to other forms of imaging or spectral data, such as hyperspectral imaging data or multispectral imaging.

[0182] Color Correction

[0183] As mentioned above, while AI / ML models can generate digitally stained images of sufficiently high quality in terms of accurate feature representation, the colors of these images may differ from those of conventionally stained tissue images, which can make interpretation challenging. Therefore, in some embodiments, color correction processing can be applied to digitally stained images so that they have colors that match those expected using conventional staining processes.

[0184] Figure 25An example processing for correcting the color of a digitally stained image is illustrated according to some embodiments. A reference RGB image 2502 can be used as the basis for adjusting an input RGB image 2504. At box 2506, the system can convert the reference RGB image 2502 into a YCbCr image. At box 2508, the system can determine the mean and standard deviation of each channel of the YCbCr image (e.g., the mean and standard deviation of Y, Cb, and Cr). At box 2510, the system can convert the input RGB image 2504 into a YCbCr image. At box 2512, the system can determine the mean and standard deviation of each channel of the YCbCr input image (e.g., the standard deviations of Y, Cb, and Cr). At box 2514, the system can determine the Y, Cb, and Cr values ​​for each pixel in the input YCbCr image. At box 2516, the system can determine the distance of each channel of each pixel from the average value of that channel (e.g., pixel (5, 7) may have a Y standard deviation of 0.3 from the average value of the Y channel, a Cb deviation of -0.1 from the average value of the Cb channel, and a standard deviation of 1.2 from the average value of the Cr channel). At box 2518, the system can scale the value of each channel of each pixel of the YCbCr input image using the average value and standard deviation of each channel of a reference YCbCr image, thereby producing a normalized YCbCr input image. Then, at box 2520, the normalized YCbCr input image can be transformed into RGB, thereby producing a normalized RGB image 2522.

[0185] The reference image can preferably have the same tissue type as the input image. However, according to some embodiments, the reference image does not necessarily have to be of the same tissue type. The reference image can be an image of a conventionally stained tissue sample that has already been stained using the same staining agent digitally applied to the input image. For example, when applying H&E digital staining agent, the reference image can be an image of a tissue sample that has already been stained with H&E.

[0186] There are various methods for scaling an input image based on a reference image. In some embodiments, color correction can be performed using RGB data. However, in some embodiments, it may be advantageous to perform the correction in another color space (e.g., HSV, HSL, YCbCr, etc.) that separates luminance information from color information. According to some embodiments, the input RGB image and the reference image can be converted using the following relationship. Then, the YCbCr values ​​of the input image can be scaled according to the following relationship: , where x is the YCbCr value of the input image, and t is the YCbCr value of the reference image. t represents the standard deviation. x can be the Y, Cb, or Cr value obtained from the YCbCr representation of the input image. t can be the Y, Cb, or Cr value obtained from the YCbCr representation of the reference image. Applying the scaling relation transforms the input Y, Cb, and Cr values ​​from the YCbCr representation of the input image into scaled values ​​Y. out Cb out and Cr out Then, a normalized RGB image can be generated using the following relation: .

[0187] Figure 26 An example of a color-corrected image according to some embodiments is shown. The color-corrected image 2603 can be generated using the digitally tinted image 2602 and the reference image 2601. For example... Figure 26 As shown, the color of the digitally stained image 2602 can appear very different from the color of the reference image 2601, while after color correction processing, the reference image 2601 and the color-corrected image 2603 can have very similar colorization.

[0188] Precision surgery

[0189] In many cases, tissue samples are collected from the patient and sent to a pathology laboratory for analysis, with the patient following up with their doctor. Turnover time can be measured in hours, days, or even weeks, and the patient can follow up with their doctor once the results are available. However, histopathology can also be used during certain types of surgery. For example, the Mozart procedure is a surgical technique commonly used to treat certain skin cancers. The Mozart procedure may involve removing certain tissues (such as cancer cells) while minimizing the removal of unrelated healthy tissue. During the Mozart procedure, tissue can be removed in small layers and analyzed under a microscope to determine if more tissue needs to be removed, and if so, the tissue is removed from where it is needed. Because tissue analysis usually occurs while the patient is waiting, the patient can be treated relatively quickly. As a result, many procedures can be completed within a day.

[0190] The Mohr's procedure has been proven effective in treating a variety of cancers. For example, some studies have found a cure rate of approximately 97% or higher for basal cell carcinoma. The Mohr's procedure is also used to treat squamous cell carcinoma, melanoma in situ, certain other types of melanoma, dermatofibrosarcoma protuberans, keratoacanthoma, spindle cell tumor, sebaceous gland carcinoma, microcystic adnexal carcinoma, Merkel cell carcinoma, breast Parkette's disease, atypical fibroxanoma, and leiomyosarcoma. Because the Mohr's procedure involves precise tissue removal, the amount of tissue removed can be reduced, which can shorten healing time, lower the risk of infection, and improve aesthetic outcomes (e.g., less scarring). For example, the margin of the removed healthy tissue can be as small as approximately 1 mm compared to other procedures where the margin can be 5 mm or larger. The Mohr's procedure is often used to treat areas where the preservation of healthy tissue is of great importance, such as the face and genitals.

[0191] During Mozart procedures, frozen tissue samples are typically used, which reduces sample preparation time compared to formalin-fixed, paraffin-embedded samples. However, the typical time for preparing and analyzing tissue samples ranges from approximately twenty minutes to approximately one hour. During this time, the patient may wait in a waiting room, operating room, etc. In some procedures, multiple rounds of tissue removal and analysis may occur. Therefore, in some cases, waiting times and total procedure time may span several hours. Some procedures may be divided into multiple sessions over a period of more than one day. While frozen tissue samples are generally used in procedures where pathological results are urgently needed (e.g., because the patient is currently undergoing surgery), it will be understood that the systems and methods described herein can be used with any tissue sample preparation method. Furthermore, frozen tissue samples may present various problems that can complicate analysis. For example, ice crystal formation can expand and disrupt cellular structures, rapid freezing can damage tissue structures, leading to tearing or folding, ice crystal and structural damage can result in uneven staining, and freezing artifacts can reduce the overall clarity of tissue sections. Although various mitigation techniques exist, such as rapid freezing and the use of cryoprotectants, which can reduce artifacts and problems seen in frozen tissue samples, such samples can still provide worse results than formalin-fixed paraffin-embedded samples.

[0192] In a typical procedure, a scalloped section of tissue is removed from the patient. The removed tissue is then frozen, sectioned, and stained. It may be important to obtain a complete view of the sample edges (e.g., margins), as the presence of cancer cells at or near the tissue edges may indicate that not all cancer cells have been removed from the patient, and thus warrant further tissue removal. In some cases, a physician, pathologist, pathologist, or other medical professional may make one or more relaxed cuts in the tissue so that the edges are visible when preparing a slide of the tissue sample; however, this may only partially address the difficulties associated with examining tissue edges. Tracking the orientation of the tissue can be important. For example, it is important to know not only whether cancer cells are present at the edges, but also where they are located, so that additional tissue can be removed from the patient in a targeted manner while minimizing the removal of uninvolved (e.g., healthy) tissue. In some embodiments, the tissue and / or slide may be marked with one or more dyes or stains to help identify the orientation of the tissue. In some embodiments, markings, dots, etc., may be made in the tissue or drawn on the slide to help indicate the orientation of the tissue. In some embodiments, medical professionals may add barcodes, QR codes, or other machine-readable codes to a slide so that it can be associated with a specific patient.

[0193] Current methods for performing Moscone or other surgical procedures that analyze tissue samples while patients are waiting have significant drawbacks. For example, H&E staining is often used to stain tissue samples to help differentiate between healthy and cancerous tissue. However, the staining process can be time-consuming, during which patients may be waiting with open wounds. Prolonged waiting times can lead to patient frustration, inefficient use of time, and increased risk of infection.

[0194] Mozart procedures are typically performed in a doctor's office rather than in a hospital setting. Doctor's offices may not have a fully-designated pathology laboratory, and equipment costs may limit the range of tools available. For example, a doctor's office may have a microscope with a digital camera (e.g., an RGB camera with a CMOS or CCD sensor) and the necessary equipment for preparing tissue slides, but may not have other specialized equipment such as multispectral or hyperspectral scanners.

[0195] In some cases, surgeons performing procedures may also examine tissue slides. However, this can consume valuable physician time. In some situations, a physician's office may employ or contract with pathologists and / or pathology technicians to prepare and / or analyze tissue slides. Either approach can result in time loss, increased costs, lack of expertise, etc. For example, if only a limited number of surgeries are performed, a pathologist may not have enough work to fully occupy their day. Physicians who review their own tissue slides can often do a good job, but exposure to only a limited number of images may limit their expertise, especially in challenging cases. Furthermore, as discussed in this article, routine staining can lead to artifacts that make image analysis more difficult.

[0196] One challenge associated with the Mohr's procedure is distinguishing between clusters of basal cell carcinoma and hair follicles, which can appear similar. Analyzing more sections can reduce misidentification; however, surgeons often review one or two tissue sections, which may not be sufficient to differentiate between clusters of basal cell carcinoma and hair follicles, and preparing and analyzing additional tissue sections can be too time-consuming to perform when the procedure is not possible while the patient is waiting with an open wound. In some cases, compression artifacts, freezing artifacts, cauterization artifacts, tissue folds, squeezing artifacts (e.g., due to compression by forceps), loose incision artifacts, fat compression, poor staining, etc., may occur due to tissue removal and sample preparation. These issues can make the interpretation of tissue sample slides difficult. As an example, staining agents can extend along the surgical margins and stain the edges, creating edge artifacts that can give the false impression that the entire margin is clean.

[0197] In some cases, cancerous tissue can be difficult to identify when significant inflammation is present. This problem can exist, for example, in squamous cell carcinoma complicated by local infection, intrinsic lymphoproliferative disorders (such as chronic lymphocytic leukemia), etc. As another example, perineural spread can be difficult to detect.

[0198] Analysis can be more or less difficult, depending on the region where the tissue is removed. For example, tissue removed from relatively flat areas can be analyzed relatively easily because removing clean tissue slices with the desired shape (e.g., scallop-shaped) is relatively simple. However, regions with more complex shapes, such as ears and eyelids, may present challenges in removing tissue samples with the desired geometry.

[0199] This process is destructive when using conventional physical or chemical staining. Therefore, if an error occurs, it may be impossible to correct. This is particularly problematic for the Mozart procedure, as the sample needs to be analyzed to determine whether more tissue should be removed. In some cases, errors in sample preparation can lead to the removal of additional tissue that may be unnecessary in others.

[0200] Digital staining can alleviate many problems associated with Moser's procedure. For example, slide preparation time can be reduced because physical or chemical staining of tissue samples (e.g., using H&E) is not required. In some embodiments, the need for a field pathologist can be eliminated, and samples can instead be sent to an external party for analysis. In some embodiments, artificial intelligence and / or machine learning models can be trained on images of tissue slides, enabling the models to identify common problems in tissue images, such as cancer cells.

[0201] In some embodiments, physicians, technicians, or other individuals in medical facilities such as physician offices, outpatient surgery centers, etc., may use a microscope equipped with a digital imaging device (e.g., a camera with a CCD or CMOS sensor) to capture images of tissue sample slides. In some embodiments, the images may have a magnification of about 1x to about 10x, for example, 2x to 10x. In some cases, the facility may have access to a hyperspectral scanner or a multispectral scanner, but in many cases, an RGB camera or a similar camera with a small number of channels (e.g., four or fewer) may be used to capture images. For example, the camera may use Bayer filters, CYGM filters, CYYM filters, RGBE filters, RYYB filters, RGBW filters, or any other filter array.

[0202] Figure 27 An example of the Mozart procedure according to some embodiments is shown. The patient may have irrelevant tissue 2702 and relevant tissue 2704. For example, irrelevant tissue 2702 may be healthy tissue, while relevant tissue 2704 may be tissue including cancer cells. At (1), the patient may present irrelevant tissue 2702 and relevant tissue 2704. Relevant tissue 2704 may be, for example, visible skin cancer, such as basal-scale carcinoma. At (2), the surgeon may remove tissue section 2706. Tissue section 2706 may include relevant and irrelevant tissue. Tissue section 2706 may be frozen, sectioned, and placed on slide 2712, while irrelevant tissue 2702 and relevant tissue 2704 may be visible. Figure 27 In the example, some related tissue 2704 is retained after (2). At (3), additional tissue 2708 is removed. Figure 27As shown, after the second tissue removal, additional related tissue is removed; however, some related tissue is retained. At (4), additional tissue 2710 is removed, and the slide shows only healthy tissue, indicating that the procedure is complete. Although Figure 27 The image shows circular sections and symmetrical tissue removal, but it should be understood that this is not necessary, and one advantage of the Mohs procedure is that it can target only the relevant tissue while minimizing damage to or removal of irrelevant tissue.

[0203] As described herein, in some embodiments, a technician or other individual may perform a relaxation cut in a tissue sample. Figure 28 Several non-limiting examples of relaxation cuts (relaxation incisions) are shown below. Figure 28 As shown, sample 2800 may have one, two, or more relaxation incisions 2810. In some embodiments, the sample may be completely divided into multiple sections (e.g., sections 2800A, 2800B, 2800C, 2800D). Relaxation incisions help ensure that the entire or substantially the entire edge is visible when the sample is flattened for examination under a microscope. As described herein, relaxation incisions can lead to staining artifacts, tissue deformities, etc. These issues may be important to consider when analyzing tissue samples to reduce the likelihood that relevant tissue (e.g., cancer cells) is not detected or that unrelated healthy tissue is misidentified as cancerous.

[0204] Digital staining can eliminate or reduce many of the difficulties associated with Mozart procedures or similar procedures. Since chemical H&E staining may not be necessary, the time to obtain histopathological results can be reduced, and the need for a field pathologist or technician can be reduced or eliminated. In some embodiments, the systems and methods described herein can reduce surgical time. In some embodiments, the systems and methods described herein can improve outcomes by, for example, automatically detecting cancer cells using a trained machine learning model. As described herein, certain artifacts, distortions, etc., can make it extremely difficult for physicians to distinguish between relevant and irrelevant tissues, which can lead to incomplete removal of relevant tissues. In some embodiments, the systems and methods described herein can reduce surgical costs, surgical time, etc., for example, by eliminating or reducing the need for field pathologists.

[0205] Different providers may wish to utilize digital histopathology platforms in different ways. For example, some providers may wish to utilize digital staining but could continue to review the stained (e.g., digitally stained) images themselves or have their own pathologists review the images. In some cases, providers may wish to have the histopathology platform perform the analysis. In some cases, providers may wish to use only human analysis. In some cases, providers may wish to utilize only machine learning-based analysis. In some cases, providers may wish to utilize both human-based and ML-based analysis. For example, in some cases, providers may utilize human analysis because the human pathologist may be a licensed healthcare provider, and the provider may wish to rely on another trained professional. In some cases, providers may utilize the ML-based products of the digital histopathology platform. As discussed in this paper, utilizing ML-based image analysis can improve results because humans may miss certain features that can be detected using machine learning algorithms. Additionally, processing can be faster because ML models can be executed in seconds or minutes, whereas using a human pathologist might take minutes or hours, and the pathologist may not be able to review the images immediately. In some embodiments, physicians or other trained professionals can review ML-based analyses to determine whether to accept the determination made using the machine learning model. This can be an important step because it ensures that a human element is present in the loop and that the ML model, rather than making diagnostic decisions, provides guidance to medical professionals who can ultimately use the ML model's output along with their own experience and expertise to make medical decisions.

[0206] Figure 29An example process 2900 for performing a Mozart procedure or similar procedure using digital staining, according to some embodiments, is illustrated. At operation 2902, a medical facility (e.g., a physician at the medical facility) can extract tissue from a patient. At operation 2904, the medical facility (e.g., a physician, technician, etc.) can prepare an unstained tissue slide, such as a frozen tissue slide. At operation 2906, the medical facility can capture one or more microscopic images of the unstained tissue slide. At operation 2908, the medical facility can send one or more images to a histopathology platform, for example, by accessing an online portal, via an application, via email, etc. At operation 2910, the histopathology platform can receive one or more images. At operation 2912, the histopathology platform can digitally apply H&E staining agent and / or one or more other staining agents to one or more images. At operation 2914, the histopathology platform can provide one or more digitally stained images to the medical facility, for example, via email, a portal, an application, etc. At operation 2916, the medical facility can receive one or more digitally stained images. At operation 2918, the medical facility can analyze one or more digitally stained images, for example, to examine cancer cells at or near the tissue edges. At operation 2920, based on this analysis, the medical facility can extract additional tissue. At operation 2922, the medical facility can prepare a second unstained tissue slide, and at operation 2924, one or more images of the second unstained tissue slide can be captured. At operation 2926, the medical facility can send one or more images of the second unstained tissue slide to the histopathology platform. At operation 2928, the histopathology platform can receive one or more unstained images of the second tissue sample. At operation 2930, the histopathology platform can digitally stain one or more unstained images of the second tissue sample. At operation 2932, the histopathology platform can provide one or more digitally stained images of the second tissue sample to the medical facility. At operation 2934, the medical facility can analyze one or more digitally stained images of the second tissue sample. If no relevant tissue is found at the image edges, the processing can end. If so, the medical facility can continue to remove tissue, image the removed tissue, send the images of the removed tissue to a histopathology platform, and analyze the digitally stained images received from the histopathology platform until the digitally stained images indicate that no related tissue is present at the edges.

[0207] exist Figure 29In processing 2900, the medical facility performs analysis of the digitally stained images. However, in some embodiments, a histopathologist separate from the medical facility may analyze the digitally stained images. For example, a third party may analyze the digitally stained images. In some embodiments, the operator of the histopathology platform provides the analysis of the digitally stained images. For example, the operator of the histopathology platform may employ or contract with a trained pathologist capable of analyzing digitally stained images, or the medical facility may work with a histopathologist who is not an employee of the medical facility. Such an approach may be beneficial for medical facilities whose histopathology needs are insufficient to employ a histopathologist.

[0208] Figure 30 This is a flowchart illustrating an example process 3000 for performing a Mozart procedure or similar procedure according to some embodiments. At operation 3002, the medical facility prepares an unstained tissue slide. At operation 3004, the medical facility captures a microscopic image of the unstained tissue slide. At operation 3006, the medical facility is able to send the image to a histopathology platform. At operation 3008, the histopathology platform can digitally stain the received image, for example, using digital H&E staining agent. At operation 3010, the histopathologist can analyze the digitally stained image. At operation 3012, the histopathology platform can provide the digitally stained image and analysis to the medical facility. At operation 3014, the medical facility can extract additional tissue based on the digitally stained image and analysis. At operation 3016, the medical facility prepares a second unstained tissue slide. At operation 3018, the medical facility captures an image of the second slide. At operation 3020, the medical facility is able to send the second image to the histopathology platform. At operation 3022, the histopathology platform can, for example, digitally stain the second image using digital H&E staining agent. At operation 3024, the histopathologist is able to analyze the second digitally stained image. At operation 3026, the histopathology platform can provide the second digitally stained image and analysis to the medical facility. At operation 3028, the medical facility can review the second digitally stained image and analysis. If no relevant tissue is found at the edge, the process can stop. If relevant tissue is present at the edge, the process can continue; that is, the medical facility removes additional tissue, prepares additional unstained slides, and sends an image of the unstained slide to the histopathology platform for digital staining and analysis by the histopathologist.

[0209] In some embodiments, the histopathologist may work directly for the histopathology platform. In some embodiments, the histopathologist may be a third party, such as an independent histopathologist working on a contractual basis with the histopathology platform or medical facility.

[0210] In some embodiments, a histopathology platform may use one or more machine learning models to analyze digitally stained images. For example, supervised learning may be used to train the image analysis model. For instance, the image analysis model may be trained to detect cancer cells by providing it with digitally stained images in which features have been labeled, such as irrelevant tissue, relevant tissue, artifacts due to sample preparation, hair follicles, nerves, etc. The model's weights may be adjusted to train the model to distinguish relevant tissue from other tissues or artifacts. In some embodiments, the histopathologist may review the output of the machine learning model and indicate whether the output is correct. In some embodiments, the machine learning model may undergo retraining based on feedback received from the histopathologist.

[0211] Figure 31 This is a flowchart illustrating an example procedure 3100 for performing a Mozart procedure or similar procedure according to some embodiments. At operation 3102, the medical facility prepares an unstained tissue slide. At operation 3104, the medical facility captures a microscopic image of the unstained tissue slide. At operation 3106, the medical facility sends the image to a histopathology platform. At operation 3108, the histopathology platform digitally stains the received image, for example, using digital H&E staining. At operation 3110, an image analysis model analyzes the digitally stained image. At operation 3112, the histopathology platform provides the digitally stained image and analysis to the medical facility. At operation 3114, the medical facility extracts additional tissue based on the digitally stained image and analysis. At operation 3116, the medical facility prepares a second unstained tissue slide. At operation 3118, the medical facility captures an image of the second slide. At operation 3120, the medical facility sends the second image to the histopathology platform. At operation 3122, the histopathology platform can, for example, digitally stain the second image using digital H&E staining agent. At operation 3124, the image analysis model can analyze the second digitally stained image. At operation 3126, the histopathology platform can provide the second digitally stained image and / or analysis to the medical facility. At operation 3128, the medical facility can review the second digitally stained image and analysis. If no relevant tissue is found at the edge, the process can stop. If relevant tissue is present at the edge, the process can continue; that is, the medical facility removes additional tissue, prepares additional unstained slides, and sends an image of the unstained slides to the histopathology platform for digital staining and analysis via the image analysis model.

[0212] Using image analysis models offers several advantages. For example, it can reduce costs, improve accuracy (e.g., image analysis models can be trained on hundreds, thousands, or more images, and they can better distinguish relevant tissues from other artifacts and features that may appear in the image), and reduce processing time.

[0213] In some embodiments, when a medical facility sends images to a histopathology platform, it may also provide indications of the condition, indications of the areas where tissue has been removed, etc. In some embodiments, training data labeled with the condition and regions may be used to train the image analysis model. Such information can be important because, for example, different cancers or conditions may have different appearances and / or tissue extracted from different parts of the body may have different appearances.

[0214] In some embodiments, supervised learning is used to train an image analysis model to identify cancerous tissue within a tissue sample (e.g., within chemically stained and / or digitally stained images of a prepared tissue sample slide). For example, multiple training images (e.g., multiple digitally stained training images) may be labeled as having or not having cancerous tissue, and the model may be trained to classify new images (e.g., new digitally stained images) as showing or not showing cancerous tissue. In some embodiments, the trained model classifies tissue images (such as digitally stained tissue images) as showing or not showing cancer.

[0215] As described in this article, a significant benefit of the Mozart procedure is the precision with which cancerous tissue can be removed, reducing the amount of healthy tissue lost, which can be important for recovery, reducing the likelihood of infection, and minimizing scarring. Classifier models can effectively help practitioners identify the presence of cancerous tissue in a tissue sample and thus determine whether further tissue removal is permissible. However, classifying the entire image may not provide practitioners with sufficient information about where cancerous tissue is located to allow for further removal. In some implementations, object localization and / or recognition models can be used to identify regions within an image (e.g., within a digitally stained tissue image) where cancerous tissue is more likely to be present. In some implementations, region-based convolutional neural networks (R-CNN) or You Only Look Once (YOLO) models are used to identify regions within an image that may depict cancer cells.

[0216] In some implementations, classification and / or location information is provided to the practitioner. The practitioner can review this information when deciding whether to remove additional tissue, and if so, decide where to remove the additional tissue from.

[0217] While this document primarily describes digital staining, it will be understood that machine learning models for classification, object location, or both can be additionally or alternatively trained to operate on images of chemically stained and / or physically stained tissue. In some embodiments, separate models are used for digitally stained images and for images of chemically stained and / or physically stained tissue. However, this approach is not mandatory. For example, the digital staining techniques described herein can be used to generate digitally stained images that are similar to or even indistinguishable from, or nearly indistinguishable from, conventional physically / chemically stained images.

[0218] In some embodiments, a training image set is used to train the classification and / or localization model. This training image set includes images of tissue from different regions of a patient's body (eyelids, nose, ears, genitals, cheeks, etc.). In some embodiments, multiple models (e.g., a set of models) are trained, and each model can be trained to analyze images of tissue removed from one or more specific regions. For example, one model may be trained to analyze images of tissue removed from the eyelids, another model may be trained to analyze images of tissue removed from the ears, and yet another model may be trained to analyze images from relatively flat areas. In some embodiments, when using a single model, the model can be trained using images labeled according to the location where tissue is extracted from the images, and the model can accept tissue location as input.

[0219] As described herein, the Mohr's procedure can be used to treat a variety of cancers. In some embodiments, the same model can be used for different cancer types. For example, the model can be trained using images labeled with the cancer types shown in the images, and the model can accept the cancer type as input. In some embodiments, a set of models is trained, and a model is selected from this set of models based on a specified cancer type.

[0220] As described in this paper, numerous issues can affect images of unstained tissue, such as freezing artifacts, edge artifacts, cauterization artifacts, compression artifacts, squeezing artifacts, tissue folds, and edge artifacts caused by loose incisions. These issues can pose challenges to identifying cancerous tissue in images captured from tissue removed during a Moscone procedure, where the need for further tissue removal is at least partly guided by the presence of cancer cells at the edges. Therefore, it is important to use images of tissue captured as part of a Moscone procedure to train machine learning models so that the models can learn to identify cancerous tissue even in the presence of such artifacts.

[0221] 3D co-registration

[0222] This paper describes a system and method for 2D co-registration of stained and unstained images. The system and method described herein can be readily applied to 3D imaging. In some embodiments, a 3D image can be formed from a collection of 2D images, for example, by preparing 2D images by slicing a sample into thin layers and imaging each layer individually.

[0223] In some embodiments, 3D images can be created by changing the focal length of a camera lens or imaging system and capturing 2D images at different focal lengths. Changing the focal length can change the camera's field of view and / or depth of field. By changing the focal length, different depths can be focused. Further details of systems and methods for capturing 3D images by changing the focal length can be found in U.S. Patent No. 8,725,237, which is incorporated herein by reference. It should be understood that changing the focal length to capture 3D images is not limited to any particular emission or detection technology. For example, changing the focal length can be applied to grayscale (e.g., single-channel) imaging, color imaging (e.g., three-channel imaging), multispectral imaging, hyperspectral imaging, etc. The co-registration method described herein can also be applied to other forms of 3D imaging, such as CT scans, MRI, PET scans, 3D ultrasound, etc.

[0224] Electric motors, gears, and other components can be used to adjust focus, move detectors, move transmitters, etc. While these can achieve precise movement, alignment errors can occur in some cases due to tilting or drift in positioning over time. In some cases, the alignment error may be small relative to the slice or layer thickness. In others, the alignment error can be significant. For example, modern CT scanners can scan with slice thicknesses less than about 1 mm. In the case of microscopic imaging (e.g., imaging of tissue), when prepared using conventional slicing methods, the sample thickness may be only a few cells thick, such as about two cells thick. A typical skin cell diameter can be about 30 micrometers. Therefore, even small alignment errors at the micrometer scale can result in a displacement of one or more cell thicknesses.

[0225] Figure 32 An example method for registration in the depth direction z is shown according to some embodiments. For example... Figure 32 As shown, in some embodiments, a 3D image may include multiple 2D horizontal planes 3200. In some embodiments, planar slices 3202 may be obtained through the horizontal planes 3200. In some embodiments, a column of pixels 3204 may be extracted from the horizontal planes 3200.

[0226] A horizontal pixel plane or pixel column can be extracted from different 3D images, and the horizontal plane or column can be co-registered by, for example, translating, expanding, shrinking or rotating at least part or all of the column or plane.

[0227] Three-dimensional co-registration may present some challenges that are not found in 2D co-registration. For example, adjusting the registration in the vertical dimension can lead to changes in the registration in the horizontal plane. Figure 33 An iterative co-registration process according to some embodiments is illustrated. The iterative co-registration process 3300 includes, at operation 3302, performing in-plane co-registration, for example, as described herein with respect to 2D images. At operation 3304, the system may perform depth co-registration as described herein, for example, by extracting a vertical plane or a column of pixels from each of the two images and performing co-registration in a manner similar to or identical to that described herein for 2D images. As described herein, co-registration in the depth direction results in a change in the in-plane co-registration. At operation 3306, the system may evaluate the in-plane co-registration, for example, to determine whether the in-plane co-registration error is within a threshold. At operation 3308, if the in-plane co-registration is within a predetermined limit, the process may stop. If not, the system may iteratively perform in-plane co-registration and depth co-registration until the in-plane co-registration is within the predetermined limit.

[0228] exist Figure 32 The diagram illustrates an example of using planar or column extraction from multiple layers along the depth direction. While such methods may be effective, they can struggle to detect tilt issues (e.g., if the sample stage becomes tilted between images, or if samples become compressed on one side between images). In the case of planar slices (e.g., planar slice 3202), tilt can be detected if it lies in the direction captured by the planar slice. However, if the tilt is orthogonal, it may be missed entirely. If the tilt has a component in the planar slice, a portion of the tilt can be captured, but complete information about the tilt (e.g., tilt amount, tilt direction) may not be fully determined. In some embodiments, orthogonal slices can be acquired, and the direction and magnitude of the tilt can be determined from them.

[0229] Figure 34 Examples are shown of slices that can be obtained from a 3D image used for co-registration along the depth direction z, according to some embodiments. Figure 34As shown, a cylindrical slice 3402 can be obtained through a horizontal plane 3200. The cylindrical slice 3402 may have a diameter d. Pixels can be extracted along the surface of the cylindrical slice 3402. In some embodiments, the cylindrical slice 3402 can be unfolded to project the cylindrical slice 3402 onto a plane, thereby forming a planar representation 3402'. If a tilt is present, the tilt can be represented as a sinusoidal shape 3406 in the planar representation 3402'. The amplitude and phase of the sinusoidal shape 3406 can indicate the magnitude and direction of the tilt. Using this information, the tilt can be corrected. The diameter d can be significant. For example, if the diameter d is small, any local tilt can be identified, but a more global tilt may not be observed. If the diameter d is large, for example, approximately the size of the field of view, distortion and artifacts at or near the edges of the tissue sample may incorrectly indicate or exaggerate the amount of tilt.

[0230] Image metadata

[0231] When performing digital histopathology, it is important to maintain tracking information such as the patient associated with the tissue sample, the orientation of the tissue sample, the time of tissue sample collection or slide preparation, and the time of image capture. In some embodiments, digital images may be tagged with metadata indicating at least some of this information and / or other information. In some embodiments, the metadata may be included in the image file. In some embodiments, the metadata may be included in the image itself. For example, in some embodiments, the image may include a barcode, QR code, serial number, or other machine-readable information that may link the image to a specific patient and / or other metadata.

[0232] TOLL

[0233] Using a digital histopathology platform can offer many benefits, such as reduced staffing, equipment, and time to results. However, using a digital histopathology platform can present challenges, such as charging for services. In some embodiments, the histopathology platform can charge a healthcare facility for services, and the healthcare facility can charge the patient, the patient's insurance, etc. In some embodiments, the histopathology platform can receive insurance information, patient contact information, etc., and can charge the patient's insurance, the patient, or both directly. For example, the histopathology platform can determine coverage levels, any deductibles, etc., and can charge the patient's insurance for histopathology services and charge the patient a portion of the cost. In some embodiments, the histopathology platform can be configured to connect to an insurance billing platform, which can determine coverage amounts, deductibles, and other related costs. In some embodiments, the histopathology platform can be configured to determine appropriate charges and payments and charge accordingly.

[0234] Figure 35 Example processing for providing a histopathology platform service and charging for such service, according to some embodiments, is illustrated. At step 3502, the histopathology platform may receive unstained images from a medical facility. At step 3504, the histopathology platform may perform digital staining. At step 3506, the histopathology platform may perform histopathological analysis, for example, by employing the services of a pathologist and / or using an image analysis machine learning model. In some embodiments, steps 3504 or 3506 may be skipped. For example, in some embodiments, the medical facility may send stained images to be analyzed by a pathologist, or the medical facility may request digital staining services instead of analysis services. At step 3508, the histopathology platform may provide results to the medical facility. At step 3510, the histopathology platform may determine the charge amount. In some embodiments, the histopathology platform may communicate with an insurance system 3512 to determine the correct charge amount. For example, an insurance company may limit the amount it will pay for a particular service, and patients may receive maximum deductibles, co-payments, etc., as defined by their insurance plans. Importantly, determining the correct charge amount ensures that the histopathology platform is properly charged for the services provided, preventing patients from being overcharged or undercharged. As mentioned above, the histopathology platform can charge one or more of the following: the medical facility, the patient, and / or the patient's insurance. At step 3514, the histopathology platform can charge the medical facility for the services provided. At step 3516, the histopathology platform can charge the patient for the services provided. At step 3518, the histopathology platform can charge the patient's insurance for the services provided.

[0235] Although Figure 35 The example shown illustrates digital staining, but the method described in this paper can be readily applied to other services. For instance, instead of sending unstained tissue images, medical facilities could send CT scans, MRI scans, PET scans, ultrasound scans, etc., for the platform to analyze.

[0236] Computer System

[0237] Figure 36 This is a block diagram illustrating an embodiment of a computer hardware system configured to run software for implementing one or more embodiments of the health testing and diagnostic systems, methods, and apparatuses disclosed herein.

[0238] In some embodiments, using such Figure 36The computing system shown implements the systems, processes, and methods described herein. Example computer system 3602 communicates with one or more computing systems 3620, portable devices 3615, and / or one or more data sources 3622 via one or more networks 3618. Although Figure 36 An embodiment of computing system 3602 is shown, but it should be recognized that the functionality provided in the components and modules of computer system 3602 can be combined into fewer components and modules, or further separated into additional components and modules.

[0239] Computer system 3602 may include module 3614 that performs the functions, methods, actions, and / or processes described herein (e.g., processes as described above). Module 3614 is executed on computer system 3602 by central processing unit 3606, which is discussed further below.

[0240] Generally, as used herein, the term "module" refers to logic implemented in hardware or firmware, or a collection of software instructions with entry and exit points. Modules are written in programming languages ​​such as Java, C or C++, Python, etc. Software modules can be compiled or linked into executable programs installed in dynamic link libraries, or they can be written in interpreted languages ​​such as BASIC, PERL, LUA, or Python. Software modules can be called from other modules or from themselves, and / or can be called in response to detected events or interrupts. Hardware-implemented modules include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0241] Generally, the modules described herein refer to logical modules that can be combined with other modules or divided into submodules, regardless of their physical organization or storage. Modules are executed by one or more computing systems, and modules can be stored on or within any suitable computer-readable medium, or implemented wholly or partially within specially designed hardware or firmware. Not all computations, analyses, and / or optimizations require the use of a computer system, but the use of a computer can facilitate any of the methods, computations, processes, or analyses described above. Furthermore, in some embodiments, the process blocks described herein may be modified, rearranged, combined, and / or omitted.

[0242] Computer system 3602 includes one or more processing units (CPUs) 3606, which may include microprocessors. Computer system 3602 also includes physical memory 3610, such as random access memory (RAM) for temporary storage of information, read-only memory (ROM) for permanent storage of information, and mass storage devices 3604, such as backup storage devices, hard disk drives, spinning disks, solid-state drives (SSDs), flash memory, phase-change memory (PCM), 3D XPoint memory, disk or optical media storage devices. Alternatively, mass storage devices may be implemented in a server array. Typically, components of computer system 3602 are connected to the computer using a standards-based bus system. Various protocols can be used to implement the bus system, such as Peripheral Component Interconnect (PCI), Micro Channel, SCSI, Industry Standard Architecture (ISA), and Extended ISA (EISA).

[0243] Computer system 3602 includes one or more input / output (I / O) devices and interfaces 3612, such as a keyboard, mouse, touchpad, and printer. The I / O devices and interfaces 3612 may include one or more display devices, such as a monitor, which allows data to be visually presented to a user. More specifically, for example, the display device provides a GUI presentation as application software data and multimedia presentation. The I / O devices and interfaces 3612 may also provide communication interfaces to various external devices. Computer system 3602 may include one or more multimedia devices 3608, such as speakers, video cards, graphics accelerators, and microphones.

[0244] Computer system 3602 can run on various computing devices, such as servers, Windows servers, Structured Query Language servers, Unix servers, personal computers, laptops, etc. In other embodiments, computer system 3602 can run on clustered computer systems, mainframe computer systems, and / or other computing systems to be suitable for controlling and / or communicating with large databases, performing high-volume transaction processing, and generating reports from large databases. Computer system 3602 is typically controlled and coordinated by operating system software, such as z / OS, Windows, Linux, UNIX, BSD, SunOS, Solaris, macOS, iOS, iPadOS, Android, or other compatible operating systems, including proprietary operating systems. The operating system controls and schedules the computer processes used for execution, performs memory management, provides file systems, networking and I / O services, and provides user interfaces such as graphical user interfaces (GUIs).

[0245] Figure 36The computer system 3602 shown is coupled to a network 3618, such as a LAN, WAN, or the Internet, via a communication link 3616 (wired, wireless, or a combination thereof). Network 3618 communicates with various computing devices and / or other electronic devices. Network 3618 communicates with one or more computing systems 3620, one or more portable devices 3615, and one or more data sources 3622. Module 3614 can access or be accessed by the computing systems 3620, portable devices 3615, and / or data sources 3622 through a web-enabled user access point. The connection can be a direct physical connection, a virtual connection, or other connection types. The web-enabled user access point may include a browser module that uses text, graphics, audio, video, and other media to present data and allows interaction with the data via network 3618.

[0246] A user access point enabled by a web page, such as a personal computer, cellular phone, smartphone, laptop computer, tablet computer, e-reader device, audio player, or another device capable of connecting to the network 3618, can access the computer system 3602 via the computing system 3620, portable device 3615, and / or the module 3614 of the data source 3622. Such a device may have a browser module implemented to present data using text, graphics, audio, video, and other media and allow interaction with the data via the network 3618.

[0247] Output modules can be implemented as a combination of fully addressable displays (such as cathode ray tube (CRT), liquid crystal display (LCD), plasma display, or other types of displays) and / or as a combination of displays. Output modules can be implemented to communicate with input devices 3612, and they also include software with a suitable interface to allow users to access data using stylized screen elements such as menus, windows, dialog boxes, toolbars, and controls (e.g., radio buttons, checkboxes, sliders, etc.). Furthermore, output modules can communicate with a set of input and output devices to receive signals from the user.

[0248] One or more input devices may include a keyboard, a ball, a pen and stylus, a mouse, a trackball, a voice recognition system, or a pre-defined switch or button. One or more output devices may include a speaker, a display screen, a printer, or a speech synthesizer. Additionally, a touchscreen may act as a hybrid input / output device. In another embodiment, the user can interact more directly with the system, such as through a system terminal connected to the score generator, without communication via the Internet, WAN, LAN, or similar networks.

[0249] In some embodiments, system 3602 may include a physical or logical connection established between a remote microprocessor and a mainframe computer for the explicit purpose of uploading, downloading, or viewing interactive data and databases online in real time. The remote microprocessor may be operated by an entity operating computer system 3602 (including a client-server system or a mainframe system), and / or may be operated by one or more of data sources 3622, one or more of portable devices 3615, and / or one or more of computing systems 3620. In some embodiments, terminal emulation software may be used on the microprocessor to participate in the micro-host link.

[0250] In some embodiments, a computing system 3620 within an entity operating a computer system 3602 may internally access module 3614 as an application or process run by CPU 3606.

[0251] In some embodiments, one or more features of the systems, methods, and apparatus described herein may utilize URLs and / or cookies, for example, for storing and / or sending data or user information. A Uniform Resource Locator (URL) may include the web address and / or reference to a web resource stored in a database and / or on a server. A URL may specify the location of a resource on a computer and / or computer network. A URL may include mechanisms for retrieving network resources. The source of a network resource may receive a URL, identify the location of the web resource, and send the web resource back to the requester. A URL may be translated into an IP address, and the Domain Name System (DNS) may look up a URL and its corresponding IP address. A URL may be a reference to a web page, file transfer, email, database access, and other applications. A URL may include a sequence of characters that identify paths, domain names, file extensions, hostnames, queries, fragments, schemes, protocol identifiers, port numbers, usernames, passwords, flags, objects, resource names, etc. The systems disclosed herein may generate, receive, send, apply, parse, serialize, render URLs, and / or perform actions on URLs.

[0252] Cookies, also known as HTTP cookies, web cookies, internet cookies, and browser cookies, can include data sent from a website and / or stored on a user's computer. This data can be stored by the user's web browser while the user is browsing. Cookies can include useful information that websites remember from previous browsing, such as shopping carts in online stores, clicked buttons, login information, and / or records of previously visited web pages or online resources. Cookies can also include information entered by the user, such as name, address, password, credit card information, etc. Cookies can also perform computer functions. For example, authentication cookies can be used by applications (e.g., web browsers) to identify whether a user is logged in (e.g., logged into a website). Cookie data can be encrypted to provide security to consumers. Tracking cookies can be used to compile an individual's browsing history. The systems disclosed herein can generate and use cookies to access personal data. The system can also generate and use JSON web tokens to store authenticity information, HTTP authentication as an authentication protocol, IP addresses, URLs, etc., for tracking session or identity information.

[0253] The computing system 3602 may include one or more internal and / or external data sources (e.g., data source 3622). In some embodiments, relational databases, such as DB2, Sybase, Oracle, codebases, and Microsoft SQL Server, as well as other types of databases, such as flat file databases, entity-relational databases, and object-oriented databases and / or record-based databases, may be used to implement one or more of the aforementioned data stores and data sources.

[0254] Computer system 3602 can also access one or more databases 3622. Databases 3622 can be stored in a database or data store. Computer system 3602 can access one or more databases 3622 via a network 3618, or directly via I / O devices and interfaces 3612. The data store storing one or more databases 3622 can reside within computer system 3602.

[0255] Additional Examples

[0256] In the foregoing description, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made to the invention without departing from its broader spirit and scope. Therefore, the description and drawings should be considered illustrative rather than restrictive.

[0257] In fact, although the invention has been disclosed in the context of certain embodiments and examples, those skilled in the art will understand that the invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses of the invention and their obvious modifications and equivalents. Furthermore, while several variations of embodiments of the invention have been shown and described in detail, other modifications within the scope of the invention will be apparent to those skilled in the art based on this disclosure. Various combinations or sub-combinations of specific features and aspects of the embodiments are also contemplated, and these still fall within the scope of the invention. It should be understood that various features and aspects of the disclosed embodiments can be combined or substituted with each other to form variations of the disclosed embodiments of the invention. It is not necessary to perform any of the methods disclosed herein in the stated order. Therefore, the scope of the invention disclosed herein should not be limited to the specific embodiments described above.

[0258] It will be understood that the systems and methods of this disclosure have several innovative aspects, wherein no single aspect is solely responsible for or required for the desired properties disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure.

[0259] Some features described in the context of a single embodiment may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof. No single feature or group of features is necessary or indispensable for every embodiment.

[0260] It will also be understood that the conditional language used herein, such as “could,” “might,” “may,” etc., unless otherwise specifically stated or understood in the context in which they are used, is generally intended to indicate that some embodiments include certain features, elements, and / or steps, while other embodiments do not include these features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are necessary in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without author input or prompting, whether such features, elements, and / or steps are included in any particular embodiment or whether they are to be performed in any particular embodiment. The terms “comprising,” “including,” etc., are synonyms and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Additionally, the term “or” is used in its inclusive sense (rather than its exclusive sense) such that, for example, when used to connect a list of elements, the term “or” indicates one, some, or all of the elements in that list. Additionally, unless otherwise stated, the articles “a” and “the” used in this application and the appended claims should be interpreted as meaning “one or more” or “at least one”. Similarly, while operations may be shown in a specific order in the drawings, it should be understood that it is not necessary to perform such operations in the specific order shown or in a sequential order, or to perform all the shown operations, to achieve the desired result. Furthermore, the drawings may schematically illustrate one or more example processes in the form of flowcharts. However, other operations not shown may be combined with the schematically shown example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the shown operations. Additionally, in other embodiments, operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Additionally, other embodiments are also within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result.

[0261] Furthermore, while the methods and apparatus described herein are readily adaptable to various modifications and alternatives, specific examples have been shown in the accompanying drawings and described in detail herein. However, it should be understood that the invention is not limited to the specific forms or methods disclosed, but rather, the invention will cover all modifications, equivalents, and alternatives falling within the spirit and scope of the various embodiments described and the appended claims. Furthermore, any specific feature, aspect, method, property, characteristic, quality, attribute, element, etc., disclosed herein, in combination with embodiments or examples, may be used in all other embodiments or examples set forth herein. It is not necessary to perform any of the methods disclosed herein in the stated order. The methods disclosed herein may include certain actions taken by a practitioner; however, these methods may also explicitly or implicitly include any third-party instructions regarding these actions. The scope of this disclosure also covers any and all overlapping, sub-scopes, and combinations thereof. Languages ​​such as “up to,” “at least,” “greater than,” “less than,” “between,” etc., include the stated numbers. Numbers preceded by terms such as “about” or “approximately” include the stated numbers and should be interpreted on a case-by-case basis (e.g., as reasonably accurate as possible in that case, such as ±5%, ±10%, ±15%, etc.). For example, "about 8.5 mm" includes "3.5 mm". Phrases preceded by terms such as "substantially" include the stated phrase and should be interpreted on a case-by-case basis (e.g., as reasonably as possible in that case). For example, "substantially constant" includes "constant". Unless otherwise stated, all measurements are performed under standard conditions including temperature and pressure.

[0262] As used herein, the phrase “at least one” in the list of items refers to any combination of those items, including a single member. As an example, “at least one of A, B, or C” is intended to cover: A, B, C, A and B, A and C, B and C, and A, B, and C. Unless explicitly stated otherwise, associative language, such as the phrase “at least one of X, Y, and Z”, is generally understood in the context to mean that an item, term, etc., can be at least one of X, Y, or Z. Therefore, such associative language is not generally intended to mean that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z. The headings provided herein, if any, are for convenience only and are not intended to affect the scope or meaning of the devices and methods disclosed herein.

[0263] Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a model for classifying digitally stained tissue images, the method comprising: Receive the first image of the unstained tissue sample. The first image is a color image captured by a camera, which includes at least one of a charge-coupled device sensor or a complementary metal-oxide-semiconductor sensor and a Bayer filter. The unstained tissue sample was collected during the Moschus procedure; The first image is digitally colored to produce a first digitally colored image; The first digitally colored image is fed into the machine learning model; The machine learning model is used to determine the presence of cancerous tissue depicted in the digitally stained image; and An indication of the presence of cancerous tissue in the first digitally stained image is generated and displayed to the user.

2. The method according to claim 1, wherein, The machine learning model is a classifier model configured to classify digitally stained images as depicting cancerous tissue or not. The machine learning model is trained using supervised learning on a training image set. Each training image in the training image set is a digitally colored image. Each training image in the training image set is labeled as either depicting cancerous tissue or not depicting cancerous tissue.

3. The method according to claim 1, wherein, The machine learning model is an object recognition model, configured to identify one or more regions of cancerous tissue depicted in a digitally stained image. The machine learning model is trained using supervised learning on a training image set. Each training image in the training image set is a digitally colored image. Wherein, at least one training image in the training image set depicts at least one region of cancerous tissue, and Each of the at least one region of the cancerous tissue is labeled as the depicted cancerous tissue.

4. The method according to claim 1, wherein, The first image depicts at least one of the following: basal cell carcinoma, melanoma in situ, dermatofibrosarcoma protuberans, keratoacanthoma, spindle cell tumor, sebaceous gland carcinoma, microcystic adnexal carcinoma, or leiomyosarcoma.

5. The method according to claim 1, wherein, The machine learning model is trained using a training image set, wherein each training image in the training image set depicts a frozen tissue sample, and the model is trained to identify the presence of cancerous tissue in images depicting frozen artifacts.

6. The method according to claim 1, wherein, The first image is captured at a magnification of approximately 2X to approximately 10X.

7. The method according to claim 1, further comprising: Identify markings in the first image, wherein the markings include at least one of the following: a mark or dot on a slide, a mark or dot on tissue depicted in the image, a dye or stain on the slide, or a dye or stain on tissue depicted in the image.

8. The method according to claim 7, further comprising: The orientation of the unstained tissue sample is determined based on the location of the marker.

9. The method according to claim 1, wherein, The unstained tissue samples were digitally stained using digital hematoxylin and eosin staining agents.

10. The method according to claim 1, wherein, The unstained tissue sample includes one or more loose incisions.

11. A system for identifying cancerous tissue, comprising: At least one processor; as well as A non-transitory computer-readable medium having instructions stored thereon, which, when executed by the at least one processor, cause the system to: Receive the first image of the unstained tissue sample. The first image is a color image captured by a camera, which includes at least one of a charge-coupled device sensor or a complementary metal-oxide-semiconductor sensor and a Bayer filter. The unstained tissue sample was collected during the Moschus procedure; The first image is digitally colored to produce a first digitally colored image; The first digitally colored image is fed into the machine learning model; The machine learning model is used to determine the presence of cancerous tissue depicted in the digitally stained image; and An indication of the presence of cancerous tissue in the first digitally stained image is generated and displayed to the user.

12. The system according to claim 11, wherein, The machine learning model is a classifier model configured to classify digitally stained images as depicting cancerous tissue or not. The machine learning model is trained using supervised learning on a training image set. Each training image in the training image set is a digitally colored image. Each training image in the training image set is labeled as either depicting cancerous tissue or not depicting cancerous tissue.

13. The system according to claim 11, wherein, The machine learning model is an object recognition model, configured to identify one or more regions of cancerous tissue depicted in a digitally stained image. The machine learning model is trained using supervised learning on a training image set. Each training image in the training image set is a digitally colored image. Wherein, at least one training image in the training image set depicts at least one region of cancerous tissue, and Each of the at least one region of the cancerous tissue is labeled as the depicted cancerous tissue.

14. The system according to claim 11, wherein, The first image depicts at least one of the following: basal cell carcinoma, melanoma in situ, dermatofibrosarcoma protuberans, keratoacanthoma, spindle cell tumor, sebaceous gland carcinoma, microcystic adnexal carcinoma, or leiomyosarcoma.

15. The system according to claim 11, wherein, The machine learning model is trained using a training image set, wherein each training image in the training image set depicts a frozen tissue sample, and the model is trained to identify the presence of cancerous tissue in images depicting frozen artifacts.

16. The system according to claim 11, wherein, The first image is captured at a magnification of approximately 2X to approximately 10X.

17. The system according to claim 11, wherein, The instructions are also configured to cause the system to: Identify markings in the first image, wherein the markings include at least one of the following: a mark or dot on a slide, a mark or dot on tissue depicted in the image, a dye or stain on the slide, or a dye or stain on tissue depicted in the image.

18. The system according to claim 17, wherein, The instructions are also configured to cause the system to: The orientation of the unstained tissue sample is determined based on the location of the marker.

19. The system according to claim 11, wherein, The unstained tissue samples were digitally stained using digital hematoxylin and eosin staining agents.

20. The system according to claim 11, wherein, The unstained tissue sample includes one or more loose incisions.

Images