Machine learning processing for photon absorption remote sensing signals

Abstract

Systems and methods for analyzing a sample are provided. The methods may include receiving a plurality of signals from the sample, the plurality of signals comprising radioactive and non-radioactive signals, extracting a plurality of features based on processing at least one of the plurality of signals, the features providing contrast information provided by at least one of the plurality of signals, and applying the plurality of features to a machine learning architecture to generate inferences about the sample. Methods for training the machine learning architecture and generating stained images are also provided (FIG. 38A).

Description

[Technical Field]

[0001] This application relates to the field of optical imaging, and in particular to machine learning processing of photon absorption remote sensing (PARS) systems for analyzing samples comprising biological tissue in vivo, ex vivo, or in vitro. [Background technology]

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 382906, filed November 9, 2022, U.S. Provisional Patent Application No. 63 / 424647, filed November 11, 2022, U.S. Provisional Patent Application No. 63 / 443838, filed February 7, 2023, and U.S. Provisional Patent Application No. 63 / 453371, filed March 20, 2023, the contents of each of which are incorporated herein by reference in their entirety. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2016 / 0113507 [Patent Document 2] US Patent Application Publication No. 2017 / 0215738 [Non-patent literature]

[0004] [Non-Patent Document 1] A. Krull, T.-O. Buchholz, and F. Jug, "-Noise2Void-Learning Denoising from Single Noisy Images" arXiv, Apr.05, 2019.doi:10.48550 / arXiv.1811.10980 [Non-patent document 2] JEDTweel, BREcclestone, M. Boktor, JATSimmons, P. Fieguth, and PHReza, “Virtual Histology with Photon Absorption Remote Sensing using a Cycle-Consistent Generative Adversarial Network with Weakly Registered Pairs.”arXiv,Jun.26, 2023.doi:10.48550 / arXiv.2306.08583 Summary of the Invention [Problem to be solved by the invention]

[0005] Improved machine learning processing for photon absorption remote sensing signals is desired. [Means for solving the problem]

[0006] According to one aspect, a computer-implemented method for analyzing a sample is provided, which may include receiving a plurality of signals from the sample, the plurality of signals comprising light absorption radiative and non-radiative relaxation signals, extracting a plurality of features based on processing at least one of the plurality of signals, the features providing contrast information provided by the at least one of the plurality of signals, and applying the plurality of features to a machine learning architecture to generate inferences about the sample.

[0007] In some embodiments, the radiative and non-radiative signals comprise radiative and non-radiative absorption relaxation signals. In some embodiments, the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal.

[0008] In some embodiments, the radioactive signal comprises one or more autofluorescence signals. In some embodiments, the contrast may comprise one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast.

[0009] In some embodiments, processing the plurality of signals may comprise exciting the sample at an excitation location with an excitation beam, where the excitation beam is focused at or below the sample, and interrogating the sample with an interrogation beam directed towards the excitation location of the sample, where the interrogation beam is focused at or below the sample.

[0010] In some embodiments, extracting a plurality of features comprises processing both radioactive and non-radioactive signals. In some embodiments, the plurality of signals comprises an absorption spectrum signal.

[0011] In some embodiments, the plurality of signals comprises scattered signals. In some embodiments, the sample is an in vivo or in situ sample. In some embodiments, the sample is unstained.

[0012] In some embodiments, the sample is stained. In some embodiments, the plurality of features is complemented by at least one informative feature of image data acquired from a complementary modality.

[0013] In some embodiments, the complementary modalities comprise at least one of ultrasound imaging, positron emission tomography (PET) scanning, computed tomography (CT) scanning, and magnetic resonance imaging (MRI).

[0014] In some embodiments, image data acquired from complementary modalities may include photoactive labels to contrast or highlight specific regions within the image. In some embodiments, the plurality of features is complemented by at least one informative feature of the patient information.

[0015] In some embodiments, the processing step comprises converting at least one of the plurality of signals into at least one image. In some embodiments, the converting step comprises applying a simulated stain to the at least one image.

[0016] In some embodiments, the simulated staining comprises at least one of hematoxylin and eosin (H&E) staining, Jones staining (MPAS), PAS and GMS staining, toluidine blue, Congo red, Masson's trichrome staining, Lilly's trichrome, and Verhoeff's staining, immunohistochemistry (IHC), histochemical staining, and in situ hybridization (ISH).

[0017] In some embodiments, the simulated staining is made applicable to frozen tissue sections, archived tissue samples, or fresh, unprocessed tissue. For example, a preserved tissue sample may comprise a sample preserved using formalin or alcohol fixed using an alcohol fixative.

[0018] In some embodiments, converting the at least one image comprises converting to at least two images and applying a different simulation stain to each of the images.

[0019] In some embodiments, the converting step comprises applying a colorized machine learning architecture. In some embodiments, the colorized machine learning architecture comprises a generative adversarial network (GAN).

[0020] In some embodiments, the colorized machine learning architecture comprises a cycle-consistent generative adversarial network (CycleGAN). In some embodiments, the colorized machine learning architecture comprises a conditional generative adversarial network (cGAN), which may comprise, for example, a pix2pix model.

[0021] In some embodiments, the inference comprises at least one of survival time, drug response, drug resistance, phenotypic characteristics, molecular characteristics, mutational burden, tumor molecular characteristics, parasites, toxicity, inflammation, transcriptomic features, protein expression features, patient clinical outcome, suspicious signals, biomarker location or value, cancer grade, cancer subtype, tumor border area, and grouping of cancer cells based on cell size and shape.

[0022] In some embodiments, the method may further comprise generating a signal to cause rendering, at a display device, a user interface (UI) showing a visualization of the inference.

[0023] According to another aspect, a computer system for analyzing a sample is provided, the system comprising: a processor operating in conjunction with a computer memory and a non-transitory computer-readable storage; the processor is configured to receive from the sample a plurality of signals comprising radioactive and non-radioactive signals; extract a plurality of features based on processing at least one of the plurality of signals, the features providing contrast information provided by at least one of the plurality of signals; and apply the plurality of features to a machine learning architecture to generate an inference about the sample.

[0024] In some embodiments, the radiative and non-radiative signals comprise radiative and non-radiative absorption relaxation signals. In some embodiments, the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal.

[0025] In some embodiments, the radioactive signal comprises one or more autofluorescence signals. In some embodiments, the contrast may comprise one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast.

[0026] According to yet another aspect, a computer system for training a machine learning architecture is provided. The system includes a processor operating in conjunction with a computer memory and a non-transitory computer-readable storage. The processor is configured to instantiate, at each training iteration, the machine learning architecture including a neural network having a plurality of nodes and weights stored in a memory device, acquire true total absorption (TA) images, generate simulated stained images based on the true total absorption TA images, generate false total absorption TA images based on the generated stained images, calculate a first loss based on the generated false total absorption TA images and the true total absorption TA images, acquire labeled stained images, calculate a second loss based on the generated simulated stained images and the labeled stained images, and update the weights of the neural network based on at least one of the first loss and the second loss.

[0027] According to yet another aspect, a computer-implemented method for training a machine learning architecture for generating simulated stained images is provided. The machine learning architecture includes a plurality of nodes and weights stored in a memory device. The method includes, at each training iteration, acquiring a true total absorption (TA) image, generating a simulated stained image based on the true total absorption TA image, generating a false total absorption TA image based on the generated stained image, calculating a first loss based on the generated false total absorption TA image and the true total absorption TA image, acquiring a labeled stained image, calculating a second loss based on the generated simulated stained image and the labeled stained image, and updating weights of the neural network based on at least one of the first loss and the second loss.

[0028] In some embodiments, the simulated stained image is generated by a second neural network comprising a second set of nodes and weights, the second set of weights being updated during each iteration based on at least one of the first loss and the second loss.

[0029] In some embodiments, the fake total absorption TA image is generated by a third neural network comprising a second set of nodes and weights, the third set of weights being updated during each iteration based on at least one of the first loss and the second loss.

[0030] In some embodiments, calculating the second loss based on the generated simulated stained images and the labeled stained images may comprise processing the generated simulated stained images through a first identifier network, processing the labeled stained images through a second identifier network, and calculating the second loss based on respective outputs from each of the first identifier network and the second identifier network.

[0031] In some embodiments, the method may further comprise processing each output from each of the first and second identifier networks through a respective classification matrix before calculating the second loss.

[0032] In some embodiments, the machine learning architecture comprises a cycle-consistent generative adversarial network (CycleGAN) machine learning architecture. In some embodiments, the machine learning architecture comprises a conditional generative adversarial network (cGAN), which may comprise, for example, a pix2pix model.

[0033] In some embodiments, the labeled stain image is a labeled photon absorption remote sensing PARS image. In some embodiments, the labeled photon absorption remote sensing PARS images are automatically labeled based on unlabeled photon absorption remote sensing PARS images prior to training of the neural network.

[0034] In some embodiments, automatically labeling the unlabeled photon absorption remote sensing PARS image comprises labeling the unlabeled photon absorption remote sensing PARS image based on an existing labeled stained image from a database, where the existing labeled stained image and the unlabeled photon absorption remote sensing PARS image share structural similarities.

[0035] In some embodiments, the database is an H&E database. A portion of the interrogation, signal enhancement, excitation, or autofluorescence from the sample can be collected to form an image. These signals can be used to resolve the size, shape, features, dimensions, properties, and composition of the sample. In a given architecture, any portion of the light returning from the sample, such as the detection, excitation, or thermal enhancement beam, can be collected. The returning light can be analyzed based on wavelength, phase, polarization, etc., to capture any absorption-induced signals, including pressure, temperature, and optical emission. In this way, photon absorption remote sensing (PARS) can simultaneously capture, for example, scattering, autofluorescence, and polarization contrast resulting from each detection, excitation, and thermal enhancement source. Furthermore, photon absorption remote sensing (PARS) laser sources can be specifically selected to emphasize these different contrast mechanisms.

[0036] Other aspects will become apparent from the following specification and claims. [Brief explanation of the drawings]

[0037] [Figure 1] An overview of the PARS photon absorption remote sensing system is shown. [Figure 2] 1 shows an overview of a photon absorption remote sensing PARS system having photon absorption remote sensing PARS excitation and photon absorption remote sensing PARS detection. [Figure 3] An implementation of photon absorption remote sensing (PARS) combined with other modalities is presented. [Figure 4] The signal processing path of the PARS photon absorption remote sensing signal is shown. [Figure 5] 1 shows an exemplary architecture for total absorption (TA) photon absorption remote sensing (PARS), where an autofluorescence detection system is used as an example. [Figure 6] 1 shows visualizations generated by an autofluorescence-sensitive total absorption-photon absorption remote sensing (TA-PARS) architecture. [Figure 7]1 shows an exemplary signal evolution of a total absorption-photon absorption remote sensing TA-PARS signal. [Figure 8] Examples of radioactive and non-radioactive signals are shown. [Figure 9] An exemplary architecture is shown that uses two excitation sources, one detection source, and multiple photodiodes. [Figure 10] A comparison of non-radiative absorption (image (a)), radiative absorption (image (b)), and scattering (image (c)) provided by the total absorption-photon absorption remote sensing TA-PARS system is shown. [Figure 11] An example of total absorption-photon absorption remote sensing TA-PARS imaging is shown. [Figure 12A] 1 illustrates an exemplary application of quantum efficiency ratio (QER). [Figure 12B] 1 illustrates an exemplary application of quantum efficiency ratio (QER). [Figure 13] An example of total absorption-photon absorption remote sensing TA-PARS imaging using quantum efficiency ratio (QER) acquisition processing is shown. [Figure 14] 1 shows a comparison of imaging using quantum efficiency ratio (QER) acquisition processing with conventional staining. [Figure 15] 1 shows an exemplary photon absorption remote sensing PARS signal evolution. [Figure 16] An example of a lifetime photon absorption remote sensing (PARS) image in resected rat brain tissue is shown. [Figure 17] 1 shows an exemplary photon absorption remote sensing PARS signal evolution associated with the rapid lifetime extraction technique. [Figure 18] 1 illustrates an exemplary architecture for a multi-path (MP) photon absorption remote sensing PARS system. [Figure 19] Multiphoton-photon absorption remote sensing (PARS) is compared with conventional photon absorption remote sensing (PARS). [Figure 20A] 1 shows a reconstructed grayscale photon absorption remote sensing PARS image and corresponding staining. [Figure 20B] 1 shows a reconstructed grayscale photon absorption remote sensing PARS image and corresponding staining. [Figure 21A] Based on the principal components, the principal components of the time domain-photon absorption remote sensing TD-PARS signal and the synthesized staining are shown. [Figure 21B] Based on the principal components, the principal components of the time domain-photon absorption remote sensing TD-PARS signal and the synthesized staining are shown. [Figure 22] 1 illustrates an exemplary architecture for analyzing time-domain-photon absorption remote sensing (TD-PARS) signals. [Figure 23] 1 shows a graph of the time domain-photon absorption remote sensing TD-PARS signal and centroid. [Figure 24] 1 shows a visualization using a clustering method. [Figure 25] 1 shows the visualization of three different regions of brain tissue using a clustering method. [Figure 26] 1 illustrates an exemplary clustering algorithm analyzing time domain-photon absorption remote sensing TD-PARS signals and determining the image. [Figure 27] This shows how to classify images using a clustering algorithm. [Figure 28] 1 illustrates non-radiative signal extraction. [Figure 29] 1 illustrates various filtered instances of a photon absorption remote sensing PARS signal. [Figure 30] 1 illustrates the expected spatial correlation between adjacent points or signals. [Figure 31] In relation to feature extraction, two signals with different lifetimes are illustrated. [Figure 32] A comparison of the original image and the denoised image is shown. [Figure 33] 1 shows the chirp pulse signal and acquisition. [Figure 34]1 illustrates an exemplary time domain-photon absorption remote sensing (TD-PARS) acquisition by imposing a delay to reconstruct the signal. [Figure 35] Refers to data compression using digital and / or analog techniques. [Figure 36] 1 illustrates an exemplary fast acquisition approach. [Figure 37] 1 shows the direct construction of a colorized image. [Figure 38A] We present two exemplary architectures for generating one or more inferences about a sample. [Figure 38B] We present two exemplary architectures for generating one or more inferences about a sample. [Figure 39] 13 illustrates another exemplary architecture for generating one or more inferences about a sample. [Figure 40] 38A, 38B, or 39 illustrates an exemplary user interface that renders one or more inferences generated by the architecture of FIG. 38A, 38B, or 39. [Figure 41] 1 illustrates an exemplary machine learning architecture that may be used to implement an image generator. [Figure 42] 1 illustrates an exemplary process for preparing one or more training data for training an image generator. [Figure 43] 1 illustrates an exemplary neural network that can be used to implement an image generator. [Figure 44] An example of contrast extracted from a photon absorption remote sensing (PARS) signal in a tissue slide is shown. [Figure 45] An example of contrast combination from photon absorption remote sensing PARS signal combination into a unique contrast is shown. [Figure 46] Two hypothetical (simulated) stained photon absorption remote sensing PARS images are shown. [Figure 47A] An example of an unlabeled photon absorption remote sensing PARS virtual H&E image is shown. [Figure 47B] 47B shows a historically labeled H&E image correlated to the image in FIG. 47A. [Figure 48] 1 shows examples of different tissue types identified and imaged using a machine learning architecture. [Figure 49] 1 shows the unique keratin pearl features identified and isolated within an exemplary simulated staining image. [Figure 50] 1 shows biomarkers of localized inflammation and malignancy identified and encompassed based on exemplary simulated staining images. [Figure 51] 1 shows different cell types and tissue regions identified and delineated within an exemplary simulated staining image. [Figure 52] 1 shows an example of an abnormal tissue region identified and delineated from an exemplary simulated stain image. [Figure 53] FIG. 1 is a schematic diagram of a computing device that may be used to implement a computing device used to train or run (during inference) a machine learning model. [Figure 54] 38A, 38B, or 39 illustrates processing performed by a processor of an exemplary embodiment of the machine learning system or architecture of FIG. 39. [Figure 55] 38A, 38B, or 39 shows an exemplary heatmap generated by an exemplary embodiment of the machine learning system or architecture of FIG. 38A, 38B, or 39. [Figure 56] 38A, 38B, or 39 shows an exemplary multi-stain image generated by an exemplary embodiment of the machine learning system or architecture of FIG. 39. [Figure 57A] 38A, 38B, or 39 shows an exemplary embodiment of an image generator connected to a photon absorption remote sensing (PARS) system. The image generator can be part of the machine learning system or architecture of FIG. 38A, 38B, or 39. [Figure 57B]1 illustrates another exemplary embodiment of an image generator connected to a photon absorption remote sensing PARS system. [Figure 58] 10 illustrates yet another exemplary embodiment of an image generator connected to a pre-processing module. [Figure 59] 38A, 38B, or 39 illustrates an exemplary user interface for analyzing one or more images generated by the architecture of FIG. 38A, 38B, or 39. [Figure 60] 38A, 38B, or 39 illustrates an exemplary user interface for displaying one or more images generated by the architecture of FIG. 38A, 38B, or 39. [Figure 61] 38A, 38B, or 39 illustrates another exemplary user interface for displaying one or more images generated by the architecture of FIG. 38A, 38B, or 39. [Figure 62] 1 illustrates an exemplary user interface for scanning and processing one or more images. [Figure 63] 1 illustrates an exemplary user interface for displaying an annotated image. [Figure 64] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 65] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 66] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 67] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 68] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 69] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 70A] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 70B] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 70C] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 71] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 72] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 73] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 74]1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 75] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 76] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 77] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 78] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 79] 1 illustrates various schematic diagrams of example embodiments of machine learning architectures or processes for generating one or more inferences based on output from a photon absorption remote sensing (PARS) system. [Figure 80] An example of raw photon absorption remote sensing PARS data in a total absorption-photon absorption remote sensing TA-PARS image denoised using the Noise2Void (N2V) framework is shown. [Figure 81] 10 illustrates an exemplary implementation of an error correction sub-module for denoising photon absorption remote sensing PARS images. [Figure 82A] 1 shows an exemplary visualization of the data preparation and virtual staining processes of an image. [Figure 82B] 1 shows an exemplary visualization of the data preparation and virtual staining processes of an image. [Figure 83]10 shows exemplary denoising results using denoising and error correction processes applied to both raw photon absorption remote sensing PARS image data. [Figure 84] 1 shows exemplary photon absorption remote sensing PARS non-radiative time domain signatures extracted from a photon absorption remote sensing PARS event. [Figure 85] 1 illustrates an exemplary multi-channel virtual staining architecture for signal processing and virtual staining of photon absorption remote sensing PARS image data. [Figure 86] 1 shows a comparison of virtual staining results using different combinations of photon absorption remote sensing PARS feature images as input. [Figure 87] 1 illustrates an exemplary photon absorption remote sensing PARS data vector or feature vector. [Figure 88] An exemplary photon absorption remote sensing PARS virtual multi-stain image based on the same photon absorption remote sensing PARS image data. DETAILED DESCRIPTION OF THE INVENTION

[0038] Reference will now be made in detail to the embodiments of the present disclosure, which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. In the following discussion, relative terms such as "about," "substantially," "approximately," and the like are used to indicate possible variations in the numerical values ​​set forth.

[0039] A recently reported photoacoustic technique known as photoacoustic remote sensing (PARS) microscopy (Patent Documents 1 and 2) solves many of these sensitivity issues through a novel detection mechanism. Photon absorption remote sensing (PARS) allows for direct detection of the excited photoacoustic region, rather than detecting acoustic pressure at an external surface once away from the source. This is achieved by monitoring changes in material optical properties that coincide with photoacoustic excitation. These changes then encode various salient material properties, such as optical absorption, physical target dimensions, and constituent chromophores, to name a few.

[0040] Because photon absorption remote sensing (PARS) devices may utilize only two optical beams, which may be in a confocal configuration, the spatial resolution of the imaging technique may be defined as excitation-defined (ED) or interrogation-defined (ID), depending on which of the beams provides a tighter focus at the sample. This aspect may also facilitate imaging of deeper targets beyond the limits of the optical resolution device. This may be achieved by leveraging deep-penetrating (long transport mean free path) detection wavelengths, such as short-wave infrared (e.g., 1310 nm, 1700 nm, or 10 μm), which may provide spatial resolution to depths superior to those provided by a given excitation (e.g., 532 nm or 266 nm) within highly scattering media such as biological tissue. If more than two beams are used, such that the system is configured with more than two focal points at the sample, a clear expansion of these components is expected. For example, if additional beams are added that amplify the signal within that focal region, this may also contribute to defining the expected resolution of the system.

[0041] Intensity-modulated photon absorption remote sensing (PARS) signals depend not only on optical absorption and incident excitation fluence, but also on the detection laser wavelength, fluence, and sample temperature. Photon absorption remote sensing (PARS) signals can also arise from other effects, such as scatterer position modulation and surface vibration. Similar analogs may exist for photon absorption remote sensing (PARS) devices that utilize other modulating optical properties, such as intensity, polarization, frequency, phase, fluorescence, nonlinear scattering, and nonlinear absorption. Because material properties depend on ambient temperature, there is a corresponding temperature dependence in the photon absorption remote sensing (PARS) signal. At some intensity levels, additional saturation effects may also be exploited.

[0042] The above mechanisms point to important sources of scattering position or scattering cross-section modulation that can be easily made measurable when the probe beam is focused to sense a limited excitation volume. However, these large local signals are not the only potential source of the photon absorption remote sensing PARS signal. Acoustic signals propagating to the sample surface can also result in changes to the photon absorption remote sensing PARS signal. These acoustic signals can also generate surface vibrations that result in phase modulation of the photon absorption remote sensing PARS signal.

[0043] These generated signals can be intentionally controlled or influenced by secondary physical effects such as vibration, temperature, stress, surface roughness, mechanical bending, among others. For example, temperature can be introduced into the sample, which may enhance the generated photon absorption remote sensing PARS signal compared to that generated without the introduction of this additional temperature. Another example can involve introducing mechanical stress (such as bending) into the sample, which in turn can affect the sample's material properties (e.g., density or local optical properties such as birefringence, refractive index, absorption coefficient, scattering behavior, etc.), thereby perturbing the generated photon absorption remote sensing PARS signal compared to that which would have been generated without the introduction of this mechanical stress. To enhance the generated photon absorption remote sensing PARS signal, additional contrast agents can be added to the sample, including, but not limited to, dyes, proteins, specially designed cells, liquids, and optical agents or windows. Targets can be optically modified to provide optimized results.

[0044] Some techniques may simply monitor intensity backreflections and extract the amplitude of these time-domain signals. However, additional information can be extracted from the time-varying aspects of the signal. For example, some of the scattering, polarization, frequency, and phase content associated with a photon absorption remote sensing (PARS) signal may be due to the size, shape, characteristics, and dimensions of the region that generated the signal. This may encode additional unique / orthogonal information useful for improving the fidelity of the final image, classifying sample regions, sizing constituent chromophores, and classifying constituent chromophores, to name a few. Because such techniques can generate independent data sets for the same interrogated region, they can be combined or compared with each other. For example, frequency information may describe the microscopic structure within a sample, which can be combined with traditional photon absorption remote sensing (PARS) that uses scattering modulation to highlight regions that are both absorbing and of a particular size.

[0045] Referring to Figure 1, photon absorption remote sensing (PARS) microscopy is an all-optical non-contact optical absorption microscopy technique. Photon absorption remote sensing (PARS) can generate and detect optical absorption contrast in various specimens using a confocal excitation and detection laser pair. In photon absorption remote sensing (PARS), the excitation laser can include a pulsed excitation laser, which can be used to deposit optical energy into the sample. When light is absorbed by a chromophore, photon energy is captured by the specimen. The absorbed energy can then be dissipated through either light emission (radiative) or non-radiative relaxation. During non-radiative relaxation, the absorbed light energy is converted to heat. In certain cases, heat generation can cause thermoelastic expansion, resulting in photoacoustic pressure and photothermal signals. During radiative relaxation, the absorbed light energy is released by the emission of photons. Generally, the emitted photons exhibit a different energy level compared to the absorbed photons.

[0046] Local temperature and pressure changes result in nanosecond-scale perturbations in the optical and material properties of the sample. A detection laser confocalized with the excitation spot can capture absorption-induced perturbations in optical properties as scattering intensity modulation. By then measuring perturbations in the detection laser scattering, PARS is enabled to measure the nonradiative absorption contrast of different biomolecules. Simultaneously, by capturing the undisturbed back-reflected and back-reflected excitation energy of the detector, PARS can capture the light scattering contrast due to the excitation and detection sources, respectively.

[0047] Throughout this disclosure, excitation pulses generated by pulsed excitation lasers may be described as being on a particular scale. Whenever it is said that an excitation pulse is generated by nanoseconds, it should be understood that it may be generated on the microsecond or picosecond scale as well. For example, a picosecond-scale pulsed excitation laser may induce radiative and non-radiative (thermal and pressure) perturbations in a sample.

[0048] Figure 1 shows a high-level view of a photon absorption remote sensing (PARS) system, which consists of a photon absorption remote sensing (PARS) system (101), an optical combiner (102), and an imaging head (104). The photon absorption remote sensing (PARS) system may further include other systems (e.g., a signal enhancement system), and the optical combiner may combine beams from the photon absorption remote sensing (PARS) system (101) and these other systems.

[0049] Figure 2 shows a high-level diagram depicting the photon absorption remote sensing PARS excitation (202), photon absorption remote sensing PARS detection (204), and optical combiner (203), which can be combined with other systems (e.g., signal enhancement systems) and imaging heads (205).

[0050] 3 shows a high-level embodiment of a photon absorption remote sensing PARS system combined with other modalities (305). It consists of a photon absorption remote sensing PARS system (301), an optical combiner (302), and an imaging head (304), which can be combined with a variety of other modalities (305), such as brightfield microscopy, scanning laser ophthalmoscopy, ultrasound imaging, stimulated Raman microscopy, fluorescence microscopy, two-photon and confocal fluorescence microscopy, coherent anti-Raman Stokes microscopy, Raman microscopy, other photon absorption remote sensing PARS, photoacoustic, and ultrasound systems, among others.

[0051] Figure 4 shows the signal processing path, which consists of a photodetector (401), a signal processing unit (402), a digitizer (403), a digital signal processing unit (404), and a signal extraction unit (405).

[0052] (Total Absorption-Photon Absorption Remote Sensing TA-PARS) When a sample absorbs light, there are a limited number of interactions that can occur. The absorbed energy is converted to temperature and pressure, or to light of different wavelengths. Temperature and pressure signals are captured by the photon absorption remote sensing (PARS) detection beam, while light emission can be detected by a total absorption-photon absorption remote sensing (TA-PARS) system, which can be sensitive to radiative relaxation. In this way, all or nearly all absorption of light by the tissue (whether in the form of generated pressure, non-radiative signals such as generated temperature, radiative relaxation such as fluorescence, multiphoton fluorescence, or stimulated Raman scattering), and / or scattering signals such as localized scattering signals can be captured by photon absorption remote sensing (PARS).

[0053] FIG. 5 illustrates an exemplary architecture for photon absorption remote sensing (PARS) sensitive to radiative relaxation. By way of example, the radiative relaxation can be fluorescence or autofluorescence, but the embodiments disclosed herein are not limited thereto. For example, the radiative relaxation can include Raman scattering, fluorescence, autofluorescence, multiphoton fluorescence, etc. For ease of explanation, an autofluorescence-sensitive total absorption-photon absorption remote sensing (TA-PARS) system will be described as an example with reference to FIG. 5. A multi-wavelength fiber pump laser (5812) is used to generate a photon absorption remote sensing (PARS) signal. The excitation beam (5817) passes through a multi-wavelength unit (5840) and a lens system (5842) to adjust its focus on the sample (5818). The optical subsystem used to adjust the focus can be constructed using components known to those skilled in the art, including, but not limited to, a beam expander, an adjustable beam expander, an adjustable collimator, an adjustable reflective expander, a telescope system, etc.

[0054] The signal signature is interrogated using either a short or long coherence length probe beam (5816) from a detection laser (5814) that is confocalized and co-aligned with the excitation spot on the sample (5818). The interrogation / probe beam (5816) passes through a lens system (5843), a polarizing beam splitter (5844), and a quarter-wave plate (5856) to direct reflected light (5820) from the sample (5818) to a photodiode (5846). However, this architecture is not limited to including a polarizing beam splitter (5844) and a quarter-wave plate (5856). The aforementioned components may be replaced with fiber-based equivalent components, such as circulators, couplers, Faraday rotators, electro-optic modulators, WDMs, and / or double-clad fibers, which are non-reciprocal elements. Such an element may receive light from a first path, but then redirect the light to a second path.

[0055] The interrogation beam (5816) is combined with the excitation beam using a beam combiner (5830). The combined beam (5821) is scanned by a scanning unit (5819). This passes through an objective lens (5855) and is focused onto the sample (5818).

[0056] The reflected beam (5820) returns along the same path. The reflected beam is filtered by a beam combiner / splitter (5831) to separate the detection beam (5816) from any autofluorescence returning from the sample. The autofluorescence (5890) passes through a lens system (5845) to focus it onto an autofluorescence-sensitive photodetector (5891). The isolated detection beam (5820) is transmitted through the beam splitter (5831) toward the signal collection / analysis path, where the returned detection light is redirected by a polarizing beam splitter (5844). The detection path consists of a photodiode (5846), an amplifier (5858), a high-speed data acquisition card (5850), and a computer (5852). The autofluorescence-sensitive photodetector can be any such device, including a camera, photodiode, photodiode array, etc. The autofluorescence detection path may include more beam splitters and photodetectors to further separate and detect light of specific wavelengths.

[0057] Figure 6 shows an example visualization potentially provided by autofluorescence-sensitive total absorption-photon absorption remote sensing (TA-PARS). Collecting any portion of the light returning from the sample, excluding the detection beam, allows for analysis based on wavelength. Isolating specific wavelengths of light emission from the sample allows for visualization of specific molecules of interest. For example, autofluorescence-sensitive photon absorption remote sensing (TA-PARS) can be applied to imaging tissue. Here, photon absorption remote sensing (PARS) excitation is selected to capture nuclear absorption contrast. In this case, UV excitation is used to generate pressure and temperature signals due to nuclei in the tissue. Simultaneously, the autofluorescence contrast generated by the photon absorption remote sensing (TA-PARS) excitation is captured. In this case, non-nuclear regions of the tissue are highly fluorescent. In this way, visualization of nuclear and non-nuclear structures in the tissue is simultaneously provided. Furthermore, the resulting visualization may require only a single (or only one, or exactly one) excitation wavelength to capture. As previously mentioned, this method can be used with other radiative relaxation-sensitive photon absorption remote sensing (TA-PARS) techniques, where radiative relaxation other than autofluorescence can be generated and captured.

[0058] For example, photon absorption remote sensing (PARS) radiative signals can be implemented in a photon absorption remote sensing (PARS) absorption spectrometer to accurately measure the total absorption of light by a sample. Furthermore, radiative relaxation (e.g., autofluorescence in FIG. 5) sensitive photon absorption remote sensing (PARS) can be used to measure the fraction of absorbed energy that is converted to heat and pressure or light, respectively. This can enable sensitive quantum efficiency measurements in a wide range of biological and non-biological samples.

[0059] The total absorption-photon absorption remote sensing TA-PARS signal can be collected on a single (only one or exactly one) detector, as highlighted in Figure 7. Given that the prominent components of the total absorption-photon absorption remote sensing TA-PARS signal can appear distinct from one another, a single detector can adequately characterize these components. For example, the initial signal level (scattering) may indicate the unperturbed intensity reflectance of the detection beam from the sample at the interrogation location, encoding the scattering intensity. Then, following excitation by an excitation pulse (at 100 ns in Figure 7), the photon absorption remote sensing PARS excitation signals associated with nonradiative relaxation (e.g., thermal, temperature) and radiative relaxation (e.g., fluorescence or autofluorescence) can be observed as unique overlapping signals (labeled PA and AF in the figure).

[0060] If these excitation signals are measurably unique from one another (e.g., in amplitude or amplitude and / or evolution time), they can be resolved from the combined signal to extract these amplitudes along with their characteristic lifetimes. This wealth of information improves the available contrast and, by providing additional multiplexing capabilities, may be useful for providing characteristic molecular signatures of the constituent chromophores. In addition, such an approach may offer the practical advantage that only a single detector and a single (only one or exactly one) detection path may be required, significantly reducing the physical hardware complexity and cost. Acquisition of signals over time is discussed in more detail in the section covering time-domain photon absorption remote sensing (TD-PARS).

[0061] Referring to Figure 8, any given photon absorption remote sensing PARS excitation event always generates some radiative and non-radiative relaxation. Total absorption-photon absorption remote sensing TA-PARS facilitates capturing the total chromophore absorption profile. Thermal and pressure perturbations can generate corresponding modulations in local optical properties. Total absorption-photon absorption remote sensing TA-PARS microscopes can capture the scattering and total absorption (radiative and non-radiative relaxation) visualization of the chromophore in a single (only one or exactly one) excitation event. Non-radiative relaxation leads to thermal and pressure-induced modulations, which in turn cause back-reflected intensity fluctuations in the detection beam. The photon absorption remote sensing PARS signal is determined by some change in reflectivity of the incident detection (RI). det ) multiplied by the radiative absorption pathway. The radiative absorption pathway captures light emission due to radiative relaxation, such as stimulated Raman scattering, fluorescence, and multiphoton fluorescence. Emission is a spectrum of several wavelengths and energies of light emission (hv em ) The local scattering contrast is captured as the unmodulated backscatter of the detection beam (pre-excitation pulse). The scattering contrast is calculated by adding the incident detection power (σ s I det )

[0062] In total absorption-photon absorption remote sensing (TA-PARS), nonradiative relaxation-induced modulations are detected at the excitation site by the probe beam. Photon absorption remote sensing (PARS) can then visualize any photothermal heating or photoacoustic pressure that causes modulation of local optical properties. At the same time, total absorption-photon absorption remote sensing (TA-PARS) utilizes an additional detection pathway to capture nonspecific optical emissions from the sample (apart from excitation and detection) regardless of their wavelength, frequency, polarization, or other characteristics. These emissions can then be attributed to any radiative relaxation effects, such as stimulated Raman scattering, fluorescence, and multiphoton fluorescence.

[0063] Using this detection pathway, sensitivity to any range of chromophores can be enhanced. Unlike conventional modalities that independently incorporate portions of radiative or nonradiative absorption, in total absorption-photon absorption remote sensing (TA-PARS), contrast may not be constrained by efficiency factors such as photothermal conversion efficiency or fluorescence quantum yield. By incorporating nonradiative and radiative absorption contrast in addition to scattering for excitation and detection, total absorption-photon absorption remote sensing (TA-PARS) can incorporate all or nearly all optical properties of chromophores, such as absorption coefficients, scattering coefficients, quantum efficiency, and nonlinear interaction coefficients, providing sensitivity to most chromophores simultaneously.

[0064] (Quantum Efficiency Ratio (QER) and Label-Free H&E Visualization) Incorporating both radioactive and non-radiative absorption fractions can also provide additional information. Total absorption-photon absorption remote sensing (TA-PARS) can provide an absorption metric proposed as the quantum efficiency ratio (QER), which visualizes the proportional radioactive and non-radiative absorption response of biomolecules. Total absorption-photon absorption remote sensing (TA-PARS) can effectively provide label-free hematoxylin and eosin (H&E)-like visualization, providing label-free visualization of various biomolecules that allows for a convincing analog to traditional histochemical staining of tissues.

[0065] The quantum efficiency ratio (QER) is a measure of the non-radiative photon absorption remote sensing (PARS) (P nr ) for the radioactive photon absorption remote sensing PARS signal (P r ) can be defined as the ratio of

[0066]

number

[0067] This ratio is specific to a given chromophore. For example, a biomolecule such as collagen exhibits high emissive contrast and low non-emissive contrast, providing a high quantum efficiency ratio (QER). Conversely, DNA exhibits low emissive contrast and high non-emissive contrast, providing a high quantum efficiency ratio (QER). Calculating the quantum efficiency ratio (QER) from emissive and non-emissive absorptions can allow properties such as chromophore composition, density, and quantity to be extracted in a single (only one or exactly one) event. This can also enable single-shot functional imaging.

[0068] For example, a picosecond-scale pulsed excitation laser can induce radiative and non-radiative (thermal and pressure) perturbations in a sample. The thermal and pressure perturbations generate corresponding modulations in local optical properties. A secondary probe beam confocalized with the excitation can capture non-radiative absorption-induced modulations to local optical properties as changes in backscattering intensity.

[0069] These backscattering modulations can be directly correlated to local nonradiative absorption contrast. Due to the nature of the probe architecture, undisturbed backscatter (pre-excitation events) also incorporate scattering contrast as seen by the probe beam. Unlike traditional photoacoustic methods, rather than relying on pressure waves to propagate through the sample before being detected via an acoustic transducer, total absorption-photon absorption remote sensing TA-PARS probes can instantaneously detect induced modulations at the excitation site. Thus, total absorption-photon absorption remote sensing TA-PARS offers noncontact operation, facilitating the imaging of delicate and sensitive samples that would otherwise be impractical for imaging with traditional contact-based PAM methods.

[0070] Because TA-PARS relies solely on the generation of heat and subsequent pressure to provide contrast, the absorption mechanism is nonspecific and highly sensitive to small changes in relative absorption. This allows PARS to detect any type of absorption mechanism, including vibrational absorption, stimulated Raman absorption, and electronic absorption. Previously, PARS demonstrated label-free, nonradiative absorption contrast for hemoglobin, DNA, RNA, lipids, and cytochromes in specimens such as chicken embryo models, excised tissue specimens, and live mouse models. In TA-PARS, a unique secondary detection pathway captures radiative relaxation contrast in addition to nonradiative absorption. The radiative absorption pathway was designed to comprehensively collect all optical emission at any wavelength of light, excluding excitation and detection. As a result, the radiative detection pathway captures nonspecific optical emission from the sample, regardless of its wavelength, frequency, polarization, and other characteristics.

[0071] 9 , to improve the sensitivity of total absorption-photon absorption remote sensing TA-PARS and thereby facilitate detection of radiative absorption contrasts, a total absorption-photon absorption remote sensing TA-PARS 900 can include excitation at distinct first and second excitation wavelengths (e.g., 266 nm and 515 nm excitation) to provide sensitivity to DNA, heme proteins, NADPH, collagen, elastin, amino acids, and various fluorescent dyes. The total absorption-photon absorption remote sensing TA-PARS can include a specific optical path with a dichroic filter and an avalanche photodiode to separate and detect radiative absorption contrasts. As illustrated in FIG. 9 , the total absorption-photon absorption remote sensing TA-PARS system can include excitation at a first excitation wavelength from a first excitation source 920 (e.g., visible light, such as 515 nm visible excitation) and excitation at a second excitation wavelength from a second excitation source 940 (e.g., UV light, such as 266 nm UV excitation). The first excitation source 920 may include a first excitation laser 902, such as a 50 kHz-2.7 MHz, 2 ps pulsed, 1030 nm fiber laser (e.g., YLPP-1-150-v-30, IPG Photonics), although the embodiments disclosed herein are not limited thereto. The second harmonic may be generated by a lithium triborate crystal or LBO 922. The first (e.g., 515 nm) harmonic may be separated via a dichroic mirror 906 and then spatially filtered with a pinhole 908 before being used in the imaging system. The first excitation source 902 may include one or more lenses or plates, such as a half-wave plate or HWP 924, a filtering lens, and / or a lens assembly 928, disposed between the LBO 922 and the first excitation laser 902. The pinhole 908 may be disposed between two lenses or lens assemblies 928, as an example.

[0072] The second excitation source 940 may comprise a second excitation laser 904, such as a 50 kHz, 400 ps pulsed diode laser (e.g., Wedge XF266, RPMC), but the embodiments disclosed herein are not limited thereto. The output from the second excitation laser 904 may be separated from residual excitation (e.g., 532 nm excitation) using a prism 910 and then expanded (e.g., using a variable beam expander or VBE 926) before use in the imaging system.

[0073] The total absorption-photon absorption remote sensing TA-PARS system may include a detection system 950 shared between a first excitation source 920 and a second excitation source 940. As illustrated in FIG. 9 , the total absorption-photon absorption remote sensing TA-PARS detection system 950 may include a probe beam 912, which may include a 405 nm laser diode, such as a 405 nm OBIS-LS laser (OBIS_LS405, Coherent). Here, detection may be fiber coupled to the system via a circulator 914, where detection may be combined with excitation via one or more dichroic mirrors 916 and / or directed via mirror 934. The combined excitation and detection may be confocalized onto the sample using a lens 918, such as a 0.42 NA UV objective. Back-reflected detection from the sample may return to the circulator 914 by the same path as forward propagation. Back-reflection detection includes photon absorption remote sensing (PARS) non-radiative absorption contrast as nanosecond-scale intensity modulation that can be captured by a photodiode. The detection system 950 can also include a collimator and / or collimator assembly 936 for collimating the detected light.

[0074] This probe wavelength offers improved scattering resolution, which improves the confocal overlap between the photon absorption remote sensing (PARS) excitation and detection points on the sample. When combined with a circulator-based probe beam path and an avalanche photodetector, total absorption-photon absorption remote sensing (TA-PARS) offers improved sensitivity compared to previous implementations. The visible wavelength probe also offers improved compatibility between visible and UV excitation wavelengths.

[0075] The radiative relaxation from each of the first and second excitations (266 nm and 515 nm excitations) can be captured independently with different (or first and second) photodiodes 930 and 932. The radiative relaxation induced from the first excitation (515 nm induced radiative relaxation) can be isolated with a dichroic mirror 916 and then captured using the first photodiode 930. The radiative relaxation induced from the second excitation (266 nm induced radiative relaxation) can be isolated by redirecting some portion (e.g., 1%-50%) of the total light intensity returned from the sample towards a photodetector and / or second photodiode 932. This light can then be spectrally filtered (e.g., via lens assembly 936) to remove residual excitation and detection before measurement.

[0076] To form an image, a mechanical stage can be used to scan the sample through the objective lens. The excitation sources 920 and 940 can be pulsed continuously (e.g., at 50 kHz), while the stage speed can be adjusted to achieve the desired pixel size (interval between interrogation events). Each time the excitation lasers 902 and / or 904 are pulsed, a collection event can be triggered. During the collection event, a high-speed digitizer (e.g., RZE-004-200, Gage_Applied) can be used to collect hundreds of nanosecond segments from four input signals. These signals can include the laser input reference measurements (excitation and detection), the photon absorption remote sensing PARS scattering signal, the photon absorption remote sensing PARS nonradiative relaxation signal, the photon absorption remote sensing PARS radiative relaxation signal, and the position signal from the stage. The time-resolved scattering, absorption, and position signals can then be compressed into a single characteristic feature. This helps significantly reduce the amount of data acquisition during collection.

[0077] To reconstruct absorption and scattering images, the raw data can be fitted to an orthogonal grid based on the location signal in each study. The raw images can then be Gaussian filtered based on the histogram distribution and rescaled before visualization.

[0078] The fidelity of TA-PARS visualization is assessed by a one-to-one comparison with conventional H&E stained images. The TA-PARS total absorption and quantum efficiency ratio (QER) contrast mechanisms are also validated with a range of dyes and tissue samples. Results show a high correlation between the radiative relaxation properties and the quantum efficiency ratio (QER) measured by TA-PARS for a variety of fluorescent dyes and tissues. These quantum efficiency ratio (QER) visualizations are used to extract regions of specific biomolecules, such as collagen, elastin, and nuclei, in tissue samples. This enables the realization of a broadly applicable high-resolution absorption contrast microscopy system. TA-PARS can provide unprecedented label-free contrast in any variety of biological specimens, providing visualizations inaccessible by other methods.

[0079] Figure 10 shows a comparison of three different contrasts (image (a) nonradiative absorption, image (b) radiative absorption, and image (c) scattering) provided by the total absorption-photon absorption remote sensing TA-PARS system using 266 nm excitation in thin sections of formalin-fixed, paraffin-embedded (FFPE) human brain tissue. The nonradiative relaxation signal was captured based on nanosecond-scale pressure- and temperature-induced modulations in the collected backscattered 405 nm detection beam from the sample. The radiative absorption contrast was captured as light emission from the sample, excluding the excitation and detection wavelengths blocked by optical filters. Simultaneously, the unperturbed backscattering of the 405 nm probe captured localized light scattering from the sample. This contrast captured most of the prominent tissue structures. The nonradiative absorption contrast primarily highlighted nuclear structures, while the radiative contrast captured extranuclear features. The light scattering contrast captured the morphology of the thin tissue section. In resected tissue, this scattering contrast became less applicable and was therefore not investigated in other samples.

[0080] Figure 11 shows an example of total absorption-photon absorption remote sensing TA-PARS imaging. In image (a), total absorption-photon absorption remote sensing TA-PARS captured the epithelial layer at the boundary of excised human skin tissue. The stratum corneum was simultaneously captured with radioactive and non-radioactive visualization. Radioactive visualization provides improved contrast in retrieving these tissue layers compared with non-radioactive imaging. In another subcutaneous region of excised human skin tissue in image (b), total absorption-photon absorption remote sensing TA-PARS captured connective tissue with sparse nuclei and elongated fibrin features.

[0081] The disclosed system was also applied to imaging resected, intact rat brain tissue. In Image I, the total absorption-photon absorption remote sensing (TA-PARS) acquisition highlights the gray matter layer in the brain, revealing dense regions of nuclear structures. The nuclei in the gray matter layer present higher contrast to the surrounding tissue in the non-radioactive image compared to the radioactive representation. Because nuclei do not provide significant radioactive contrast, nuclear structures in the radioactive image appear as voids or lack of signal within the specimen. While some potential nuclei can be observed, they may not be identified with significant confidence compared to those in the total absorption-photon absorption remote sensing (TA-PARS) non-radioactive representation. Along the top right of the non-radioactive acquisition, it is possible to identify structures resembling myelinated neurons surrounding the more sparsely packed nuclei in that area.

[0082] In image (d), further acquisition of adjacent regions highlights obvious myelinated neuronal structures. Dense structures representing overlapping and interconnected dendritic and axonal webs are evident within these regions, and tightly woven neuronal processes arranged around voids within the tissue are observed. Then, in image (e), zooming out to a larger, nearby imaging field, a different tissue section was recovered with non-radioactive contrast. The left side of the field contains dense bundles representing myelin processes into the underlying gray matter, which, in contrast to the right side, has larger nuclei, which is the underlying white matter, which has more myelinated structures with reduced nuclear density.

[0083] Referring to Figure 12, the quantum efficiency ratio, QER, or the ratio of non-radiative to radiative absorption fractions, is expected to contain additional biomolecular-specific information. Ideally, the local absorption fraction should directly correlate to the radiative relaxation properties. Relative radiative and non-radiative signal intensities can be plotted, and the quantum efficiency ratio, QER, can be plotted against the reported quantum efficiency (QE) values.

[0084] In one example, total absorption-photon absorption remote sensing (TA-PARS) was applied to measure a series of fluorescent dyes with various quantum efficiencies, using 515 nm excitation to generate simultaneously captured radiative and non-radiative relaxation signals.

[0085] An example of the relative radiative and non-radiative signal intensities was plotted as shown in the image in Figure 12 (Figure 12A). The quantum efficiency ratio QER was then plotted against the reported quantum efficiency QE value of the sample as shown in the image (Figure 12B). The radiative photon absorption remote sensing PARS signal (P r ) is QE(P r ∝QE), but the non-radiative photon absorption remote sensing (PARS) nr ) signal is QE(P nrTherefore, the fractional relationship between the non-radiative and radiative signals is expressed by a quotient of linear functions (quantum efficiency ratio QER = P r / P nr ∝QE / (1-QE)). The empirical results fit this predicted model well (R = 0.988).

[0086] Figure 13 illustrates images from the quantum efficiency ratio QER acquisition process applied to imaging thin sections of formalin-fixed, paraffin-embedded FFPE human tissue. Quantum efficiency ratio QER images were generated by calculating the quantum efficiency ratio QER for each image pixel based on the non-radioactive and radioactive signals. The results represent a dataset that encodes chromophore-specific attributes in addition to independent absorption fractions. The quantum efficiency ratio QER process helps further separate otherwise similar tissue types from only radioactive or non-radioactive acquisitions.

[0087] The colorized version of the QER image shown in Figure 13 highlights various tissue components. Low QER biomolecules (e.g., DNA, RNA) may appear as the first color (e.g., a color having a lower wavelength or light blue), while high QER biomolecules (e.g., collagen, elastin) may appear as the second and / or third colors (e.g., having higher wavelengths) different from the first color (e.g., pink and purple). Compared to H&E visualization captured after the QER imaging session (Figure 13, image (c-ii)), collagen and elastin (which may appear as the fourth color or dark red) that make up fibrous connective tissue may be easier to identify due to their low QER. Conversely, nuclear structures are prominent in the first and / or fifth colors (e.g., blue) due to their high QER. Connective tissue surrounding cancer cells is also differentiated from fibrous connective tissue in a sixth color (e.g., purple) in QER visualization compared to H&E staining images. When calculating QER from total absorption-photon absorption remote sensing (TA-PARS), complementary imaging contrast is provided, allowing for greater chromophore specificity than is accessible independently with radioactive or non-radioactive modalities. While first, second, third, fourth, fifth, and sixth colors are used, the embodiments disclosed herein may not be limited to a predetermined number of colors, such as six. The colors appearing in the visualization may have wavelengths proportional to the QER. For example, structures with higher QER may appear as colors with higher wavelengths (e.g., red), and structures with lower QER may appear as colors with lower wavelengths (e.g., blue).

[0088] The quantum efficiency ratio (QER) method presented here relies on extracted intensity values, but similar analogs can be envisaged involving similar such ratios of other signal parameters such as lifetime, rise time, signal shape, frequency content, etc.

[0089] (Unlabeled histological imaging) The total absorption-photon absorption remote sensing (TA-PARS) mechanism may offer the opportunity to accurately emulate traditional histochemical stain contrast, such as H&E staining, and total absorption-photon absorption remote sensing (TA-PARS) may provide label-free histological imaging. Nonradioactive total absorption-photon absorption remote sensing (TA-PARS) signal contrast may be similar to that provided by hematoxylin staining, while radioactive total absorption-photon absorption remote sensing (TA-PARS) signal contrast may be similar to that provided by eosin staining. Total absorption-photon absorption remote sensing (TA-PARS) may capture label-free features such as fat cells, fibrin, connective tissue, neuronal structures, and cell nuclei. Visualization of subnuclear structures may be captured with sufficient clarity and contrast to identify individual atypical nuclei.

[0090] Figure 14 shows an example of label-free histological imaging applied to formalin-fixed, paraffin-embedded, FFPE human brain tissue. Referring to Figure 14, the non-radioactive total absorption-photon absorption remote sensing (TA-PARS) signal contrast is similar to that provided by hematoxylin staining of cell nuclei (Figure 14, image (a)). A section of formalin-fixed, paraffin-embedded, FFPE human brain tissue was imaged with non-radioactive photon absorption remote sensing (PARS) (Figure 14, image (ai)). This non-radioactive information was then colorized to emulate the contrast of hematoxylin staining (Figure 14, image (a-ii)). The same tissue section was then imaged under a bright-field microscope by staining only with hematoxylin (Figure 14, image (a-iii)), providing a direct one-to-one comparison. The primary target of hematoxylin staining and the non-radioactive portion of total absorption-photon absorption remote sensing TA-PARS is the nucleus, but other chromophores also contribute to some extent, so their visualization is expected to be very similar.

[0091] A similar approach was applied to eosin staining of adjacent sections. The adjacent sections were imaged with radioactive photon absorption remote sensing (PARS) (Figure 14, image (bi)). This radioactive information was then colorized to emulate the contrast of eosin staining (Figure 14, image (b-ii)). The sections were then stained with eosin (Figure 14, image (b-iii)), providing a direct one-to-one comparison of the radioactive contrast and eosin staining. In each of the total absorption-photon absorption remote sensing (TA-PARS) and eosin-stained images, similar microvasculature and red blood cells were resolved throughout the brain tissue. These visualizations are expected to closely mirror the chromophores targeted by eosin staining of extranuclear material, since the primary targets of the radioactive moieties in total absorption-photon absorption remote sensing (TA-PARS) include hemeproteins, NADPH, flavins, collagen, elastin, and the extracellular matrix.

[0092] Because the distinct contrast mechanisms of total absorption-photon absorption remote sensing (TA-PARS) closely emulate the visualization of H&E staining, the disclosed system can provide true H&E-like contrast in a single acquisition. Total absorption-photon absorption remote sensing (TA-PARS) can provide substantially improved visualization compared to previous photon absorption remote sensing (PARS)-emulating H&E systems, which rely on scattering microscopy to estimate eosin-like contrast. Scattering microscopy-based methods cannot provide clear images in complex, scattering samples, such as bulk excised human tissue. In contrast, total absorption-photon absorption remote sensing (TA-PARS) can directly measure extranuclear chromophores via a radioactive contrast mechanism, thus providing similar contrast to H&E, regardless of specimen morphology. Here, distinct total absorption-photon absorption remote sensing (TA-PARS) visualizations were combined using linear color blending to produce an effective representation of conventional H&E staining in unstained tissue.

[0093] An example of excised formalin-fixed, paraffin-embedded FFPE human brain tissue is shown in Figure 14, image (c). The wide-field image highlights the border between cancerous and healthy brain tissue. To qualitatively compare total absorption-photon absorption remote sensing TA-PARS with conventional H&E imaging, a series of human breast tissue sections were scanned with total absorption-photon absorption remote sensing TA-PARS (Figure 14, image (di) and Figure 14, image (ei)), then stained with H&E dye and imaged under brightfield microscopy (Figure 14, image (d-ii) and Figure 14(e-ii)). The total absorption-photon absorption remote sensing TA-PARS-emulated H&E visualization is virtually identical to the H&E preparation. In both images, clinically relevant features of metastatic breast lymph node tissue are equally accessible.

[0094] (Lifetime Imaging) H&E simulations can be enhanced by extracting time-domain features, which are discussed in more detail in the following sections considering time-domain photon absorption remote sensing (TD-PARS) and feature extraction imaging. The full amplitude of the photon absorption remote sensing (PARS) modulation captures the local absorption of the excitation, but the evolution of the pressure- and temperature-induced modulation also captures the local material properties.

[0095] Figure 15 illustrates the evolution of a photon absorption remote sensing PARS signal over time. Each photon absorption remote sensing PARS excitation event incorporates scattering from the detection and excitation sources, radiative emission, and a photon absorption remote sensing PARS nonradiative relaxation time-domain signal. Referring to Figure 15, photon absorption remote sensing PARS decay or evolution time is likely tied to metrics such as thermal and pressure limit times that govern conventional photoacoustic imaging. This means that properties such as thermal diffusivity, conductivity, and speed of sound can determine the photon absorption remote sensing PARS relaxation time. By measuring the decay or evolution time, photon absorption remote sensing PARS can then provide additional chromophore-specific information about the specimen. This may enable demixing of chromophores from a single excitation event (e.g., detection, separation, or otherwise discretization of constituent species and / or subspecies) or single-shot functional imaging.

[0096] An example of a lifetime photon absorption remote sensing PARS image in resected rat brain tissue is shown in Figure 16, where nuclei (which may appear as a first color, such as white) are unmixed from surrounding gray matter (which may appear as a second color, such as green) and interwoven myelinated neuronal structures (which may appear as a third color, such as orange). This unmixing is performed based on the photon absorption remote sensing PARS lifetime signal.

[0097] Referring to Figure 17, a rapid lifetime extraction technique can be used to significantly improve photon absorption remote sensing PARS acquisition contrast. Referring to Figure 17, photon absorption remote sensing PARS amplitude can be calculated as the difference between the average pre-excitation and post-excitation signals. This acquisition is less sensitive to imaging noise compared to alternative extraction techniques. Previously, photon absorption remote sensing PARS has extracted the photon absorption remote sensing PARS signal using a (minimum-maximum) acquired signal approach. By taking the minimum minus the maximum value of the signal, photon absorption remote sensing PARS can highlight the full amplitude of the photon absorption remote sensing PARS modulation. However, this is highly susceptible to acquisition and measurement noise in the photon absorption remote sensing PARS signal.

[0098] One possible signal extraction method is performed by determining the average pre-excitation signal. The average post-excitation signal is then calculated from the initial portion of the lifetime signal. The photon absorption remote sensing PARS amplitude is then calculated as the difference between the two average signals. This metric for rapid signal extraction provides a substantial improvement in signal-to-noise ratio and sensitivity when collecting photon absorption remote sensing PARS signals. Because this technique relies on the average signal, photon absorption remote sensing PARS collection is substantially less sensitive to acquisition noise.

[0099] Additional time-based imaging methods are considered in more detail in the following sections: Time Domain-Photon Absorption Remote Sensing, TD-PARS, and Feature Extraction Imaging. First, we briefly consider two other photon absorption remote sensing methods, PARS.

[0100] (Multipath-Photon Absorption Remote Sensing MP-PARS) 18, in multi-path photon absorption remote sensing (MP-PARS), a backscattered detection can be taken and then redirected back to the sample where it interacts with the sample again before being detected. Each time the detection interacts with the sample, it can pick up additional information in the photon absorption remote sensing PARS modulation.

[0101] In photon absorption remote sensing (PARS), non-radiative absorption-induced perturbations in optical properties are visualized using a secondary confocal detection laser. The detection laser is confocalized with the excitation spot so that absorption-induced modulations can be captured as changes in the backscattering intensity of the detection laser. For a given detection intensity I det For PARS, before the excitation pulse interacts with the sample, the signal is allowed to approximate based on the following relationship: pre-ext ∝I det (R), where R is the undisturbed reflectance of the sample.

[0102] When the excitation pulse interacts with the sample, the signal can be approximated as: post-ext ∝I det (R+ΔR), where ΔR denotes the pressure and temperature induced change in reflectivity. The total photon absorption remote sensing (PARS) absorption contrast is then approximated as: sig ∝PARS post-ext -PARS pre-ext . Here PARS pre-ext and PARS post-ext Substituting the previous relation into PARS, we get: sig ∝I det (R+ΔR)-I det (R).

[0103] Then, before the excitation pulse, the backscattering of the multipath-photon absorption remote sensing MP-PARS is approximated based on the following relationship: MPPARS pre-ext ∝(I det (R)) nwhere R is the unperturbed reflectivity of the sample and n is the number of times the excitation interacts with the sample. When the excitation pulse interacts with the sample, the signal can be approximated as: MPPARS post-ext ∝(I det (R+ΔR) n where the pressure and temperature induced change in reflectance is represented by ΔR.

[0104] The total multipath-photon absorption remote sensing MP-PARS absorption contrast is then approximated as: MPPARS sig ∝MPPARS post-ext -MPPARS pre-ext Here, MPPARS for the previous relationship pre-ext and MPPARS post-ext Substituting MPPARS gives: sig ∝(I det (R+ΔR) n -(I det (R)) n , where n is the number of times the detector interacts with the sample. The photon absorption remote sensing (PARS) signal can be nonlinearly expanded by these repeated interactions of the backscatter detector with the sample. The detector can then be redirected to interact with the sample any number of times, resulting in a corresponding degree of nonlinear expansion in the nonradiative absorption contrast.

[0105] Multi-path photon absorption remote sensing MP-PARS architectures, such as architecture 1800 illustrated in FIG. 18, can be oriented so that a path consists of a reflection or transmission event. This can occur at normal incidence to the sample, or at several related transmission or reflection angles. For example, if a target is characterized by a particularly strong Mie scattering angle, it can be advantageous to orient multiple paths along this direction. The multiple paths can occur along a single (only one or exactly one) path (such as a normal incidence reflection), or along multiple paths, such as a normal incidence transmission architecture, or along an architecture having additional (two or more) paths to exploit additional spatial nonlinearities.

[0106] For example, multi-path-photon absorption remote sensing MP-PARS architecture 1800 may include an excitation source 1802 (e.g., a 266 nm excitation source or laser), one or more detection sources 1804 (e.g., a 405 nm detection source or laser), one or more photodiodes or photodetectors 1806, a circulator 1808, a collimator 1810, one or more mirrors 1810 for directing the excitation and / or detection light, a prism 1816, and a variable beam expander 1818. Additionally, multi-path-photon absorption remote sensing MP-PARS architecture 1800 may include a pair of alignment mirrors 1820 for aligning the excitation and / or detection light, and one or more scanners or scan heads 1822, 1824 positioned on different sides of the sample. The one or more scanners may include a first scanner 1822 for transmitting excitation and detection light to the sample and a second scanner 1824 arranged with mirrors 1826 to allow multiple passes. A computer 1828 analyzes received signals and / or may be used to control the excitation and detection sources 1802 and 1804.

[0107] Multipath-photon absorption remote sensing (MP-PARS) can be enabled to function as an optical amplifier of the detected photon absorption remote sensing (PARS) signal. It can be used in the same way as a laser cavity system or photomultiplier tube is implemented to further improve the sensitivity of the measurement signal. This could result in significant improvements in the fidelity of photon absorption remote sensing (PARS) imaging. Photon absorption remote sensing (PARS) can be incorporated with improved sensitivity to any or all of radiative, non-radiative, or scattering contrast, facilitating acquisition at lower imaging powers. This may facilitate acquisition of lower concentrations of chromophores, chromophores with lower optical absorption, or reduce sample exposure. These nonlinear effects can be exploited to improve the recovered imaging resolution by exploiting nonlinear spatial dependence to provide super-resolution imaging.

[0108] (Multiphoton Excitation Photon Absorption Remote Sensing (PARS)) Referring to Figure 19, multiphoton-photon absorption remote sensing (PARS) may offer several advantages over conventional photon absorption remote sensing (PARS) excitation. In multiphoton excitation, multiple photons are absorbed by the target at substantially the same instant and / or in a single (only one or exactly one) event. The energies of these photons are then added together such that the absorbed photon is equivalent to a single (only one or exactly one) higher energy and shorter wavelength photon. Here, two photons, having half the energy and twice the wavelength of the single-photon excitation event, are absorbed by a chromophore providing similar excitation.

[0109] Photon absorption remote sensing (PARS) can exploit nonlinear absorption mechanisms similar to those in fluorescence microscopy. Traditionally, photon absorption remote sensing (PARS) targets single-photon absorption effects, such as 266 nm UV excitation of DNA. However, photon absorption remote sensing (PARS) can also target multiphoton absorption properties, such as those used in multiphoton fluorescence microscopy. In multiphoton microscopy, multiple photons are absorbed by the target at essentially the same instant. The energies of these photons are then added together so that the absorbed photon is equivalent to a single higher-energy and shorter-wavelength photon.

[0110] For two-photon-photon absorption remote sensing (PARS), the excitation wavelength is selected as twice the conventional value. Two photons are then absorbed simultaneously, providing an excitation event equivalent to standard one-photon excitation (Figure 19). In the example listed above, rather than targeting DNA with 266 nm UV excitation, it is possible to target DNA absorption with 532 nm excitation. Two photons of 532 nm absorption are equivalent to a single 266 nm absorption. The embodiments disclosed herein are not limited to 532 nm excitation. The wavelength of excitation can be configured to be twice the predetermined excitation wavelength, such as twice the UV wavelength (e.g., twice 100-400 nm) or UVC wavelength (100-280 nm).

[0111] One of the key differences between multiphoton-photon absorption remote sensing (PARS) and conventional single-photon absorption remote sensing (PARS) architectures is the requirement for high instantaneous optical energy density. To minimize sample exposure levels to practical levels, this architecture may require the use of very short optical excitation pulses, on the order of 1 picosecond or less. Such a requirement may be unique to multiphoton-photon absorption remote sensing (PARS).

[0112] Multiphoton-photon absorption remote sensing (PARS) may offer several advantages over conventional photon absorption remote sensing (PARS) excitation. First, multiphoton excitation uses lower-energy, longer-wavelength photons that penetrate deeper. Second, moving toward longer wavelengths may offer additional biocompatibility, avoiding tissue damage. This is particularly prevalent in the case of in situ tissue structures, as photon absorption remote sensing (PARS)-UV excitation may not be compatible with imaging deep within the body. It may also improve the safety of photon absorption remote sensing (PARS) systems for in situ applications.

[0113] (Time Domain Photon Absorption Remote Sensing TD-PARS and Feature Extraction Imaging) Photon absorption remote sensing (PARS) works by capturing nanosecond-scale optical perturbations generated by photoacoustic pressure or photothermal temperature signals. These time-domain (TD) modulations are typically projected by amplitude to determine absorption magnitude. A single characteristic intensity value can be extracted from each time-domain TD signal to visualize the total absorption magnitude at each point. For example, the time-domain TD amplitude, calculated as the difference between the maximum and minimum values ​​of the time-domain TD signal, is commonly used to represent absorption amplitude.

[0114] However, important information about the target's material properties is contained in the time-domain TD signal. The time evolution of the PARS signal can be determined by material properties such as density, heat capacity, and acoustic impedance. H&E-like visualizations can be generated directly from PARS time-domain data by using machine learning algorithms that bypass the PARS image reconstruction step. This approach is beneficial compared to direct PARS-H&E image-to-image conversion because it provides additional valuable information that helps better distinguish different tissue types within the image.

[0115] Referring to Figures 20A and 20B, H&E-like representations can be generated by applying AI image-to-image translation algorithms based on deep neural network architectures, such as generative adversarial networks (GANs), conditional generative adversarial networks (cGANs), or cycle-consistent adversarial networks (cycleGans). These methods learn color transition mappings from paired or unpaired samples of source and reference representations. The reconstructed grayscale photon absorption remote sensing (PARS) image (20A) can then be mapped to color H&E data (20B).

[0116] Imaging modalities may scan pixel by pixel, capturing a signal at each pixel over time. While the time-lapse scan may be continuous in nature, the signal is recorded periodically or discretely using an image acquisition system. Characteristic values ​​may be extracted from each signal, which is accomplished either by using the Hilbert transform to find the signal envelope, from which the difference between the maximum and minimum values ​​may be calculated, or by directly calculating the difference between the maximum and minimum values ​​of the raw signal itself.

[0117] 21A and 21B, the methods and techniques disclosed herein can bypass the image reconstruction stage in which an image is reconstructed by extracting the amplitude of the captured optical absorption signal or averaging those values ​​over time. Instead of the pixels of the reconstructed image, the methods and techniques disclosed herein can directly use the signal representation as input to an artificial intelligence-based colorization algorithm. In this way, additional valuable information about the underlying tissue can be included to create a virtual H&E-like image.

[0118] To make the colorization algorithm more computationally efficient, several compressed representations of the time-domain signal may be used. These may include, for example, but not limited to, the signal's main linear components, coefficients of other signal decomposition methods, prominent signal points, etc. Such techniques reduce the dimensionality of the dataset, enhancing interpretability while minimizing information loss. An example of creating an H&E-like visualization by applying the Pix2Pix algorithm is shown in FIGS. 21A and 21B. FIG. 21A shows the three main components of the time-domain signal, and FIG. 21B shows the corresponding composite H&E image. The differences between FIGS. 20A-20B and 21A-21B may not be immediately apparent in black and white and may be better appreciated in color form. For example, FIG. 21A may show some colorization, while FIG. 20A may be black and white and / or grayscale. Additionally, FIG. 21B may have less granularity and / or show more color than FIG. 20B.

[0119] (Intelligent clustering method) Using unsupervised clustering techniques, colorized composite H&E images can be created without reconstructing grayscale images. Clustering techniques can learn time-domain TD features related to underlying biomolecular properties. This technique identifies features related to constituent biomolecules, enabling single-acquisition virtual tissue labeling. Colorized visualizations of tissues are generated, highlighting specific tissue components. Clustering can be performed on any or all of the photon absorption remote sensing (PARS) emissive, non-emissive, and scattering channels.

[0120] For a given biomolecule with consistent material properties, the photon absorption remote sensing (PARS)-time domain TD signal may have a specific shape. However, the signal from a given target may vary in amplitude (e.g., based on concentration) and may suffer from noise. By clustering the signals by shape and learning prototypes associated with each cluster, they can be used to determine constituent time domain features that capture material-specific information of the underlying tissue target, regardless of the noise and amplitude variations present in the time domain TD signal.

[0121] As an example, a modified K-means clustering method can be used. Measured signals are treated as vectors, and the vector angle resembles the signal shape. The distance or difference between time-domain TD signals is the sine of the tilt angle, so that orthogonal signals have maximum distance and scaled or inverted signals have zero distance. Cluster centroids are then calculated as the first principal component of the union of each cluster and its negative, making the learned centroids robust to noise. Once the time-domain TD features (centroids) are learned, the corresponding feature amplitudes are extracted by performing a basis change from the time domain to the feature domain.

[0122] Referring to FIG. 22, which illustrates an exemplary architecture 2200, broadly absorbed UV excitation (e.g., 266 nm) can target several biomolecules, such as collagen, elastin, myelin, DNA, and RNA, with a single excitation (only one or exactly one). A clustering approach can then be used to create a visualization of the enhanced absorption contrast and extract biomolecular features from the time-domain TD signal. UV excitation can be provided by an excitation light source 2202, such as a 50 kHz, 266 nm laser (e.g., WEDGE_XF266, Bright_Solutions). The excitation can be spectrally filtered with a prism 2204 and then expanded (e.g., with a variable beam expander or VBE 2206) before combining with the detection beam. The excitation light can be directed through one or more mirrors 2208.

[0123] Detection light may be provided by a detection light source 2212, such as a continuous wave 405 nm OBIS_LS laser. The detection light may be fiber coupled through a circulator 2214, collimated (e.g., with a collimator 2216), and then combined with the excitation beam via a dichroic mirror 2210. The detection light may be directed through one or more mirrors 2218.

[0124] The combined excitation and detection passes through a pair of alignment mirrors 2200 and can be confocalized onto the specimen through a UV-transparent window. Back-reflected light from the sample can return to the collimator 2216 and circulator 2214 via the same path as the forward propagation. The circulator 2214 can redirect the backscattered light to a photodiode 2222, which captures nanosecond-scale intensity modulation. During image acquisition, the stage 2226 can raster scan the specimen over the objective while the excitation is continuously pulsed. The analog photodiode output can be captured for each excitation event using a high-speed digitizer to form a photon absorption remote sensing PARS-time domain TD signal. Using the stage position signal, each photon absorption remote sensing PARS-time domain TD signal can then be mapped to a pixel in the final image, which can be output on an electronic display and / or computer 2228.

[0125] Referring to Figures 23-24, instead of defining pixel values ​​by time-domain TD signal amplitude, K-means may utilize time-domain TD features depending on the number of extracted clusters. If only a single feature is required (K = 1), the clustering algorithm yields a feature containing similar time-domain TD shapes across all tissue components. This feature can then be used as the basis for matched filtering, a technique designed to optimally extract the amplitude of a known signal shape in the presence of additive noise. This provides a robust, noise-resistant method for measuring absorption amplitude or pixel "brightness." When applied to tissue, this extraction provides a very substantial improvement in structural image quality and noise suppression compared to conventional time-domain TD amplitude projections, as shown in image (a) of Figure 24.

[0126] If additional clusters are required (K>1), tissue-specific time-domain features are learned. In this case, the feature amplitude at each pixel is extracted by performing a basis change from the time domain to the feature domain. To visually illustrate the effectiveness of the learned features, the time-domain signal was clustered for the K=2 required features. Projecting the high-dimensional time-domain data onto a two-dimensional plane containing the learned features allows for visualization of the time-domain TD signal (dots) relative to the identified features (arrows). In the visualization, each dot is colored proportionally to the signal content due to its constituent features.

[0127] Three features (K=3) are used to generate further visualizations of excised mouse brain tissue. Extracted feature amplitudes are mapped to independent red, green, and blue (R, G, B) color channels to form a colorized visualization. Thus, pixel color represents a proportional mix of each feature's contribution to the time-domain signal, while intensity represents the overall magnitude of absorbed energy. Referring to Figure 24, image (c), the K=3 colorization demonstrates the potential of the disclosed technique to recover biomolecular information. The structure of a single myelinated neuron (white matter) from the brainstem, illustrated in pink, protrudes into the brain. Simultaneously, unmyelinated neurons (gray matter) appear in green on the right side of the frame. Finally, nuclear structures scattered throughout the brain tissue appear in white.

[0128] Referring to Figure 25, three distinct regions of brain tissue were selected based on visual inspection: gray matter (image (a)), gray matter (image (c)), and the transition or boundary between white and gray matter (image (b)). Each unique region was imaged with a photon absorption remote sensing PARS microscope before being colorized using the same K=3 model. In each of the selected regions, time-domain TD colorization highlights the same biomolecular structures identified in the initial colorized image (e.g., image (c) of Figure 24).

[0129] The time-domain TD signals may be clustered by shape, but not necessarily by amplitude. A given pixel (and its corresponding time-domain TD signal) may be represented in terms of one or more target characteristic signal shapes and residual terms. Specifically, for a given signal s and a learned characteristic signal shape (feature) {f i}, the signal is s=Σ i α i f i , so that the weights {α i} specifies the proportion of each characteristic signal shape. The residual term r is included to encapsulate any error as a result of modeling or measurement noise.

[0130] The time-domain TD signal can be a vector in a space Rn, where the dimension n of the space is simply the number of discrete time-domain TD samples. Since the time-domain TD signal is treated as an orthogonal vector, then the signal shape resembles a vector angle. A unit vector pointing in the direction of the non-noise part of a given cluster can define the centroid. A union can be constructed with the cluster and its negation point, and the centroid can be found as the direction of maximum variance (principal component from sample covariance), allowing higher amplitude signals to have the greatest influence.

[0131] The clustering algorithm is reflected in Figure 26. The corresponding method 2700 is reflected in Figure 27. The calculation of the cluster centroids is reflected in line 16, and may use singular value decomposition (SVD) to extract the first principal components. For input, the clustering algorithm takes a set S = {S i (t)} and the desired number of clusters K (equal to the number of learned features). Furthermore, the convergence criterion is specified by the difference between the minimum number of moves criterion and the mean residual criterion, which are necessary to ensure convergence.

[0132] The algorithm may be run several times, and only the optimal solution (in terms of the smallest mean residual) may be returned. The algorithm initializes by randomly selecting K time-domain TD signals to serve as initial cluster centroids, as shown in lines 1-3 and step 2702. Next is the "membership update" step, as shown in lines 7-12 and steps 2704 and 2706, which evaluates the distance from each point to each centroid in step 2704. The cluster membership of all points (Photon Absorption Remote Sensing (PARS)-Time Domain TD signals) is updated by assigning membership to the associated cluster of the furthest centroid in step 2706. The number of points that move (change cluster membership) is recorded (lines 9-11). Next, in step 2708, the mean residual is evaluated, starting from zero for the first iteration, as well as the change in mean residual (line 13) from the previous iteration (line 14). Next is the "Centroid Update" step, shown in lines 16-21 and step 2710, where the centroids are updated and calculated as the first principal component of the union of each cluster and its negative. Essentially, this is calculated via singular value decomposition (SVD), shown in line 19. In step 2712, the centroids are normalized so that they are unit magnitude. Finally, in step 2714, a convergence criterion is checked. If the algorithm has not converged ("No" in FIG. 27), the "Membership Update" step, followed by the "Centroid Update" step, are repeated until the convergence criterion is met ("Yes" in FIG. 27). The algorithm returns as output, in step 2716, a set of cluster labels indicating which cluster each Photon Absorption Remote Sensing PARS-Time Domain TD signal is associated with, and a set of K cluster centroids, which are the learned time-domain features.

[0133] Photon absorption remote sensing PARS-time domain TD signals may contain sufficient information to identify biomolecules based on their clustered time-domain TD features. Such features may be transferable across images of different tissue specimens. Feature identification can be performed on a first specimen and then transferred to other specimens, producing similarly compelling results. Furthermore, this technique offers a unique advantage because the clustering approach requires no prior information, except for the number of clusters. Training can be performed blindly across signals captured within the specimen of interest. This is particularly beneficial for complex specimens, such as the excised brain tissue explored here. The challenge is that blind clustering on a preselected number of features does not guarantee that a single biomolecule / tissue type is isolated per feature. Each cluster simply targets a unique characteristic of the photon absorption remote sensing PARS-time domain TD signal that can be used to highlight different tissue components.

[0134] Biomolecules can be visualized based on their photon absorption remote sensing (PARS)-time domain TD characteristics. This method allows a single (only one or exactly one) widely absorbed excitation source to simultaneously target the optical absorption of several biomolecules, providing otherwise inaccessible material specificity. This allows enhanced visualization of the acquired absorption contrast in a fraction of the time compared to similar multi-wavelength approaches. By adding an additional dimension to the absorption contrast, this enables several new avenues for label-free photon absorption remote sensing (PARS) microscopy, greatly expanding the possibilities for biomolecule specificity.

[0135] Referring to FIG. 28, additional methods of signal extraction have been devised to provide superior PARS non-radiative signal extraction. As previously described with reference to FIG. 17, averaging of regions both immediately before and after modulation can be used as a method of noise reduction. However, additional extensions of this concept can provide improved performance in more challenging scenarios. In particular, when the interrogation point is rapidly moving across the surface of the sample, it may be subject to additional non-PARS-based modulation due to spatial variations about the sample. In these instances, additional steps may be required to estimate the non-modulated scattering. While the method described with reference to FIG. 17 may be referred to as a "step" process, a similar "angled step" process may be envisioned. Here, the non-modulated scattering may be approximated by using an average of both the pre- and post-modulation regions, which allows for the extraction of PARS amplitude and time-domain information. More sophisticated approaches, such as specific pre- and post-modulation partial curve fitting, may also be envisioned with the same end goal.

[0136] Referring to FIG. 29, additional information can also be provided by recording various analog filtered instances of a single (only one or exactly one) photon absorption remote sensing PARS signal. For example, a relatively unfiltered signal can be obtained along with a high-bandpass signal by splitting the original analog signal from the photodetector and recording it onto two separate channels. From these, intelligent methods, such as the aforementioned K-means approach, can be independently utilized on the various recorded filtered iterations. Because these each represent highly independent signal measurements, additional signal fidelity can be extracted from such processing, enabling improved sensitivity.

[0137] Referring to Figure 30, additional information can also be provided by exploiting the expected spatial correlation between adjacent points. For example, data volumes can be reconstructed in two conventional horizontal image axes, with a third axis containing the respective time domains. This can facilitate horizontal processing operations prior to time-domain signal extraction. Here, the interdependent and mutually independent dependencies along the horizontal and time axes can be exploited to approximate a significantly lower-noise central signal. Similar non-intelligent approaches can be implemented with any or all of the photon absorption remote sensing (PARS) radiative, non-radiative, and scattering channels.

[0138] (Photon Absorption Remote Sensing PARS Time Domain Signature) Intelligent clustering methods have been used to analyze photon absorption remote sensing (PARS) time-domain (TD) signals when multiple absorption events occur closely together or simultaneously in time. This can result in overlapping photon absorption remote sensing (PARS)-time-domain TD signals. In this case, intelligent clustering approaches can be used to extract and isolate distinct absorption events, effectively unmixing distinct photon absorption remote sensing (PARS) events and thus isolating the time-resolved signals from each other, even if they overlap in time.

[0139] Conversely, it is also possible to extract maximally distinct signal combinations from combined photon absorption remote sensing PARS time-domain signals using intelligent clustering methods. In an example graph 8400 shown in Figure 84, two different wavelengths of photon absorption remote sensing PARS excitation (e.g., 266 nm and 532 nm) are introduced into a sample in close proximity. The result is a blend of photon absorption remote sensing PARS non-radiative time-domain TD signals. An intelligent clustering method, in this example, K-means, is applied to optimally extract information from the blended signals.

[0140] Figure 84 shows an example of photon absorption remote sensing PARS non-radiative time-domain features extracted from overlapping 532 nm and 266 nm excitation events. The three features represent maximally distinct absorption combinations that optimally define the photon absorption remote sensing PARS signal based on differences in absorption contrast at the two wavelengths.

[0141] Rather than attempting to isolate one photon absorption remote sensing PARS event from another, an exemplary algorithm is applied to determine the maximally different absorption combination. In the example shown in Figure 84, this results in three different signal combinations corresponding to: Feature 1 (8410): higher amplitude of the first absorption event, lower amplitude of the second absorption event; Feature 2 (8420): equal first and second absorption amplitudes; and Feature 3 (8430): low first absorption amplitude, high second absorption amplitude.

[0142] Converting the photon absorption remote sensing PARS signal to view them in terms of these clusters can provide enhanced resolution of the underlying biomolecules because the signal is expressed based on the magnitude difference between the two absorption events rather than their direct absorption magnitude at each excitation event.

[0143] An intelligent clustering method is described herein that uses a modified approach to apply K-means clustering to calculate cluster centroids. The intelligent clustering method aims to identify K characteristic shapes in a signal, described as a set of K centroids, F={fi(t)},i=1,...K. A time-domain TD signal can be represented as an orthogonal vector in space,

[0144]

number

[0145] where n corresponds to the number of time-domain TD samples. Thus, the shape of the signal is related to the angle of the corresponding vectors, and the distance between time-domain TD signals is quantified by the sine of the angle between them, resulting in maximum distance for orthogonal signals and zero distance for scaled or inverted signals. Cluster centroids are calculated as the principal components of each cluster and the combined set of its negatives, ensuring that the learned centroids are resilient to noise. Following K-means clustering, a set of feature vectors can be used to represent the signal as a weighted sum

[0146]

number

[0147] These feature vectors are then converted into a matrix of features:

[0148]

number

[0149] The amplitudes of the learned time-domain TD features (centroids) contained within each time domain are extracted by transforming from the time domain to the feature domain. This is performed by multiplying each time-domain TD signal by the pseudo-inverse of F[2]. The result is an array of K feature images, Mf = [mf1, mf2, ..., mfK].

[0150] The extracted features can then be used for colorization, direct visualization, or further pixel-level analysis, as discussed in the next section on photon absorption remote sensing PARS data vectors. In some embodiments, photon absorption remote sensing PARS-time domain TD features can be used to reduce the amount of data for colorization. Using only the most informative features and eliminating redundant data improves the predictive ability of the model by increasing the contrast between selected features and reducing training volume and time.

[0151] An example of a multi-channel virtual staining architecture 8500 for signal processing and virtual staining of photon absorption remote sensing (PARS) image data is shown in Figure 85. As shown, a feature learning process 8510 of K features is performed using a representative subset of the NR time-domain TD signals. A feature extraction process 8520 is then performed on all time-domain TD signals to generate K feature images. These K distinct feature images, along with the NR signal amplitude (normal PARS extraction method) from each excitation wavelength (266 nm and 532 nm) and the R signal amplitude, were then fed into a feature selection module 8530 to generate input data for a multi-channel (MC)-GAN model 8540.

[0152] First, feature learning process 8510 for K features is performed using a subset of the NR channel time-domain TD signals (shown in red boxes). Second, feature extraction is performed on all time-domain TD signals of the available data, forming K feature images. NR images for each excitation wavelength (266 nm and 532 nm in this case) and the R image are extracted separately and passed along with the K feature images to the feature selection phase. The selected features are then used as input data for an exemplary virtual staining machine learning model 8540, which can be an MC-GAN model, and the true H&E image 8550 is used as model ground truth.

[0153] When used for colorization, using the extracted features may enhance the predictive ability of the model by eliminating redundant data and increasing the contrast between the selected features. An example of this is shown below in the set of five images 8600 in Figure 86, which shows that feature-based colorization implemented using the architecture shown in Figure 85 outperforms alternative methods.

[0154] Figure 86 shows a comparison of virtual staining results using different combinations of photon absorption remote sensing PARS feature images as input: (a) RGB images of raw photon absorption remote sensing PARS data highlighting different parts of a human skin tissue sample, R: NR532, G: R266, B: NR266 (displayed for visualization). (b)-(d) Each shows the worst, medium, and best results using the feature combination labeled II. (e) True H&E image of the same field of view.

[0155] (Photon Absorption Remote Sensing PARS feature vector) For each photon absorption remote sensing PARS event, a photon absorption remote sensing PARS feature vector can be formed. An exemplary photon absorption remote sensing PARS feature vector is a photon absorption remote sensing PARS data vector. The photon absorption remote sensing PARS data vector for a given pixel can be thought of as a Euclidean vector in an "n"-dimensional space, where "n" is the number of photon absorption remote sensing PARS features in the given vector. This feature vector or data vector may comprise primary measurements, such as radiative and non-radiative signal amplitudes and energies, radiative and non-radiative signal lifetimes or signal characteristics, or may comprise secondary measurements extracted from the primary signals as different combinations, calculations, or ratios.

[0156] Examples of secondary measurements may include quantum efficiency ratio (quantum efficiency ratio QER), or total absorption (TA), or the ratio of radiative or nonradiative absorption at different wavelengths. An example photon absorption remote sensing PARS feature vector 8700 is presented in FIG. 87, which shows an example photon absorption remote sensing PARS data vector 8700, under "Error! Reference Source Not Found." Data included in this example may include radiative and nonradiative signal energy absorption (NR+R) at 266 nm and 532 nm, and quantum efficiency ratio (quantum efficiency ratio QER=(NR-R) / TA) at 266 nm and 532 nm. The exemplary photon absorption remote sensing PARS feature vector 8700 is not presented as an exhaustive list, but rather is only a representation of some of the data that may be extracted for each image pixel. The photon absorption remote sensing PARS feature vector may contain any information collected and extracted from each photon absorption remote sensing PARS event.

[0157] The photon absorption remote sensing PARS feature vectors can then be further processed for pixel-level analysis or passed directly to a colorization / visualization algorithm, e.g., an image generation model.

[0158] In one example of pixel-level analysis, photon absorption remote sensing PARS feature vectors can be directly correlated to ground truth tagging, such as histochemical or immunohistochemical staining. This can provide a one-to-one mapping between photon absorption remote sensing PARS data vectors and different histochemical stains or their underlying biomolecular targets. This process allows a photon absorption remote sensing PARS "signature / fingerprint" or ground truth photon absorption remote sensing PARS data vector to be calculated for a given biomolecule or mixture of biomolecules. For example, this can be used to generate a fingerprint for cells expressing the HER2 protein. This can then be used as ground truth for testing whether cells express the HER2 protein. The same process of generating a "ground truth" photon absorption remote sensing PARS data vector can be performed on any biomolecule or mixture of biomolecules.

[0159] Alternatively, instead of using ground truth metrics, intelligent blinding methods such as clustering (e.g., k-means or principal component analysis) can be applied directly to the photon absorption remote sensing PARS data vectors to identify unique groups of constituent features. This can provide different representations of the data that better separate the underlying biomolecules. These methods can also be used to determine which components of the photon absorption remote sensing PARS data vector provide optimal discrimination of specific tissue features of interest. This approach can then be used to reduce the amount of photon absorption remote sensing PARS data while retaining as much detail as possible of the underlying composition.

[0160] One advantage of the photon absorption remote sensing PARS data vector representation is that the signal can be processed in any vector space (i.e., polar, Euclidean, etc.). This allows for many different vector processing methods to be utilized. For example, the relative presence of a biomolecule at a given pixel can be calculated by projecting that pixel's photon absorption remote sensing PARS vector onto the ground truth vector of the target biomolecule. In Euclidean space, this operation is performed by taking the cross product of the ground truth photon absorption remote sensing PARS data vector and the pixel's photon absorption remote sensing PARS data vector. This method may be ideal for use in hardware-accelerated processing such as CUDA or graphics card-based processing.

[0161] (Functional Extraction (from Radioactive, Non-Radioactive, and Scattered)) As explained above with respect to the quantum efficiency ratio (QER), properties such as thermal diffusivity, conductivity, and sound speed can determine the photon absorption remote sensing (PARS) relaxation time. Features such as temperature, sound speed, and molecular information can be extracted from the time-domain signal. As an example, two targets may have the same or similar optical absorption but slightly different other properties, such as different sound speeds, which can result in different decay, evolution, and / or shape of the signal. The decay, evolution, and / or shape of the signal can be used to determine or add new molecular information to the photon absorption remote sensing (PARS) image.

[0162] Various optical and mechanical properties can cause these differences in signal shape. For example, the rate at which a signal returns to background scattering levels can be determined by the local thermal diffusivity. As a result, for example, regions with higher thermal diffusivity can be characterized by a shorter signal length, as opposed to regions with lower thermal diffusivity. This can be used to distinguish between cell nuclei and surrounding regions with similar optical absorption. Similarly, signal lifetime can also be affected by the local speed of sound. One example would be for use in distinguishing between two different metals. Aluminum and copper are characterized by different thermal diffusivities and speeds of sound, facilitating multiplexing by measuring signal lifetimes separately. Figure 31 illustrates two signals with different lifetimes.

[0163] (Post-imaging correction) Referring to FIG. 32, by acquiring two (or more) unique absorption-based measurements (radioactive and non-radioactive), local variations in these acquisitions can be used to compensate for variations in excitation pulse energy. For example, two acquisitions can be compared for similar local (pixel-level) variations at or near resolution, or sub-resolution, in intervals. Because the system is not expected to provide such a level of spatial discrimination, rapid local variations are unlikely to be caused as a result of spatial variations in the sample. In this way, similar variations can be interpreted as similar reconstruction errors between the two visualizations. This interpretation can then be used to provide post-imaging intensity corrections that provide additional qualitative recovery. While FIG. 32 shows an example of autofluorescence-based compensation, the embodiments disclosed herein are not limited to autofluorescence and may use other absorption-based measurements.

[0164] (Chirped Pulse Photon Absorption Remote Sensing (PARS) acquisition) Referring to Figure 33, given that photon absorption remote sensing (PARS) acquisitions are typically performed by using a single photodetector element to capture the time-varying sample response, the practical bandwidth and noise limitations of such devices can provide a significant barrier to high speeds. One potential solution to this could be streak detection of the photon absorption remote sensing (PARS) signal. Streak detection involves spatially separating various time components across several detectors, such as those in line-scan or standard cameras, which can be achieved in several ways.

[0165] For example, chirped pulses (pulses with wavelengths that vary along their length) can be used for detection, and the various wavelength components that can encode temporal information can be spatially separated using one or more diffractive or dispersive elements, such as prisms or gratings. This process can provide significant improvements in temporal resolution while maintaining high signal fidelity by spanning detection across a significant number of detectors. Such architectures have a clear role to play, such as in combination with line-scan architectures where detection is performed on a large array, such as a camera, whose two spatial coordinates encode one spatial dimension and one temporal dimension from the sample. Other methods of streaking the temporal axis across a sensor array, such as using a high-speed optical scanner, can also be envisioned.

[0166] (Time Domain Photon Absorption Remote Sensing (PARS) acquisition from the integration of a photodetector unit) Many imaging sensors have minimum integration times that may not be able to capture nanosecond-scale modulations in the photon absorption remote sensing (PARS) signal evolution. This can be limiting due to the potentially rich time-domain information provided in photon absorption remote sensing (PARS) signals. A general process utilizing rolling shutter / trigger sequence / delay binning that captures modulations within the integration time of these photodetectors is described herein.

[0167] In this photon absorption remote sensing PARS acquisition regime, backscattered detection light with photon absorption remote sensing PARS modulation can be distributed across an array of integrated photodetection units. At the beginning of acquisition, an adjustable delay can be introduced between the integration start times of each photodetection unit (e.g., by using a rolling shutter, a predetermined trigger sequence, delayed binning, and / or capturing differently timed sections of the recovered signal). If the delay time is shorter than the integration time of the photodetection unit, the signal can be reconstructed with a time resolution defined by the imposed delay. For example, photon absorption remote sensing PARS time-domain information can be extracted by taking the derivative of the integration window over these time intervals and / or by analyzing their intersection when plotted. A visual depiction of this acquisition method is shown in Figure 34. For example, instead of high-sample-rate photodetectors, CCD / CMOS camera sensors can be utilized to resolve the time-domain signal. In this case, the CCD / CMOS camera array is the photodetection unit capturing the signal in a rolling shutter fashion. The imposed delay between individual photodetector lines allows a photon absorption remote sensing PARS time domain signal to be constructed with greater time resolution than a single integrated sensor.

[0168] (Data compression) Referring to Figure 35, data can be compressed using digital and / or analog techniques. For example, in a K-means approach, raw time-domain signals can be adequately represented by their respective K-means weights. For example, if three such prototypes were used on a particular data set, rather than storing the full time-domain (approximately 200 or more samples), the time axis could be adequately compressed to just three values ​​or floats. Similar such extracted features could be used in place of the full uncompressed time-domain for purposes such as reducing system RAM usage, reducing data bandwidth requirements, reducing system storage load, etc.

[0169] (fast acquisition) Acquisition at higher interrogation rates may require more complex acquisition processes. Various problems may arise while interrogating samples at higher acquisition rates, which involve logical movement of the interrogation spot relative to the sample and higher frequency light scattering signals. Rapid lateral movement of the interrogation spot relative to the sample may be achieved through hybrid scanning approaches that combine both high-speed optical scanning methods, such as resonant scanners and polygon scanners, alongside bulk scanning approaches, such as mechanical scanning stages. Such methods in other optical microscopy approaches have facilitated interrogation rates in the 10 s MHz range and may offer similar advantages over the photon absorption remote sensing (PARS) modality.

[0170] However, such fast motion of the interrogation spot relative to the sample can also induce additional undesired scattering frequency content that can confound time-domain signal processing of the collected photon absorption remote sensing PARS signal. Thus, as shown in Figure 36, it can be beneficial to operate the detection focal spot on the sample at a larger size relative to the excitation spot so that the excitation spot can be scanned relative to a relatively stationary or slower-moving detection spot, which reduces the impact of fast optical scanning on the detection.

[0171] (Data colorization) 37, the techniques and methods disclosed herein may bypass grayscale or scalar amplitude-based reconstruction and allow for the direct construction of colorized H&E simulated images. The colors used may emulate those traditionally used in H&E staining, such as various shades of pink, purple, and / or blue. However, the embodiments disclosed herein are not limited to pink, purple, and / or blue colors, and systems and processors may be configured to use other colors. For example, red, green, and blue color channels may be used to represent three extracted K-means prototypes.

[0172] (Augmented Reality Interface) Upon completing the data visualization or image processing, these visualizations may be displayed in combination and / or overlaid with other visualizations on the user interface screen. For example, a bright-field image of a sample may form the background of a presented photon absorption remote sensing (PARS) visualization. Such an extension may be used to help maintain orientation between the desired visualization and the original sample.

[0173] (Machine learning processing of photon absorption remote sensing PARS signals) 38A and 38B show two exemplary architectures 3800, 3850 for generating one or more inferences about a sample. Architectures 3800, 3850 may include a photon absorption remote sensing PARS system 3801, which may include one or more of a total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multi-photon excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time-domain photon absorption remote sensing (TD-PARS) subsystem. Photon absorption remote sensing PARS system 3801 may be, for example, the photon absorption remote sensing PARS system from FIG. 5 above.

[0174] The photon absorption remote sensing PARS system 3801 detects generated signals in the detection beam(s) returning from a given sample. These perturbations may comprise, but are not limited to, changes in intensity, polarization, frequency, phase, absorption, nonlinear scattering, and nonlinear absorption, and may be caused by a variety of factors, such as pressure, thermal effects, etc.

[0175] The sample, being an unstained sample, can be an in vivo or in situ sample. For example, it can be tissue under the skin of a patient. In another example, it can be tissue on glass.

[0176] In some embodiments, the photon absorption remote sensing PARS system 3801, 3901 can operate by capturing nanosecond-scale (or picosecond-scale) optical perturbations generated by photoacoustic pressure or photothermal temperature signals. These time-domain (TD) modulations are typically projected by amplitude to determine the magnitude of absorption. A single characteristic intensity value can be extracted from each time-domain TD signal to visualize the magnitude of total absorption at each point. For example, the time-domain TD amplitude, calculated as the difference between the maximum and minimum values ​​of the time-domain TD signal, is commonly used to represent the absorption amplitude.

[0177] In some embodiments, the photon absorption remote sensing PARS system 3801, 3901 may operate by incorporating optical perturbations generated by thermal pressure perturbations in addition to, or as an alternative to, the optical perturbations generated by photoacoustic pressure and photothermal temperature signals.

[0178] The signals detected by the photon absorption remote sensing PARS systems 3801, 3901 may comprise, for example, absorption spectrum signals, radiative signals, non-radiative signals, scattering signals, or any combination of the above signals.

[0179] Absorption spectroscopy refers to a spectroscopic technique that measures the absorption of radiation as a function of frequency or wavelength due to its interaction with a given sample. The sample absorbs energy, i.e., photons, from a radiation field. The intensity of the absorption varies as a function of frequency, and this variation is the absorption spectrum. The absorption spectrum signal can provide some contrast across a broad spectrum of wavelengths to characterize the response of biomolecules across an excitation range (e.g., 190 nm to 20 μm).

[0180] As a non-exhaustive list of examples, the following is a list of various types of signals that may be processed by the architectures 3800, 3850, 3900: a.Radioactive: ● Signal amplitude and energy, ● Emission spectrum, ● Lifespan / decay rate, This is affected by conductivity, viscosity, temperature and polarity.

[0181] b. Non-radioactive: ● Signal amplitude and energy, ● Non-radiative signal lifetime (or decay rate); · This is influenced by material properties such as speed of sound, density, compressibility, shear modulus, pressure, stiffness, bulk modulus, viscoelasticity, thermal diffusivity, heat capacity, conductivity, viscosity, size and shape of the absorber, and temperature.

[0182] ● Rise time: This is affected by conductivity, viscosity, temperature and polarity. ● Phase shift, ● Polarization shift.

[0183] c. Scatter: ● Amplitude, ● Polarization, and d. Combinations of any of the above, e.g., ● Total energy absorption = non-radiative + radiative signal, ● Quantum efficiency ratio = (non-radiative - radiative) / total absorption, ● Life relaxation ratio, ● Total relaxation time (radioactive and non-radioactive), etc.

[0184] Various signals from the photon absorption remote sensing PARS systems 3801, 3901 may be processed to extract one or more photon absorption remote sensing PARS features 3804, 3904. For example, the one or more photon absorption remote sensing PARS features 3804, 3904 may represent one or more contrasts. The one or more photon absorption remote sensing PARS features 3804, 3904 may comprise one or more photon absorption remote sensing PARS diagnostic vectors.

[0185] In some embodiments, the one or more photon absorption remote sensing PARS features 3804, 3904 may represent one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast.

[0186] In some embodiments, a feature vector for a machine learning architecture (e.g., architectures 3800, 3850, 3900) for making one or more decisions to aid in diagnosis comprises one or more of photon absorption remote sensing PARS features 3804, 3904, absorption magnitude and intensity values, time domain TD post-excitation averages, emission channels, scatter channels, features extracted from time domain (TD) modulation such as H stain, E stain, Jones stain (MPAS), PAS and GMS stain, toluidine blue, Congo red, Masson's trichrome stain, Lilly trichrome, and Verhoeff stain.

[0187] In some embodiments, the extracted photon absorption remote sensing PARS features 3804, 3904 are used to segment nuclei and can be used for quantification that may be required to make a diagnosis. The quantification can be cancer quantification. The quantification can include, for example, quantification of nucleoli, nuclei, shape, size, and circularity.

[0188] Figure 44 shows an example of contrast 4400 extracted from photon absorption remote sensing PARS signals in a tissue slide. Examples include, for example, non-radioactive, radioactive, and scattering contrast. Figure 45 shows an example of contrast 4500 combination from the combination of photon absorption remote sensing PARS signals into unique contrasts.

[0189] Processing the signal may comprise exciting the sample at an excitation location with an excitation beam by a photon absorption remote sensing PARS system 3801, 3901, where the excitation beam is focused at or below the sample, and interrogating the sample with an interrogation beam directed towards the excitation location of the sample, where the interrogation beam is focused at or below the sample.

[0190] In some embodiments, focusing the excitation beam at or below the sample may comprise being at or below the surface of the sample. In some embodiments, a system or systems other than a photon absorption remote sensing PARS system 3801 (e.g., photothermal, autofluorescence, etc.) may be used to generate a range of absorption spectral signals, radiative signals, non-radiative signals, attenuation signals, scattering signals, or any combination of the above signals, used to generate one or more features 3804. These systems may comprise conventional imaging systems or imaging modalities.

[0191] In some embodiments, the extracted photon absorption remote sensing PARS features 3804, 3904 may comprise features that provide information about the attenuation contrast provided by at least one of the multiple signals. For example, the attenuation may be a decrease in intensity of the excitation beam produced by the photon absorption remote sensing PARS system 3801, 3901 as it traverses a material (e.g., tissue). For example, contrast between tissues may be generated by differences in beam signal attenuation, which may be affected by the density and atomic number of each tissue.

[0192] 38A may be trained and deployed to generate simulated stained images 3806, such as H&E-like stained images. The machine learning model 3802 may also be trained and deployed to generate one or more inferences 3808 that are enabled for display on a user interface 4000 of a user application 3825 that may be installed on a user device. A database 3815 stores the one or more simulated stained images 3806 and may be used to transmit the one or more simulated stained images 3806 to the user application 3825 for display or further processing.

[0193] The simulated stained image 3806 may comprise an image stained with at least one of, for example, hematoxylin and eosin (H&E) stain, Jones stain (MPAS), PAS and GMS stain, toluidine blue, Congo red, Masson's trichrome stain, Lilly trichrome, and Verhoeff stain.

[0194] In some embodiments, the simulated staining comprises at least one of hematoxylin and eosin (H&E) staining, Jones staining (MPAS), PAS and GMS staining, toluidine blue, Congo red, Masson's trichrome staining, Lilly's trichrome, and Verhoeff's staining, immunohistochemistry (IHC), histochemical staining, and in situ hybridization (ISH).

[0195] In some embodiments, the simulated staining can be applied to frozen tissue sections, preserved tissue samples, or fresh, unprocessed tissue. For example, preserved tissue samples can include samples preserved with formalin or alcohol fixed with an alcohol fixative.

[0196] 38B , an image generator 3812 may be used to generate a simulated stained image 3806, such as an H&E-like stained image. A machine learning model 3822 may be trained and deployed to generate one or more inferences 3808 that are enabled for display on a user interface 4000 of a user application 3825, which may reside on a user device. A database 3815 stores the one or more simulated stained images 3806 and may be used to send the one or more simulated stained images 3806 to the user application 3825 for display or further processing.

[0197] Referring now to FIG. 39 , FIG. 39 shows yet another exemplary machine learning architecture 3900 for generating one or more inferences 3908 based on features 3904 extracted from a sample. The extracted features may be photon absorption remote sensing (PARS) features 3904, which may be generated in a similar manner as the photon absorption remote sensing (PARS) features 3804 from FIGS. 38A and 38B . For example, the features 3904 may be extracted by exciting the sample at an excitation location with an excitation beam, where the excitation beam is focused at or below the sample, and interrogating the sample with an interrogation beam directed toward the excitation location of the sample, where the interrogation beam is focused at or below the sample. The excitation beam and the interrogation beam may be generated by a photon absorption remote sensing (PARS) system 3901, which is a system similar to the photon absorption remote sensing (PARS) system 3801.

[0198] The extracted features 3904 may be processed by an image generator 3912, which may be similar to image generator 3812 in Figure 38B, to generate (or convert features into) one or more simulated stained images 106 (e.g., H&E-like stained images). A machine learning model or architecture 3922, which may be similar to machine learning model 3822 in Figure 38B, may be used to generate one or more inferences 3908 based on the one or more simulated stained images 106. The inferences may be sent to a user application 3925 for display or further processing.

[0199] The machine learning model 3802 and image generator 3812, 3912 are configured to generate one or more simulated stain images 3806, 106 based on one or more extracted photon absorption remote sensing PARS features 3804, 3904 extracted based on one or more photon absorption remote sensing PARS signals. For example, the one or more photon absorption remote sensing PARS signals may comprise radioactive and non-radioactive signals. The non-radioactive signals may be processed to generate features representing amplitude or absorption contrast similar to that provided by hematoxylin staining, while the radioactive signals may be processed to generate features representing amplitude or absorption contrast similar to that provided by eosin staining. Thus, the machine learning model 3802 and image generator 3812, 3912 are trained and configured to generate H&E-like images as one type of simulated stain image 3806, 106 based on the radioactive and non-radioactive signals.

[0200] The radiative and non-radiative signals may be acquired from a photon absorption remote sensing PARS system 3801, 3901. In some embodiments, the radiative and non-radiative signals may be acquired from different systems or imaging modalities. For example, the non-radiative signals may be acquired via photothermal microscopy and photoacoustic microscopy. For example, the radiative signals may be acquired via multi-wavelength / single-wavelength autofluorescence microscopy, stimulated / spontaneous Raman spectroscopy, or autofluorescence lifetime microscopy.

[0201] In some embodiments, the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal. In some embodiments, the radioactive signal comprises one or more autofluorescence signals.

[0202] The image generators 3812, 3912 may include a stain selector 3914 that selects one or more stains to apply to the generated image (e.g., a photon absorption remote sensing PARS black and white image) based on the photon absorption remote sensing PARS features 3904.

[0203] The image generators 3812, 3912 may comprise a colorized machine learning architecture such as a generative adversarial network (GAN), for example, a cycle-consistent generative adversarial network (CycleGAN).

[0204] In some embodiments, image generator 3812, 3912, or an image generator in machine learning model 3802 may be implemented using one of a CycleGAN, a Pix2Pix model (a type of conditional GAN), a stable diffusion model, a U-Net model, an encoder-decoder model, a convolutional neural network, a regional convolutional network, etc.

[0205] Image segmentation is the process of extracting regions of interest (ROIs) through semi-automatic or automatic processes, which divide an image into areas based on specific descriptions, such as segmenting body organs / tissues in medical applications for boundary detection, tumor detection / segmentation, and mass detection.

[0206] Image registration is the process of aligning two images from two domains (total absorption-photon absorption remote sensing (TA-PARS) and H&E) through a semi-automated or automated process. The total absorption-photon absorption remote sensing (TA-PARS) image can be the input image, and the H&E image can be the reference image. The system can be configured to select points of interest in the two images (e.g., the input image and the reference image), associate each point of interest in the input image with its corresponding point in the reference image, and transform at least one of the input image and the reference image so that both images are aligned, for example, via image generators 3812, 3912. In some cases, both images are transformed and aligned.

[0207] The image generators 3812, 3912 may include one or more machine learning techniques for image segmentation, including, for example, the following: (1) traditional methods: threshold segmentation, region growing segmentation; (2) classification and clustering methods: K-nearest neighbors (KNN), kernel principal component analysis (kPCA), fuzzy C-means (FCM), Monte Carlo random field models (MRF), dataset-based guided segmentation, expectation maximization (EM), Bayesian methods, support vector machines (SVM), artificial neural networks (ANN), random forest methods, and convolutional neural networks (DNN); and (3) deformation model methods: parametric deformation models, geometric deformation models.

[0208] The selectable stains may comprise, for example, at least one of the following (or any combination thereof): Jones stain (MPAS) - typically for kidneys, -AS and GMS - for fungal infections (stained chitin), Toluidine blue, Congo Red - identification of amyloid material, Masson's trichrome staining, Lily trichrome, Verhoeff staining - visualizes elastic tissues (blood vessels, skin, bladder, etc.).

[0209] In the image generators 3812, 3912, one or more stain images 3806, 106 can be generated from the same sample. Additionally, it is possible to generate additional stains by generating and combining separate stains. The image generators 3812, 3912 can be configured to virtually generate constituent stains.

[0210] For example, Masson's trichrome staining is a three-color staining procedure that comprises: (1) Hematoxylin - for nuclear staining, (2) acid dyes (e.g., red carvert acid + acid fuchsin) - for the cytoplasm, and (3) Toluidine blue - for collagen.

[0211] In some embodiments, the image generator 3812, 3912, or an image generator within the machine learning model 3802, can be configured to apply each constituent stain of any particular stain by applying each stain to an image (e.g., a black and white photon absorption remote sensing PARS image) that has already been generated based on the photon absorption remote sensing PARS features 3804, 3904.

[0212] In some embodiments, image generator 3812, 3912, or an image generator within machine learning model 3802, during inference, can be trained to generate an image showing a tissue map overlay by processing photon absorption remote sensing (PARS) features. The overlay can identify at least one salient feature, where the at least one salient feature comprises a biomarker, cancer, cancer grade, parasite, toxicity, inflammation, and / or cancer. The overlay can suppress non-salient features.

[0213] Additionally, through the user application 3825, 3925 the user can switch between different stained images 3806, 106. This is not always possible and is rarely practical after chemically labeling a sample in the conventional manner.

[0214] The machine learning architectures 3850, 3800, 3900 can provide the same contrast as the constituent stains. The machine learning architectures 3850, 3800, 3900 mimic these individual stains and are enabled to digitally mix / match them together to create different stain combinations. Because stains are combined digitally (with unique contrasts) rather than chemically, new stain combinations can be made possible based on a given sample.

[0215] In some embodiments, the machine learning architectures 3850, 3800, 3900 may generate stains or stain combinations that may comprise stains or stain combinations not previously generated or that cannot be achieved through traditional chemical staining methods. For example, the machine learning architectures 3850, 3800, 3900 may generate molecular stains that may be impossible with traditional staining methods.

[0216] Inferences 3808, 3908 generated by architectures 3800, 3850, 3900 may include, but are not limited to, prediction of one or more of biomarkers, survival time, drug response, patient-level phenotypic / molecular characteristics, mutational burden, tumor molecular characteristics, transcript characteristics, protein expression signatures, patient clinical outcome, resistance index associated with the tumor and surrounding tissue based on one or more photon absorption remote sensing PARS signals, determining the best tissue sample in a collection of samples for testing and verifying that the selected tissue sample has a sufficient amount of tumor tissue for analysis, determining which signals among the multiple photon absorption remote sensing PARS signals are suspicious or non-suspicious and generating a report based on identifying the suspicious signals, locating biomarkers within the tumor tissue and surrounding border region, predicting treatment outcome or resistance prediction or treatment recommendation, cancer eligibility and cancer quantification of the specimen.

[0217] The inferences 3808, 3908 generated by the architectures 3800, 3850, 3900 may further comprise at least one of, but not limited to, survival time, drug response, drug resistance, phenotypic characteristics, molecular characteristics, mutational burden, tumor molecular characteristics, parasites, toxicity, inflammation, transcriptomic features, protein expression features, patient clinical outcome, suspicious signals, biomarker location or value, cancer grade, cancer subtype, tumor border area, and grouping of cancer cells based on cell size and shape.

[0218] Additional exemplary embodiments of various machine learning architectures that may be implemented to process one or more outputs (e.g., photon absorption remote sensing PARS features and signals) from a photon absorption remote sensing PARS system are described in further detail below in connection with Figures 64-79.

[0219] 48 shows examples of different tissue types imaged and identified using machine learning architectures 3850, 3800, 3900, comprising skin tissue 4800 and breast tissue 4850. The inferences 3808, 3908 may comprise, for example, a determination that the image on the left comprises skin tissue and the image on the right comprises breast tissue.

[0220] 49 shows unique keratin pearl features identified and isolated within an example simulated stain image 4900. Inferences 3808, 3908 may comprise, for example, identification of areas indicative of keratin pearls.

[0221] 50 shows localized inflammation and malignancy biomarkers identified and surrounded based on an exemplary simulated stain image 5000 with unlabeled visualization. Inferences 3808, 3908 may include, for example, identifying areas likely to belong to cancer and areas likely to belong to lymphocytes. Extracted photon absorption remote sensing PARS features 3804, 3904 may be used by the system to label unique biomarkers, such as red blood cells, tissue types, melanin, collagen, different proteins, etc.

[0222] For example, in some embodiments, the image generator 3812, 3912, or an image generator within the machine learning model 3802, during inference, is enabled to be trained to generate an image showing a tissue map overlay by processing extracted photon absorption remote sensing PARS features 3804, 3904. The overlay can identify at least one salient feature, which can be a biomarker location and a biomarker value of the identified tissue region on the image.

[0223] 51 shows different cell types and tissue regions identified and delineated within an exemplary simulated stained image 5100. Inferences 3808, 3908 may comprise, for example, identifying areas that are likely to belong to one of a hair follicle, a sebaceous gland, and an epidermal layer.

[0224] 52 shows an example of abnormal tissue regions identified and delineated from an exemplary simulated stain image 5200. Inferences 3808, 3908 may comprise, for example, identifying areas that are likely to belong to abnormal tissue.

[0225] In some embodiments, the user application 3825, 3925, when executed, may render a user interface (UI) 4000 as shown in Figure 40. The user interface UI 4000 may include a first area 2510 showing features 3804, 3904 from the photon absorption remote sensing PARS system 3801, 3901, a second area 2512 showing a first simulated stained image, and a third area 2516 showing a second simulated stained image 2516. One or more inferences 3808, 3908 may be displayed in area 2517.

[0226] One or more stain selectors 2520, 2540 may be provided to the user, each having a respective scroll bar 2528, 2538 for zooming in or out of the rendered simulated stained image shown in areas 2515, 2516. For example, moving the scroll button within scroll bar 2528 for first stain selector 2520 may zoom in or out of the first stained image in area 2515. Similarly, moving the scroll button within scroll bar 2538 for second stain selector 2540 may zoom in or out of the second stained image in area 2516. Once the user is satisfied with the stained image in 2515 or 2516, the user may proceed to finalize the simulated stained image by clicking a submit button. Alternatively, the user may cancel the rendered stain and return to a previous user interface (not shown) to select another applicable stain provided by stain selector 3914 of image generator 3912.

[0227] In some embodiments, the one or more inferences 3808, 3908 displayed in area 2517 may comprise clinically significant decisions generated by the machine learning models 3802, 3822, 3922. The user interface UI 4000 may further comprise visualizations to assist a user (e.g., a clinician), such as reports generated by the machine learning models 3802, 3822, 3922. The visualizations or reports may be interactive. The visualizations or reports may comprise visual overlays that highlight salient features while suppressing or hiding non-salient features. The visualizations may be provided in real time to assist a surgeon, for example, by showing the boundaries of tumor tissue.

[0228] In some embodiments, to generate one or more inferences 3808, 3908, the plurality of features 3804, 3904 may be supplemented with at least one informative feature of image data obtained from a complementary modality, for example, comprising at least one of ultrasound imaging, a positron emission tomography (PET) scan, a computed tomography (CT) scan, and magnetic resonance imaging (MRI).

[0229] In some embodiments, when the plurality of features 3804, 3904 are complemented by at least one of the informative features of image data obtained from a complementary modality, the image data may further comprise photoactive labels for contrasting or highlighting particular regions within the image.

[0230] In some embodiments, to generate one or more inferences 3808, 3908, the plurality of features 3804, 3904 may be supplemented with at least one informative feature of image data obtained from a complementary modality, for example, comprising at least one of ultrasound imaging, a positron emission tomography (PET) scan, a computed tomography (CT) scan, and magnetic resonance imaging (MRI).

[0231] In some embodiments, the plurality of features 3804, 3904 may be supplemented by at least one of the following informative features: ● Specific patient information: ○ Age, gender, environmental factors (job, location), ○ Genome expression.

[0232] ● Clinical history: Risk factors (previous conditions, medical history, family history, etc.), ○ Previous diagnostic reports, o Results of ancillary tests (e.g., blood tests, cytological screening), Previous photon absorption remote sensing (PARS) images, or H&E images, etc. o Images from complementary modalities (e.g., PET, CT, MRI), patient images.

[0233] • Automatic quality assessment of data sources (e.g., H&E stain quality, photon absorption remote sensing PARS scan quality). In some embodiments, a user application, which may be user application 3825, 3925 or a separate user application in the architecture of FIG. 38A, FIG. 38B, or FIG. 39, may be configured to render a user interface 5900 shown in FIG. 59 to select and analyze one or more images generated by the architecture of FIG. 38A, FIG. 38B, or FIG. 39. The user interface UI 5900 may comprise a first area 5920 illustrating a plurality of procedures 5930 and a second area 5950 illustrating corresponding procedure information for one of the plurality of procedures 5930. User input may be received by the user application to select one of the plurality of procedures 5930. Each procedure may be associated with a corresponding set of procedure information and one or more corresponding images.

[0234] The image display area in user interface UI 5900 may include several subcomponents or subsections used to navigate, visualize, or manipulate collected data. User interface UI 5900 may group procedures by date or relevance. For example, user interface UI 5900 may group procedures by date into one of the listed tabs: "In Progress," "Recent," or "Last 2+ Months."

[0235] Each tab of the user interface UI5900 may be configured to display one or more datasets visualized in a manner deemed applicable and / or appropriate to the user. For example, a user (e.g., a clinician or physician) may select to visualize collected photon absorption remote sensing PARS data in a manner that emulates traditional pathological staining procedures, such as hematoxylin and eosin or toluidine blue, to highlight particular structures. Such virtual stains are presented and may be combined and overlapped as they might appear through traditional staining and light microscopy techniques.

[0236] In some embodiments, the user application may similarly be configured to display each stain (e.g., hematoxylin or eosin from H&E) separately from one another to further elucidate prominent morphologies. Similarly, other virtual stains, acquired channels, or layers of the dataset may be displayed by themselves or in combination based on user preferences. Based on user settings or preferences, a single image layer may occupy the entire display area to provide additional context to the user and clearly show image details while maintaining a wide field of view. Similarly, two or more such images or image layers may be arranged in horizontal and / or vertical divisions with their own separate display areas. The order and orientation of the images may be set or modified by the user through graphical user interface elements.

[0237] 60 and 61, which illustrate exemplary user interfaces 6000, 6100 for displaying one or more images generated by the architectures of FIG. 38A, 38B, or 39, the displayed visualization or image may be represented as a combination of various data layers. For example, individual stains may be presented in an overlapping manner or as isolated individual layers. For example, FIG. 60 illustrates a user interface UI 6000 displaying a virtual H&E image 6020 of tissue and a single non-radioactive image 6050 of the same tissue.

[0238] The visibility and combination of different image layers can be switched, managed, and manipulated via GUI elements located, for example, at the top, left, or right of the screen area relative to the presented image frame. For example, graphical user interface elements such as drop-down menus 6010, 6030 can receive user input, and the user interface UI 6000 can be enabled to display a selected image, layer, or stain based on the user input received via the drop-down menus 6010, 6030.

[0239] In some embodiments, image manipulation operations such as scanning, panning, zooming in / out, location, contrast adjustment, color adjustment, opacity, etc. may be performed on the visualization or image. In the case of multiple adjacent windows showing different images or layers of the same tissue, such as virtual H&E image 6020 and non-radioactive image 6050, it may be desirable to lock them to the same field of view. A graphical user interface element, such as a "link images" checkbox located at the bottom of user interface UI 6000, may be clicked by the user to lock virtual H&E image 6020 and non-radioactive image 6050, so that panning or zooming one of the images so automatically locked (e.g., virtual H&E image 6020) causes the other locked image (e.g., non-radioactive image 6050) to have the same field of view and the same display ratio.

[0240] When locked (or linked), the conversion of locked images to regions can allow a user application to highlight the same region across multiple locked images. This provides a mechanism for quickly assessing the contributions of constituent chromophores and can be useful for highlighting regions of interest or assisting in raw data imaging artifacts in pathological analysis. For example, displaying an H&E stain of a tissue region in a first display area and a separate toluidine blue stain of the same tissue region in a second display area adjacent to the first display area can highlight the unique contrast of each stain within the same region to aid in diagnosis. Such image data visualization tools can facilitate easy and quick comparisons between collected datasets based on a given sample, adjacent samples, other samples from the same patient taken from different locations and / or different times, or comparisons with other patients or other imaging sessions.

[0241] The acquired image contrasts can be displayed or processed by the system like image layers. Layers can be individually edited, made visible (or invisible), layered (as overlays), combined, or combined to create additional contrasts that may provide greater interpretability; for example, FIG. 61 shows a user interface 6100 showing a first image layer 6150 (e.g., a virtual H&E image layer) and two additional image layers, “Scattered (405 nm)” and “Emissive (266 nm),” that are selectable by the user through GUI (drop-down menu) 6130. When the user clicks the “Overlay with Image” option and selects a particular additional image layer through GUI 6130 to overlay the first image layer 6150, the user interface UI 6100 can proceed to show the first image layer 6150 with the overlay of the selected additional image layer (which in this example may be “Scattered (405 nm)” or “Emissive (266 nm)”).

[0242] Combinations of layers can be grouped and modified as groups. Grayscale or group layers can be colorized to match specific individual stains or combinations of stains. For example, a photon absorption remote sensing PARS radioactive absorbing layer can be modified and colorized to emulate an eosin stain. A photon absorption remote sensing PARS non-radioactive layer can be colorized to emulate a hematoxylin stain. These layers can be viewed separately or combined as a group layer where the stains are overlaid to emulate a hematoxylin and eosin stain combination.

[0243] Larger collections of datasets, single patient acquisition sessions, multiple patient acquisition sessions, or other projects describing one or more datasets intended to be grouped together within a collection may be presented as projects within a project display area of ​​the user application's user interface. Such project display user interface UI may be positioned as a separate sub-region of the primary display area or presented on a separate tab or similar separation. The project user interface UI may facilitate grouping or collection of similar images that may have been collected on a given image during a session or from within a given imaging project. Such imaging projects may be more easily transferable, as opposed to large collections of individual datasets. The grouping and presentation of constituent datasets may be further organized for the user's convenience by aspects such as location, collection date, patient ID number, etc.

[0244] The collected data can be visualized in a variety of distinctive ways. For example, any of the collected data channels can be visualized by plotting their respective signal values ​​as grayscale intensity values ​​mapped to their respective locations on a two-dimensional image that may correspond to their respective locations on the sample. An example of such a collected data channel can include photon absorption remote sensing (PARS) non-radiative absorption contrast, whose signal can be extracted from the collected time domain. When imaging biological tissue, such contrast can highlight areas of high DNA density, such as cell nuclei. Another example of such collected data can include photon absorption remote sensing (PARS) radiative absorption contrast. Some biological samples, such as connective tissue (fibrin, collagen), can be well represented by this contrast. Another example can involve visualization of linear backscattered light from a sample to highlight structural morphology. Other similar extractions can be processed, visualized, and displayed through a user interface, where various aspects of the time domain are extracted to generate similar conceptual visualizations. Additionally, various combinations, products, and ratios of these visualizations can be created to derive additional useful contrast.

[0245] For example, the emissive and non-emissive contrasts may be summed to generate a measure of total area absorption, while their ratio provides information related to the absorption quantum efficiency within the probed region. Such combinations may provide users with unique sets of information over single-component visualizations. Furthermore, colorizations of such combinations may be created either through algorithmic means or machine learning models 3802, 3822, 3922 to emulate other colorizations known to each user's field. As an example, while imaging tissue for pathological analysis, it may be useful to colorize data to replicate the appearance and contrast of existing staining procedures, such as hematoxylin and eosin or toluidine blue.

[0246] FIG. 62 shows an example user interface 6200 for scanning and processing one or more images using an imaging device. The user interface UI 6200 may be configured to facilitate a user's control and operation of the imaging device and may be part of the architecture of FIG. 38A, FIG. 38B, or FIG. 39. The user interface UI 6200 includes a scan control interface that includes a preview area 6250 in which the image is scanned. As rows of pixels are scanned or otherwise generated, the preview area 6250 may show the progress of the scan or the generation of the digital image.

[0247] An operator of the photon absorption remote sensing PARS system 3801, 3901 may be notified when it is safe to perform a scan. In some embodiments, if any safety conditions are not met, the operator of the photon absorption remote sensing PARS system 3801, 3901 may be prevented from performing a scan. Next to the preview area 6250, a number of icons 6210, 6220, 6230, 6240, 6260 are shown in a second area 6280.

[0248] For example, icon 6210 may indicate that the sample in question is not properly positioned or installed for scanning, or that pressure is not being applied correctly for scanning. Icon 6220 may indicate that the laser used for scanning is not heated to a sufficient level. Icon 6230 may indicate that the laser used for scanning is overheated. Icon 6240 may indicate that the scan enclosure area is not safely closed. Icon 6260 may indicate that all safety conditions have been met and scanning is allowed to proceed.

[0249] A progress bar in the second area 6280 may indicate the progress of the scan, and the user may start or stop the scan using GUI elements located in the second area 6280. In some embodiments, a collection of image processing tools may be included in the user application to assist the user in modifying and manipulating the visualization. Such image processing tools may include, but are not limited to, brightness-contrast level modifications, sharpness filters, blur filters, hue-saturation adjustments, etc. As a user option, one or more processing steps may be configured as one or more preset options, and a set of processing steps may be selected by the user to be quickly executed on subsequent data acquisitions.

[0250] In some embodiments, one or more GUI elements of a user interface rendered by a user application may display one or more machine learning results (e.g., inferences 2517) to assist the user in segmentation, image optimization, labeling, diagnosis, etc. The user application may include the following example tools to assist the user in image analysis: a tool to automatically select tumor boundaries, a tool to perform an image search in a Photon Absorption Remote Sensing (PARS) or H&E database to provide similar examples (e.g., from an architectural or diagnostic standpoint), a tool to provide an automated diagnosis to serve as quality assurance for pathologists, a tool to automatically identify tumor type, including treatment management, a tool to allow the user to segment prominent subregions of tissue to highlight cell nuclei, fibrous tissue, melanin, adipose tissue, red blood cells, etc.

[0251] In some embodiments, the image is enabled to be annotated by one or more users (e.g., medical professionals) through a user interface UI rendered by a user application. FIG. 63 shows an example user interface 6300 for displaying an annotated image 6350. User selection(s) are made via GUI elements in an annotation selection area 6310 to enable display of some or all comments 6320a, 6320b, 6320c that may be made by different users. A quick hide button located in the lower left corner of the user interface UI 6300 enables the user to hide all comments to view the image without comments. This annotation user interface UI 6300 is enabled to be accessed remotely through an online viewer, such as an online viewer application from a website or mobile application.

[0252] 41 shows an example machine learning architecture 4100 that may be used to train the image generators 3812, 3912, or the image generators in the machine learning model 3802. The image generators 3812, 3912 may be, for example, colorized machine learning models trained using a generative adversarial network (GAN), which may comprise, for example, a cycle-consistent generative adversarial network (CycleGAN) model.

[0253] The image generators 3812, 3912 may be colorized machine learning models trained using, for example, a conditional generative adversarial network (cGAN), which may comprise a pix2pix model.

[0254] The colorized machine learning model may comprise a neural network. The neural network 4300 depicted in FIG. 43 may comprise an input layer, multiple hidden layers, and an output layer. The input layer receives input features. The hidden layer maps the input layer to the output layer. The output layer provides the predictions (e.g., inference) of the neural network. Each hidden layer may comprise multiple nodes, which may comprise weights, biases, and inputs from previous layers. Weights are parameters in the neural network that transform input data in the hidden layer of the network.

[0255] In some embodiments, the initial weights of the neural network model 4300 in the colorized machine learning model are allowed to be transferred from another neural network model (the "donor model") that was trained on a large dataset of stained H&E images. In this configuration, instead of being assigned random values ​​before training the neural network model 4300, each weight in one or more initial layers of the neural network model 4300 can be assigned a value equal to a respective value from the corresponding one or more initial layers of the donor model trained on the large dataset.

[0256] In some embodiments, during training of the neural network model 4300, all layers of the neural network model 4300 are trained and fine-tuned, and the weights are updated accordingly.

[0257] In some embodiments, during training of the neural network model 4300, the weights of one or more initial layers are kept constant (i.e., equal to the weights from one or more initial layers of the donor model), and throughout the training process, only the weights of subsequent layers (after the initial layers) of the neural network model 4300 are trained or fine-tuned during training.

[0258] The CycleGan model comprises a first GAN having a first generator model 4103 and a first identifier model 4107, and a second GAN having a second generator model 4113 and a second identifier model 4117.

[0259] During training, in each training iteration, a true total absorption (TA) image 4101 may be obtained from an existing photon absorption remote sensing (PARS) image database and sent to a first generator model 4103. The first generator model 4103 may comprise a neural network configured to generate a simulated stained image 4105 ("fake" staining) based on the TA image 4101. A fake total absorption TA image 4111 is then generated by a second generator model 4113 based on the simulated stained image 4105. A first loss, a cycle consistency loss 4120, may be calculated based on comparing the true total absorption TA image 4101 and the fake total absorption TA image 4111. This loss 4120 is then used to update the weights of the first generator model 4101 and the second generator model 4113.

[0260] The simulated stained image 4105 may be processed by a first identifier model 4107 to generate an output, which may be further processed through a classification matrix 4109 to generate a first identifier output. The identifier model 4107 is configured to predict how likely the simulated stained image 4105 is from a target image collection (e.g., a collection of actual stains 4115).

[0261] During the same training iteration, labeled stained images 4115 are obtained, for example, from an existing stained image database. The labeled stained images 4115 may be processed by a second classifier model 4117 to generate an output, which may be further processed through a second classification matrix 4119 to generate a second classifier output.

[0262] The first and second identifier outputs may be used to calculate a second loss 4125. Based on one or both of the first loss 4120 and the second loss 4125, the processor may update the weights of the first generator model 4103, the second generator model 4113, the first identifier model 4107, and the second identifier model 4117.

[0263] Training may stop when the first or second loss, or both losses, reach a threshold, or may stop after a predetermined number of iterations. Once trained, the first generator network 4103 can be deployed as part of an image generator 3812, 3912, or machine learning model 3802 at inference time to generate one or more simulated stained images 3806, 106.

[0264] In some embodiments, the colorized machine learning model may comprise a one-shot GAN, a type of single-image GAN, which generates images from a training set as small as a single image, which is suitable for applications or settings where samples are limited, such as in tissue structures.

[0265] (Obtaining ground truth data for training and image retrieval) In some cases, the labeled stained images 4115 (training data for the machine learning architecture 4100) may be obtained from conventional chemical staining processes, such as spectroscopy-based methods.

[0266] 42 shows an exemplary process 4200 for preparing one or more training data 4205, 4207 for training an image generator 3812, 3912, or an image generator in a machine learning model 3802. An unstained tissue section 401 may be processed by a photon absorption remote sensing PARS system, such as total absorption-photon absorption remote sensing TA-PARS, to generate an unlabeled multi-channel image 4205, which may be provided to the machine learning architecture 4100 as a true total absorption TA image 4101.

[0267] The unstained tissue section 401 may be subjected to a conventional chemical staining process to obtain a stained slide 4203, which may be imaged under a brightfield microscope to generate a labeled stained image 4207, which may be provided to the machine learning architecture 4100 as a labeled stained image 4115.

[0268] FIG. 46 shows two hypothetical (simulated) stained photon absorption remote sensing PARS images: one simulated stained hematoxylin and eosin (H&E) image 4610 and one simulated stained toluidine blue image 462, both of which can be used as true total absorption TA image 4101 during training for different stain image generation processes.

[0269] Figure 47A shows an example of an unlabeled photon absorption remote sensing PARS virtual H&E image 4700 generated by a photon absorption remote sensing PARS system, which may be used as input to architecture 4100 in the form of a true total absorption TA image 4101. The unlabeled photon absorption remote sensing PARS virtual H&E image 4700 is correlated to a historical labeled stained (H&E) image 4750 in Figure 47B, which may be provided to machine learning architecture 4100 as labeled stained image 4115.

[0270] For example, for colorization of paraffin-embedded slides, a photon absorption remote sensing (PARS) image of a tissue sample can be generated, and then the tissue sample can be chemically stained with a stain of interest, generating a one-to-one correspondence dataset for training the colorization machine learning mode in architecture 4100.

[0271] For fresh tissue colorization, a photon absorption remote sensing PARS image of the tissue can be captured before processing the tissue through a conventional histopathology workflow, which generates correlation sections for training the colorization machine learning mode in architecture 4100.

[0272] To train the machine learning models 3802, 3822, 3922 to make one or more inferences comprising a diagnosis on the virtual histology slides, multiple pathologists are enabled to manually label a dataset of tissue slides to identify the location, type, grade, etc. of cancer within each tissue slide. The labeled datasets are then used to train the machine learning models 3802, 3822, 3922, which are then able to make appropriate inferences about one or more photon absorption remote sensing PARS images from the photon absorption remote sensing PARS system.

[0273] Assuming that historically virtually labeled images are comparable to traditional images labeled by pathologists, it may be possible to leverage existing labeled databases to provide training data for diagnostic algorithms.

[0274] By finding similarly labeled structures or images between the existing H&E database and the photon absorption remote sensing PARS data, the system may be able to train machine learning models 3802, 3822, 3922 to automatically label the photon absorption remote sensing PARS data to make one or more inferences comprising a diagnosis of the photon absorption remote sensing PARS features.

[0275] In some embodiments, an image generated based on a conventional tissue image or a photon absorption remote sensing PARS signal may contain different structures therein. The machine learning model 3802, 3822, 3922 may receive an image generated based on a conventional tissue image or a photon absorption remote sensing PARS signal (an "input image") and process the input image to generate a colorized image 5600 that is simultaneously stained or colorized with different stains.

[0276] For example, the basal layer of the input image may be stained with toluidine blue, while the inner connective tissue of the input image may be stained with H&E. For example, based on the raw data contained within the input image, certain regions of tissue may be best highlighted with H&E staining, while other areas may be best highlighted with Masson's Trichrome. The machine learning models 3802, 3822, 3922 may be trained based on historical data generated by a pathologist or other expert, the historical data comprising different structures of the tissue image and the corresponding stains for each of the different structures.

[0277] 56 shows an exemplary multi-stain image 5600 that may be generated by machine learning models 3802, 3822, 3922. In some embodiments, the different colored regions may be generated by color shifting the H&E image.

[0278] In fact, without virtual staining, the simultaneous or sequential use of histochemistry, IHC, and FISH agents on a single tissue section is impossible. Labeling processes can result in irreversible structural and chemical changes that render the specimen unacceptable for subsequent analysis. Thus, each section must be independently sectioned, mounted, and stained—a technically challenging, expensive, and time-consuming workflow. Trained histotechnicians can spend several hours preparing sections for examination, and some labeling protocols require overnight incubations, with steps spaced over multiple days. Therefore, repeating staining in a stepwise fashion or generating additional stains can delay diagnostic and treatment timelines and reduce patient outcomes. Furthermore, running multiple stained sections can rapidly consume valuable diagnostic samples, especially when diagnostic material is derived from needle core biopsies. This increases the likelihood that the patient will need to undergo further procedures to collect additional biopsy samples, resulting in delayed diagnoses and significant patient stress.

[0279] By simultaneously capturing both absorption fractions, various embodiments of the photon absorption remote sensing PARS system described herein are enabled to recover rich biomolecular contrast, such as quantum efficiency ratios, not provided by other independent modalities. In photon absorption remote sensing PARS, optical relaxation processes (radiative and non-radiative) are observed after a targeted excitation pulse is incident on the sample. Radiative relaxation generates light emission from the sample, which is then directly measured. Non-radiative relaxation causes local thermal modulation and, if the excitation event is fast enough, pressure modulation within the excitation region. These transients induce nanosecond-scale fluctuations in the local optical properties of the sample, which are captured by the confocal detection laser. Additionally, confocal detection is enabled to measure local light scattering prior to excitation. Overall, various embodiments of the photon absorption remote sensing PARS system described herein can simultaneously capture radiative and non-radiative absorption and light scattering from a single excitation event.

[0280] As can be seen in FIG. 56 , the multi-stain image 5600 has five different regions with different tissue structures: 5610, 5620, 5639, 5640, and 5650. Region 5610 is stained a light purple color (a), which may be a Masson's Trichrome stain, typically used to distinguish different types of connective tissue. Regions 5620 and 5640 are stained a blue color (b), which may be a PAS stain, used to identify areas of fungal infection within tissue. Regions 5630 and 5650 are stained a pink-purple color (c), which may be an H&E stain, used to distinguish different layers of epithelium and subcutaneous gland structures. In some embodiments, the machine learning models 3802, 3822, 3922 may automatically determine the most appropriate stain or color for a particular region within the image and apply the most appropriate stain or color to the particular region within the image.

[0281] (Photon absorption remote sensing PARS multiple staining results using photon absorption remote sensing PARS feature vectors) In some embodiments, the multi-stained image is generated based on a photon absorption remote sensing PARS feature vector (e.g., the photon absorption remote sensing PARS data vector shown in FIG. 87) and a photon absorption remote sensing PARS time-domain clustering method. Some examples of photon absorption remote sensing PARS multi-stained images are presented in FIG. 88, which shows three different virtual stains 8820, 8840, 8860 generated from the same initial photon absorption remote sensing PARS dataset 8800. In this example, the photon absorption remote sensing PARS data vector (similar to the example shown in FIG. 87, containing photon absorption remote sensing PARS amplitude and time-domain features) is then passed to a series of GAN networks used to render the virtual stained results.

[0282] Figure 88 shows an exemplary photon absorption remote sensing PARS virtual multi-stain image based on the same photon absorption remote sensing PARS image data 880. An RGB representation of the photon absorption remote sensing PARS image data 8800 is shown on the left, and three different virtual stains 8820, 8840, 8860 are shown on the right, which have been generated from the raw photon absorption remote sensing PARS image data 8800. The multi-stain result is generated using photon absorption remote sensing PARS feature vector data, which comprises multiple first- and second-order features comprising time-domain features.

[0283] In some embodiments, the image generator is designed to better leverage photon absorption remote sensing (PARS) and ground truth image data to generate accurate staining transformations. For example, an exemplary neural network within the image generator may optimize the perceived similarity between the photon absorption remote sensing (PARS) virtual staining image and the ground truth image using a trained image patch similarity or perceptual-based loss such as a "VGG" network. Additionally, more advanced GAN architectures (e.g., Wasserstein_GAN with gradient penalty, or Unrolled_GAN) can be used to generate more robust transformations.

[0284] In some embodiments, the initial weights of the machine learning models 3802, 3822, 3922 are allowed to be transferred from another neural network model ("donor model") trained on a conventional stained H&E image dataset. In this configuration, instead of being assigned random values ​​prior to training of the machine learning models 3802, 3822, 3922, each weight in one or more initial layers of the machine learning models 3802, 3822, 3922 may be assigned a value equal to each value from the corresponding one or more initial layers of the donor model trained on a conventional stained H&E image dataset.

[0285] The donor model is enabled to train on a dataset of conventional stained H&E images. These stained H&E images may be obtained from conventional chemical staining processes, such as spectroscopy-based methods. For example, the training data for the donor model may comprise a group of stained H&E images (ground truth data) and a group of corresponding grayscale H&E images converted from the group of stained H&E images. During training, the donor model receives as input the group of corresponding grayscale H&E images and may output corresponding colorized H&E images that are compared with the ground truth data, which may trigger updates to the donor model weights during each training iteration.

[0286] In another example, the training data for the donor model may include a group of stained H&E images (ground truth data) and a corresponding group of channel data for a first channel (e.g., an H channel) and a second channel (e.g., an E channel) for each stained H&E image in the group. The channel data for the H channel or E channel may include amplitude and / or intensity values ​​for each channel, similar to, for example, the RGB channels of a conventional color image.

[0287] During training, the donor model may receive groups of channel data as input and output corresponding colorized H&E images that are compared with ground truth data, and the comparison may trigger an update of the donor model weights during each training iteration.

[0288] As described above, the initial weights of the machine learning models 3802, 3822, 3922 can be obtained from a trained donor model. In this configuration, each weight of one or more initial layers of the machine learning models 3802, 3822, 3922 can be assigned a value equal to each value from the corresponding one or more initial layers of a donor model trained on a conventional stained H&E image dataset.

[0289] In some embodiments, during training of the machine learning models 3802, 3822, 3922, all layers of the machine learning models 3802, 3822, 3922 are trained and fine-tuned, and the weights are updated accordingly.

[0290] In some embodiments, during training of the machine learning model 3802, 3822, 3922, the weights of one or more initial layers are kept constant (i.e., equal to the weights from one or more initial layers of the donor model), and throughout the training process, only the weights of subsequent layers (after the initial layers) of the machine learning model 3802, 3822, 3922 are fine-tuned during training or training.

[0291] After training, the machine learning models 3802, 3822, 3922 may receive photon absorption remote sensing PARS images and / or photon absorption remote sensing PARS signals during inference and generate corresponding photon absorption remote sensing PARS virtual H&E images.

[0292] 57A , image generator 5750, which may be image generator 3812, 3912 or an image generator within machine learning model 3802, may include two neural network models 5712, 5722. First neural network model 5712 may be, by way of example, a cycleGAN trained to generate a simulated grayscale H&E image 5715 based on a total absorption-photon absorption remote sensing TA-PARS image 5720 from a photon absorption remote sensing PARS system 5710, and second neural network model 5722 may be, by way of example, a conditional GAN ​​(e.g., pix2pix) trained to generate a simulated color H&E image 5730 based on the simulated grayscale H&E image 5715 from first neural network model 5712.

[0293] The first neural network model 5712 is enabled to be trained based on a historical set of total absorption-photon absorption remote sensing TA-PARS data and corresponding grayscale H&E images. The second neural network model 5722 can be trained on a historical set of grayscale H&E images and corresponding stained H&E images, which may be obtained from a conventional chemical staining process, such as a spectroscopy-based method, or may be obtained from a dataset of virtual grayscale and color H&E images.

[0294] 57B , image generator 5850, which may be image generator 3812, 3912 or an image generator within machine learning model 3802, may include two neural network models 5812, 5822. First neural network model 5812 may be, by way of example, a cycleGAN trained to generate a separated H channel 5815 and a separated E channel 5817 based on a total absorption-photon absorption remote sensing TA-PARS image 5820 from a photon absorption remote sensing PARS system 565, and second neural network model 5822 may be, by way of example, a conditional GAN ​​(e.g., pix2pix) trained to generate a simulated color H&E image 5830 based on the separated H channel 5815 and the separated E channel 5817 from first neural network model 5812.

[0295] The first neural network model 5812 is enabled to train based on a historical set of total absorption-photon absorption remote sensing TA-PARS data and corresponding separated H-channel and E-channel data. The second neural network model 5822 can be trained on a historical set of separated H-channel and E-channel data and corresponding stained H&E images, which may be obtained from a conventional chemical staining process, such as a spectroscopy-based method, or may be obtained from a dataset of virtual color H&E images.

[0296] In some embodiments, the cycleGAN model may be implemented to be a multi-task cycleGAN model configured to perform multiple tasks, including, for example, (1) super-resolving a relatively low-resolution image into a high-resolution image (increasing the resolution of the image), (2) generating an H-stained total absorption-photon absorption remote sensing TA-PARS image, (3) generating an E-stained total absorption-photon absorption remote sensing TA-PARS image, and (4) generating an H&E-stained total absorption-photon absorption remote sensing TA-PARS image. When the cycleGAN model is trained on high-resolution images from an image dataset to convert grayscale images to color H&E images, the cycleGAN model has learned the grayscale-to-color conversion and improved resolution.

[0297] In some embodiments, architectures 3800, 3850, 3900 may comprise an image retrieval machine learning model (“image retrieval model”), which may be part of machine learning models 3802, 3822, 3922 or may be a separate machine learning model, that outputs one or more labeled images based on a given unlabeled input image. For example, the input image to the image retrieval model may be an unlabeled photon absorption remote sensing (PARS) virtual H&E image 4700, which does not have labels for any regions of interest within the image 4700. Once properly trained, the image retrieval model may output at least one labeled image (e.g., labeled stain image 4750 of FIG. 47B ) from a plurality of existing (e.g., historical) labeled images stored in an image database, where the output labeled image has the highest correlation score as the input image.

[0298] The correlation score, in some embodiments, may be determined based on features extracted from an input image (e.g., an unlabeled photon absorption remote sensing PARS virtual H&E image 4700). For example, a higher correlation score may be assigned to an image exhibiting features having greater similarity to the input image. A minimum threshold may be predetermined so that any existing labeled image(s) from the database having a correlation score above the minimum threshold may be selected as output by the image retrieval model.

[0299] Thus, the image retrieval model can be configured to acquire one or more existing labeled images from an existing imaging database based on an input image that has not yet been labeled. The acquired labeled images, which can be stained, can be used as training data for the machine learning architecture 4100 as labeled stained images 4115.

[0300] In some embodiments, the image retrieval model may be configured to obtain one or more pre-existing labeled images from an existing image database based on an input image that has not yet been labeled, and to send the one or more pre-existing labeled images to the user interface 4000 of the user application 3825, 3925 to assist in further medical diagnosis. The input image and output image(s) may be stained or unstained.

[0301] For example, a clinician, through a user application 3825, 3925, may submit a new medical image showing a patient's lungs to the image retrieval model to obtain similar medical images that have already been labeled (or annotated). The image retrieval model may output one or more output images (e.g., through the user interface UI 4000) that are enabled to assist the clinician in understanding the input medical image showing the patient's lungs. For example, if the one or more output images comprise images generally labeled with pneumonia, it is likely that the patient in the input medical image also has pneumonia.

[0302] In some embodiments, the percentage likelihood of a pathological finding in the current image acquisition may be similar to, and may be determined based on, a previous diagnosis of a similar or identical pathological finding in one or more previous images in the photon absorption remote sensing PARS acquisition.

[0303] For example, referring now to FIG. 55 , a heat map 5500 is shown. The heat map 5500 includes several heated regions 5510, 5520, 5530, and 5540. Each respective region 5510, 5520, 5530, or 5540 may represent a corresponding percentage of likelihood of pathology or pathological findings (e.g., malignancy) overlaid on an H&E stained image 5550 to aid in diagnosis and intraoperative guidance. This may help medical professionals read the current H&E stained image and draw relevant conclusions. For example, the heat map 5500 may assist a surgeon, who may not be experienced in reading pathology slides, in determining where to resect cancerous tissue during surgery.

[0304] Heated region 5510 appears to be the darkest color, followed by heated region 5530, which is then further followed by heated regions 5540 and 5520. This may indicate, for example, that the area covered by heated region 5510 has a high probability or percentage of pathology (e.g., malignancy), for example, 80%, or in the range of 80-100%, the area covered by heated region 5530 has a medium to high probability or percentage of pathology (e.g., malignancy), for example, 50%, or in the range of 50-79%, and the area covered by heated regions 5540 and 5520 has a low probability or percentage of pathology (e.g., malignancy), for example, 30%, or in the range of 30-49%. Areas not covered by any heated regions (blue) have a very low likelihood or percentage of pathological findings (e.g., malignant tumors), e.g., 0%-29%.

[0305] In some embodiments, architectures 3800, 3850, 3900 may generate an inference based on a sample (e.g., an image) of a probability of disease for at least one region within the sample, where the probability of disease is determined based on multiple features and complementary data streams received by the machine learning architecture.

[0306] In some embodiments, the inference may comprise a heat map identifying one or more regions of the sample and a corresponding probability of disease for each of the one or more regions of the sample.

[0307] In some embodiments, the corresponding probability of disease for each of one or more regions of the sample is illustrated by the corresponding intensity of color shown in each region in the heatmap, which is enabled to guide clinicians in identifying and managing disease in patients associated with the sample.

[0308] In some embodiments, the image retrieval model can be a standalone system architecture without a photon absorption remote sensing PARS system. In some embodiments, the image retrieval model may comprise a convolutional neural network (CNN) and may further comprise an autoencoder.

[0309] For molecular identification: Photon absorption remote sensing (PARS) images of tissue scans can be acquired from a photon absorption remote sensing (PARS) system before staining with molecular stains to allow image-based correlation. Photon absorption remote sensing (PARS) images can be acquired and compared to ground truth spectroscopy including mass spectrometry, mass cytometry, fluorescence spectroscopy, and transient absorption spectroscopy.

[0310] In some embodiments, instead of a historically stained image 4207 acquired through a conventional chemical staining process, the labeled stained image 4115 is a labeled photon absorption remote sensing PARS image from a photon absorption remote sensing PARS image database.

[0311] In some embodiments, the labeled photon absorption remote sensing PARS images are automatically labeled based on unlabeled photon absorption remote sensing PARS images prior to training of the neural network. In some embodiments, automatically labeling the unlabeled photon absorption remote sensing PARS images may comprise labeling the unlabeled photon absorption remote sensing PARS images based on existing labeled stained images from a database, where the existing labeled stained images and the unlabeled photon absorption remote sensing PARS images share structural similarities.

[0312] In some embodiments, the existing labeled stained images are obtained from an existing H&E database. (An example colorization process using cycleGAN and performing noise removal in the process) In some embodiments, as shown in FIG. 58 , the image generator, which may be image generator 3812, 3912, or an image generator within machine learning model 3802, may comprise a cycleGAN 5812 trained to generate a simulated color H&E image 5830 based on a total absorption-photon absorption remote sensing TA-PARS image 5820 from a photon absorption remote sensing PARS system 5810.

[0313] In some embodiments, total absorption-photon absorption remote sensing TA-PARS images 5820, which may comprise radiative and non-radiative absorption images, are preprocessed via preprocessing module 5811 and then virtually stained through cycleGAN 5812 to generate simulated color H&E images 5830. Preprocessing module 5811 may comprise, for example, a self-supervised Noise2Void denoising convolutional neural network (CNN) 5813 and an error correction submodule 5823 for pixel-level mechanical scanning and error correction. Implementations described herein are enabled to significantly enhance the recovery of submicron tissue structures, such as nucleoli and chromatin distribution. Preprocessed photon absorption remote sensing PARS image data 5826 is then virtually stained using cycleGAN 5812 by applying virtual stains to the preprocessed photon absorption remote sensing PARS image data 5826, which may comprise images representing thin, unstained sections of malignant human skin and breast tissue samples.

[0314] Figure 58 illustrates an improved virtual staining and image processing architecture 5800 for emulating tissue structure images that cannot be effectively distinguished from standard H&E pathology. The architecture presented in Figure 58 includes an optimized image preprocessing module 5811 and a cycle-consistent generative adversarial network (CycleGAN) 5812 for virtual staining. CycleGAN virtual staining does not require pixel-to-pixel alignment of training data. However, semi-aligned data is used here to reduce optical artifacts while improving the integrity of the virtual staining. In addition, the image preprocessing module 5811 reduces measurement-to-measurement variability during signal acquisition through the implementation of pulse energy correction and image denoising using a self-supervised Noise2Void network. An error correction submodule 5823 is implemented for the removal of pixel-level mechanical scan position artifacts that blur subcellular features. These enhancements provide significant improvements in the clarity of small tissue structures, such as nucleoli and chromatin distribution. Loosely or semi-registered, CycleGAN5812 facilitates the highest quality and most accurate virtual staining of any photon absorption remote sensing PARS virtual staining method explored to date. When applied to images containing whole slide sections of excised human tissue, Architecture 5800 provides detailed emulation of intracellular and nuclear diagnostic features comparable to the gold-standard H&E. This Architecture 5800 represents an important step toward the development of label-free virtual staining microscopy. Successful label-free virtual staining paves the way for the development of in vivo virtual tissue architectures, enabling pathologists to instantly access multiple specialized stains from a single slide, increasing diagnostic confidence and improving timelines and patient outcomes.

[0315] In one example embodiment, a label-free total absorption-photon absorption remote sensing TA-PARS image 5820 is captured using a photon absorption remote sensing PARS system 5810. Briefly, a 400 ps pulse, 50 kHz 266 nm UV laser (Wedge XF266, RPMC) is used to excite the sample, simultaneously inducing nonradiative and radiative relaxation processes. The nonradiative relaxation processes are sampled as time-resolved photothermal signals, and the photoacoustic signal is probed with a continuous-wave 405 nm detection beam (OBIS-LS405, Coherent). This detection beam is co-aligned and focused on the sample with the excitation light using a 0.42 numerical aperture (NA) UV objective lens (NPAL-50-UV-YSTF, OptoSigma). Radiative emission (>266 nm) from the radiative relaxation processes, as well as transmitted detection light, are collected using a 0.7 NA objective lens (278-806-3, Mitutoyo). The detection wavelength of 405 nm and the radioactive emission are spectrally separated and each directed onto an avalanche photodiode (APD130A2, Thorlabs).

[0316] To form an image, a mechanical stage moves the sample in an "S"-shaped scan pattern, laterally separating excitation events on the sample (approximately 250 nm / pixel). With each excitation pulse, a few hundred nanoseconds of time-resolved signal from each system photodiode is digitized at a 200 MHz rate (CSE1442, RZE-004-200, Gage Applied). A portion of the collected signal is pre-excitation and used to form a scattering image of the sample in its unperturbed state. Non-radiative image pixels are then derived as the percentage modulation in the detected scattering (post-excitation). Next, radiative image pixels are obtained from the peak emission amplitude recorded after each excitation event. The pixels are then arranged in an orthogonal grid based on stage position feedback, forming three co-registered, label-free image contrast stacks: non-radiative, radiative, and scattered. Finally, the excitation pulse energy and detection power recorded throughout imaging are used to correct for image noise caused by laser power and pulse energy fluctuations.

[0317] Briefly, the entire tissue area is divided into subsections (500 x 500 pm), each scanned individually at its optimal focus position. Using relative stage position and a small amount of overlap (approximately 5%), these sections are stitched together and blended into a single whole-slide image.

[0318] (Photon Absorption Remote Sensing PARS Data Preprocessing) In addition to correcting for noise due to laser power and pulse variations, a Noise2Void (N2V) denoising convolutional neural network (CNN) 5813 is used in some embodiments to further denoise raw photon absorption remote sensing PARS images. Unlike many other conventional CNN-based denoising methods, the N2V denoising CNN 5813 does not require paired training data for both noisy and clean image targets. It assumes that image noise is independent on a pixel-by-pixel basis, while the underlying image signal contains statistical dependencies. Thus, promoting a simple approach to denoising photon absorption remote sensing PARS images, denoising CNNs for the radioactive and non-radioactive contrast channels were used to train separately. Exemplary machine learning models were trained on a body of raw data taken from both human skin and breast whole-slide images. A set of 125 photon absorption remote sensing PARS tiles was used to generate models for each of the radioactive and non-radioactive images. Each model was trained over a series of 300 epochs with 500 steps per epoch using 96 neighboring pixels. The final processing step before training the virtual staining model was to correct for scan-related image artifacts, which become apparent after denoising the raw data. These artifacts are line-by-line distortions caused by slight mismatches in the mechanical scanning fast axis (x-axis) velocity, resulting in uneven spatial sampling. Thus, we correct these distortions using a custom jitter or error correction submodule 5823 before colorization using CycleGAN 5812.

[0319] (Preparing the dataset for model training) In some embodiments, the CycleGAN image transformation model 5812 is enabled for use in virtual staining. While the CycleGAN 5812 is enabled to learn image-domain mapping using unpaired data, it may be advantageous to provide the model with semi- or loosely aligned images as a form of high-level labeling to better guide the training process and enhance the model. Since one-to-one H&E and PARS whole-slide image pairs are available, it seems most appropriate to prepare the datasets accordingly. However, since the two datasets are inherently unaligned, a simple affine transformation is used. The affine transformation allows for shearing and scaling, as well as rotation and translation. Generally, it is sufficient for changes in the tissue layout on the slide that occur during the staining process. The affine transformation is determined using the geometric relationship between the three alignment points. This discovered relationship, i.e., the transformation matrix, is then applied to the entire slide image in both the non-radioactive and radioactive channels.

[0320] Figures 82A and 82B show an example visualization of the data preparation process and inversion. In Figure 82A, as shown in the example schematic block diagram (8200), the aligned full absorption image and H&E image are cut into matching tiles to generate a loosely aligned dataset. The pixel intensities of the full absorption image are then inverted to provide better initialization for training. Finally, the dataset is used to train a virtual colorization model (e.g., CycleGAN5812).

[0321] In Figure 82B, the model is applied iteratively to overlapping tiles of the total absorption image to form a virtually stained image, as shown in an exemplary schematic block diagram 8250. The overlapping tiles are then averaged to form the final virtual colorization.

[0322] After the total absorption (TA) and H&E images of the entire slide (PARS) are aligned, the entire image is sliced ​​into small tiles (512 x 512) that are paired together, as shown in Figure 82A. The total absorption (TA) image shows the raw radioactive (blue) and non-radioactive (red) images in a single combined color image. However, during training, the network uses inverted TA patches, where the radioactive and non-radioactive image pixel intensities are inverted before they are stacked into the color image. Inverting these channels provides a color image in which the white background of the PARS data is mapped onto the white background of the H&E data. After training is complete, the model can be applied to larger images, such as the entire entire slide, by virtually staining the 512 x 512 tiles in parts. This process is shown in Figure 82B, where the overlapping regions are averaged together in the final virtually stained image. Here, a 50% overlap is used.

[0323] In one study, two CycleGAN models were trained on loosely paired data using the previously described alignment and dataset preparation methods. One model was trained on human skin tissue, and the other was trained on human breast tissue. For each model, the training set consisted of 5,000 training pairs of size 512 × 512 pixels (128 × 128 pM) sourced from standard 40x magnification (250 nm / pixel) whole-slide images of each tissue type. The model generator was trained for 500 epochs with an early stopping criterion to terminate training when the loss no longer improved. The models were trained with a learning rate of 0.0002, a batch size of 1, and an 80 / 20% split of training and validation pairs. For comparison purposes, a pix2pix model and a standard unpaired CycleGAN model were also trained for each tissue type. The pix2pix model was trained on the same dataset as the paired CycleGAN model, with a tighter alignment procedure and the same model parameters. For unpaired training of the CycleGAN model, the same number of training pairs was used, but the TA and H&E domains were sourced from different whole-slide images of the same tissue type.

[0324] A current drawback of raw PARS images is the presence of measurement noise. Improvements in PARS image quality were achieved by measuring the detection power and excitation pulse energy. Image noise was then corrected based on laser energy fluctuations. Even with energy-based correction, measurement noise still exists in the nonradiative signal. This additional noise disproportionately affects signals exhibiting low nonradiative relaxation, as it generates smaller nonradiative perturbations in the detection beam.

[0325] Paired or unpaired denoising methods can be applied to raw photon absorption remote sensing PARS data of total absorption-photon absorption remote sensing TA-PARS images 5820 to remove noise before colorization using an image generator such as cycleGAN 5812. Unpaired denoising algorithms do not require matched noisy and clean image targets for training, facilitating a simple self-supervised approach to denoising photon absorption remote sensing PARS images. Paired denoising algorithms can also be used on photon absorption remote sensing PARS images. For example, clean and noisy image pairs for training can be generated by acquiring two images of the same area, one with a high pulse energy and one with a low pulse energy. A high pulse energy will result in a lower noise (i.e., cleaner) image, while a low pulse energy will result in a noisier image target.

[0326] Figure 80 shows an example of raw photon absorption remote sensing PARS data 8010 in a total absorption-photon absorption remote sensing TA-PARS image 5820 that has been denoised using the Noise2Void (N2V) framework as found in Non-Patent Document 1, the entire contents of which are incorporated herein by reference. The denoising example in Figure 80 is adapted in Non-Patent Document 2, the entire contents of which are incorporated herein by reference.

[0327] After denoising the raw photon absorption remote sensing PARS data 8010 in a total absorption-photon absorption remote sensing TA-PARS image 5820 via preprocessing module 5811, the denoised image 8020 may contain mechanical scan-related jitter artifacts, as seen in FIG. 80. These artifacts are line-to-line distortions caused by slight mismatches in the mechanical scan fast axis speed, resulting in uneven spatial sampling. Prior to colorization with an image generator, a custom jitter or error correction submodule 5823 may be used to correct these distortions in the denoised image 8020, generating an artifact-free image 8030. The generated image 8030 may be used as preprocessed photon absorption remote sensing PARS image data 5826 for input to an image generator, such as cycleGAN 5812. FIG. 81 illustrates an example implementation 8100 of the error correction sub-module 5823 for correcting line-by-line jitter distortion in a denoised photon absorption remote sensing PARS image 8020.

[0328] The implementation 8100 of the error correction submodule 5823 shown in FIG. 81 determines the optimal pixel shift for a series of chunks spaced across a given row with overlap. The chunks are then moved to their appropriate locations, and the overlapping chunk areas are summed together to form an averaged corrected row. FIG. 81 illustrates three example chunks and their optimal pixel shifts. These shifts are determined by moving the chunks left and right until the minimum mean square error between the chunk and a reference row is reached. This reference is calculated as the average between the rows above and below the given row being corrected. The error correction submodule 5823 is implemented based on the assumption that the fast axis velocity profiles differ primarily for velocity sweeps in opposite directions and minimally for velocity sweeps in the matching direction. In this way, the upper and lower rows are captured in the same direction, and averaging them together provides a suitable row spacing to use as a reference for correction.

[0329] Figure 83 shows examples of raw non-radioactive and radioactive image channels after reconstruction and laser power reference correction. At high magnification, significant noise can be seen in the raw data channels. This motivates denoising as a preprocessing step. However, noiseless photon absorption remote sensing (PARS) image targets have not been available to train conventional denoising CNNs. Therefore, the N2V denoising CNN5813 is an ideal method because it enables effective denoising without a clean image target.

[0330] Figure 83 shows exemplary denoising results based on raw photon absorption remote sensing PARS image data 8300, which includes both raw non-radioactive and radioactive image channels, after execution of the N2V-based denoising CNN 5812 and error correction sub-module 5813. Three exemplary regions are shown at higher magnification to show the effect of the denoising and jitter correction algorithms. The structures imaged here show hair follicle uptake from human skin tissue. After denoising the raw data, the jitter artifacts seen in Figure 80 become apparent and are the main source of noise in the image. These sub-resolution shifts and image line-to-line distortions can be seen embedded in the noise, but resolving and correcting them is challenging. Denoising not only improves the quality of the raw data, but also helps enable jitter correction. As shown in Figure 83, most of the artifacts are removed after applying the correction sub-module 5823.

[0331] After denoising and jitter correction of the raw data, the whole-slide radioactive and non-radioactive images are registered to the ground truth H&E image. As mentioned above, a simple affine transformation is used here to account for changes in tissue layout that occur during the staining process, which can generate over 6000 closely registered 512 x 512 training pairs for a single 40x, 1 cm2 whole-slide image.

[0332] Traditionally, stains such as Verhoeff-van Gieson (VVG), which highlight normal or pathological elastic fibers, are required to visualize the internal elastic membrane of arteries. In clinical applications, VVG staining is sometimes combined with Masson's trichrome staining to differentiate collagen and muscle fibers within tissue samples. This is done to visualize potential increases in collagen associated with diseases such as liver cirrhosis and to evaluate muscle tissue morphology for pathological conditions affecting muscle fibers. In contrast, all of these structures are well-enhanced in PARS raw data. Currently, H&E virtual staining models flatten these structures during the image transformation process. However, this highlights the potential of using the rich PARS raw data to reproduce various clinically relevant contrasts beyond H&E staining. A practical application for PARS virtual staining is to provide several emulated histochemical stains from a single acquisition. Furthermore, entirely new histochemical-like contrasts may emerge based on intrinsic photon absorption remote sensing (PARS) contrast. The photon absorption remote sensing (PARS) system described herein may be able to provide contrast to biomolecules that are inaccessible to current chemical staining methods.

[0333] As described above, metric correction and Noise2Void-based image denoising are successfully applied to improve image quality. An error correction submodule is presented to reduce pixel-level mechanical scanning position artifacts that blur submicron-scale features. These enhancements provide significant improvements in the clarity of small tissue structures, such as nucleoli and chromatin distribution. In addition, a novel virtual staining process using semi-registered CycleGAN is presented. While semi-registered CycleGAN does not require registration like pix2pix, providing semi-registered data can enhance colorization quality by reducing the presence of optical artifacts. As described herein, emulated H&E images are generated from label-free photon absorption remote sensing (PARS) images with favorable quality and contrast compared to conventional H&E staining. The colorization performance represents the best current photon absorption remote sensing (PARS) virtual staining implementation. When applied to whole sections of unstained human tissue, the presented method enables accurate recovery of subtle structural and subnuclear details. With these improvements, photon absorption remote sensing PARS virtual H&E images may be effectively indistinguishable from gold-standard chemically stained H&E scans. In some embodiments, photon absorption remote sensing PARS label-free virtual stains have the potential to provide multiple histochemical stains from a single unlabeled sample, increasing diagnostic confidence and significantly improving patient outcomes.

[0334] FIG. 53 is a schematic diagram of a computing device 5300 that may be used to implement an image generator or a computing device used to train or run (at inference) machine learning models 3802, 3812, 3912.

[0335] As depicted, computing device 5300 includes at least one processor 5302 , memory 5304 , at least one I / O interface 5306 , and at least one network interface 5308 .

[0336] Each processor 5302 may be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

[0337] The memory 5304 may comprise any suitable combination of types of computer memory, located either internally or externally, such as, for example, random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), ferroelectric RAM (FRAM®), etc.

[0338] Each I / O interface 5306 allows the computing device 5300 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touchscreen, and microphone, or one or more output devices, such as a display screen and speakers.

[0339] Each network interface 5308 enables the computing device 5300 to communicate with, exchange data with, access and connect to network resources, provide applications, and run other computing applications by connecting to a network (or multiple networks) capable of carrying data, including the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, Public Switched Telephone Networks (PSTN), Integrated Services Digital Networks (ISDN), Digital Subscriber Lines (DSL), coaxial cable, optical fiber, satellite, mobile (e.g., 4G, 5G networks), wireless (e.g., Wi-Fi, WiMAX), SS7 signaling networks, fixed lines, local area networks, wide area networks, and any combination thereof.

[0340] For simplicity only, one computing device 5300 is shown, but the system 100 may include multiple computing devices 5300. The computing devices 5300 may be the same or different types of devices. The computing devices 5300 may be connected in a variety of ways, including being directly coupled, indirectly coupled via a network, distributed over a wide geographic area, or connected via a network (which may be referred to as "cloud computing").

[0341] For example, but not limited to, the computing device 5300 may be a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a personal data assistant, a mobile phone, a smartphone device, a UMPC tablet, a video display terminal, a game console, or any other computing device capable of being configured to perform the methods described herein.

[0342] FIG. 54 illustrates the processing performed by the processor of an exemplary embodiment of the machine learning system or architecture 3800, 3850, 3900. In operation 5402, a processor receives a plurality of signals from a sample, the signals comprising radioactive and non-radioactive signals.

[0343] In some embodiments, the plurality of signals comprises an absorption spectrum signal. In some embodiments, the plurality of signals comprises scattered signals. In some embodiments, the sample is an in vivo or in situ sample.

[0344] In some embodiments, the sample is unstained. In operation 5404, the processor extracts a plurality of features based on processing at least one of the plurality of signals, the features providing information of the contrast provided by at least one of the plurality of signals.

[0345] In some embodiments, the contrast may comprise one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast.

[0346] In some embodiments, processing the plurality of signals may comprise exciting the sample at an excitation location with an excitation beam, where the excitation beam is focused at or below the sample, and interrogating the sample with an interrogation beam directed towards the excitation location of the sample, where the interrogation beam is focused at or below the sample.

[0347] In some embodiments, extracting a plurality of features comprises processing both radioactive and non-radioactive signals. In some embodiments, the plurality of features is complemented by at least one informative feature of image data acquired from a complementary modality.

[0348] In some embodiments, the complementary modalities comprise at least one of ultrasound imaging, positron emission tomography (PET) scanning, computed tomography (CT) scanning, and magnetic resonance imaging (MRI).

[0349] In some embodiments, image data acquired from complementary modalities may include photoactive labels to contrast or highlight specific regions within the image. In some embodiments, the plurality of features is complemented by at least one informative feature of the patient information.

[0350] In some embodiments, the processing step comprises converting at least one of the plurality of signals into at least one image. In some embodiments, said transforming said at least one image comprises applying a simulation stain.

[0351] In some embodiments, the simulated staining comprises at least one of hematoxylin and eosin (H&E) staining, Jones staining (MPAS), PAS and GMS staining, toluidine blue, Congo red, Masson's trichrome staining, Lilly's trichrome, and Verhoeff's staining, immunohistochemistry (IHC), histochemical staining, and in situ hybridization (ISH).

[0352] In some embodiments, the simulation staining is made applicable to frozen tissue sections, archived tissue samples, or fresh, unprocessed tissue. In some embodiments, converting the at least one image comprises converting to at least two images and applying a different simulation stain to each of the images.

[0353] In some embodiments, converting to at least one image comprises applying a colorizing machine learning architecture. In some embodiments, the colorized machine learning architecture comprises a generative adversarial network (GAN).

[0354] In some embodiments, the colorized machine learning architecture comprises a cycle-consistent generative adversarial network (CycleGAN). In some embodiments, the colorized machine learning architecture comprises a conditional generative adversarial network (cGAN), which may comprise, for example, a pix2pix model.

[0355] In operation 5406, the processor applies the plurality of features to a machine learning architecture to generate inferences 2517 about the sample. In some embodiments, the inferences 2517 comprise at least one of survival time, drug response, drug resistance, phenotypic characteristics, molecular characteristics, mutational burden, tumor molecular characteristics, parasites, toxicity, inflammation, transcriptomic features, protein expression features, patient clinical outcome, suspicious signals, biomarker location or value, cancer grade, cancer subtype, tumor border area, and grouping of cancer cells based on cell size and shape.

[0356] In optional step operation 5408, the processor generates a signal to cause a user interface (UI) 4000 showing a visualization of the inferences 2517 to be rendered on a display device.

[0357] A set of instructions configured to train a GAN may include instructions to, at each training iteration, instantiate a machine learning architecture comprising a neural network having a plurality of nodes and weights stored in a memory device, acquire a true total absorption (TA) image, generate a simulated stained image based on the true total absorption TA image, generate a false total absorption TA image based on the generated stained image, calculate a first loss based on the generated false total absorption TA image and the true total absorption TA image, acquire a labeled stained image, calculate a second loss based on the generated simulated stained image and the labeled stained image, and update weights of the neural network based on at least one of the first loss and the second loss.

[0358] According to yet another aspect, a computer-implemented method is provided for training a machine learning architecture for generating simulated stained images. The machine learning architecture includes a neural network having a plurality of nodes and weights stored in a memory device. The method includes, at each training iteration, acquiring true total absorption (TA) images, generating simulated stained images based on the true total absorption TA images, generating false total absorption TA images based on the generated stained images, calculating a first loss based on the generated false total absorption TA images and the true total absorption TA images, acquiring labeled stained images, calculating a second loss based on the generated simulated stained images and the labeled stained images, and updating the weights of the neural network based on at least one of the first loss and the second loss.

[0359] In some embodiments, the simulated stained image is generated by a second neural network comprising a second set of nodes and weights, the second set of weights being updated during each iteration based on at least one of the first loss and the second loss.

[0360] In some embodiments, the fake total absorption TA image is generated by a third neural network comprising a second set of nodes and weights, the third set of weights being updated during each iteration based on at least one of the first loss and the second loss.

[0361] In some embodiments, calculating the second loss based on the generated simulated stained images and the labeled stained images may comprise processing the generated simulated stained images through a first identifier network, processing the labeled stained images through a second identifier network, and calculating the second loss based on respective outputs from each of the first identifier network and the second identifier network.

[0362] In some embodiments, the method may further comprise processing each output from each of the first and second identifier networks through a respective classification matrix before calculating the second loss.

[0363] In some embodiments, the machine learning architecture comprises a CycleGAN machine learning architecture. In some embodiments, the machine learning architecture comprises a conditional generative adversarial network (cGAN), which may comprise, for example, a pix2pix model.

[0364] In some embodiments, the labeled stain image is a labeled photon absorption remote sensing PARS image. In some embodiments, the labeled photon absorption remote sensing PARS images are automatically labeled based on unlabeled photon absorption remote sensing PARS images prior to training of the neural network.

[0365] In some embodiments, automatically labeling the unlabeled photon absorption remote sensing PARS image comprises labeling the unlabeled photon absorption remote sensing PARS image based on an existing labeled stained image from a database, wherein the existing labeled stained image and the unlabeled photon absorption remote sensing PARS image share structural similarities.

[0366] In some embodiments, the database is an H&E database. (Additional machine learning architectures for processing photon absorption remote sensing PARS imagery) Reference is now made to FIG. 64, which is an exemplary machine learning architecture 6400 for processing one or more outputs 6404 from a photon absorption remote sensing PARS system 6402. The photon absorption remote sensing PARS system 6402 may comprise one or more of total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multi-photon excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime-photon absorption remote sensing (PARS), and time-domain-photon absorption remote sensing (TD-PARS) subsystems, similar to the photon absorption remote sensing PARS systems 3801, 3901 previously described in connection with FIGS. 38A, 38B, and 39. The photon absorption remote sensing PARS system 6402 may be, for example, the photon absorption remote sensing PARS system from FIG. 5 above.

[0367] The photon absorption remote sensing PARS system 6402 can detect generated signals in the detection beam(s) returning from a given sample. These perturbations can include, but are not limited to, changes in intensity, polarization, frequency, phase, absorption, nonlinear scattering, and nonlinear absorption, and can be caused by various factors such as pressure, thermal effects, etc. The sample, being an unstained sample, can be an in vivo or in situ sample. For example, it can be tissue under the skin of a patient. In another example, it can be tissue on glass.

[0368] In some embodiments, a computer-implemented deep learning model 6406 is provided for processing photon absorption remote sensing PARS signals and / or image data. Input 6404 to the deep learning model 6406 may comprise a plurality of photon absorption remote sensing PARS signals comprising radiative and non-radiative signals, and / or a plurality of extracted features based on processing at least one of the plurality of signals, and may comprise features indicative of contrast provided by at least one of the plurality of signals (e.g., photon absorption remote sensing PARS data / photon absorption remote sensing PARS image / photon absorption remote sensing PARS features / photon absorption remote sensing PARS image features) and other related data (i.e., genomic data, clinical characteristics).

[0369] The deep learning model 6406 may be trained and deployed to generate one or more inferences 6408 based on the output 6404 from the photon absorption remote sensing PARS system 6402. The generated inferences 6408 may then be transmitted to a user application display device 6410 for further interpretation and / or display. The user application display device 6410 may be connected to a user application (e.g., user application 3825) that may be installed on the user device.

[0370] The generated inferences 6408 may comprise one or more of the following: ● Cell or nucleus detection / segmentation / classification; ● Gland / tissue / tumor segmentation, ● Cancer detection / classification / grading; ● Survival / outcome prediction / prognosis; ● Staining normalization / transfer, ● Genomic / molecular prediction.

[0371] Depending on the application, as shown in Figure 65, the deep learning model 6406 can be based on a deep neural network model using one or more of the following types of learning: supervised learning (e.g., classification models, regression models, segmentation models) such as a convolutional neural network (CNN) or a recurrent neural network (RNN), weakly supervised learning (multiple instance learning models, other weakly supervised models), unsupervised learning, and transfer learning deep neural networks (pre-trained models, domain adaptation models) having one or more of the following architectures (and modified versions thereof): CNN, RNN, fully convolutional network (FCN), automatic decoder (AD), generative adversarial network (GAN), or pre-trained network (PRE-TN). The photon absorption remote sensing PARS machine learning model described herein can be used to generate one or more inferences comprising:

[0372] ● Cell / nucleus detection / segmentation / classification, ● Gland / tissue / tumor segmentation, ● Cancer detection / classification / grading; ● Prediction / prognosis of survival; ● Other inferences.

[0373] For example, unsupervised learning classification such as GAN and AD can be used as follows: ● Cell / nucleus detection / segmentation / classification, ● Gland / tissue / tumor segmentation, ● Cancer detection / classification / grading.

[0374] Supervised learning classification such as CNN and RNN can be used as follows: ● Cell / nucleus detection / segmentation / classification, ● Cancer detection / classification / grading.

[0375] Weakly supervised learning classification (weakly supervised CNN, RNN) can be used as follows: ● Cell / nucleus detection / segmentation / classification, ● Gland / tissue / tumor segmentation, ● Cancer detection / classification / grading; ● Prediction / prognosis of survival / outcome.

[0376] Transfer learning (CNN, GAN, PRE-TN) can be used to: ● Cell / nucleus detection / segmentation / classification, ● Gland / tissue / tumor segmentation, ● Cancer detection / classification / grading; ● Prediction / prognosis of survival; ● Stain normalization / metastasis-genomic / molecular prediction.

[0377] (Automated nuclei detection, segmentation, and classification of photon absorption remote sensing PARS data) According to one aspect, a computer-implemented machine learning architecture for automated nucleus detection, segmentation, and classification of photon absorption remote sensing PARS data is disclosed herein. As shown in FIG. 66, a deep learning model 6406 may receive a plurality of photon absorption remote sensing PARS signals and photon absorption remote sensing PARS data from a photon absorption remote sensing PARS system 6402, where the photon absorption remote sensing PARS signals may comprise radioactive and non-radioactive signals, and the photon absorption remote sensing PARS data may comprise a plurality of extracted features based on processing at least one of the plurality of signals, where the features provide information of the contrast provided by at least one of the plurality of signals.

[0378] The deep learning model 6406 may comprise one or more of a classification deep neural network 6610, a segmentation deep neural network 6620, and a nucleus detection deep neural network 6630. The deep learning model 6406 may comprise, for example, a densely connected neural network (DCNN) based on U-Net, a densely connected recurrent convolutional neural network (DCRN), and a recurrent residual convolutional neural network (RRCNN). The output of the deep learning model 6406 may comprise nucleus type, segmentation, and detection masks, which may be sent to the user application display device 6410 for further processing and display.

[0379] In some embodiments, deep learning model 6406 may receive a set of multi-structured input data that may comprise, for example, photon-absorbing remote sensing PARS images, photon-absorbing remote sensing PARS features, and photon-absorbing remote sensing PARS image features. Some or all of the multi-structured input data may comprise photon-absorbing remote sensing PARS data and / or features 6404 from photon-absorbing remote sensing PARS system 6402. Due to the nature of the deep data represented by photon-absorbing remote sensing PARS signals from photon-absorbing remote sensing PARS system 6402, principal component analysis (PCA) may be applied for dimensionality reduction to obtain the most relevant feature representation. In addition to cross-entropy and mean squared error (MSE) losses, the deep learning model 6406 can be implemented and trained using other loss calculation methods. For example, the deep learning model 6406 can be trained using a modified structural similarity index (SSIM) based on overlapping Gaussian sliding windows taking tiled image patches, and an earth mover (EM) to account for structured representations. The output of the deep learning model 6406 can include, for example, nuclei type, segmentation, and detection masks, which can be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei. Nuclei types include, for example, epithelial, fibroblast, inflammatory, and others. Past reference cases (images, diagnoses) can be provided to closely match a given case based on a computer-implemented content-based image retrieval (CBIR) system, such as the CBIR system 7800 (see, for example, FIG. 78).

[0380] (Automatic nuclear segmentation of photon absorption remote sensing PARS data) In some embodiments, a computer-implemented machine learning architecture is disclosed herein for automatic nucleus segmentation, as shown in FIG. 67. A nucleus segmentation region-based CNN 6710, which may be an example of a deep learning model 6406, is enabled to receive as input a plurality of photon absorption remote sensing PARS signals, features, and images from a photon absorption remote sensing PARS system 6402. The photon absorption remote sensing PARS signals may comprise radiative and non-radiative signals. The photon absorption remote sensing PARS features may comprise a plurality of extracted features based on processing at least one of the plurality of photon absorption remote sensing PARS signals, the features providing contrast information provided by at least one of the plurality of signals. The nucleus segmentation region-based CNN 6710 may comprise a backbone network (e.g., a region proposal network (RPN)) 6712, a feature map generator 6715, and a mask module 6717. The backbone network 6712 may be implemented to find areas that may contain objects. The nucleus segmentation region-based CNN 6710 may predict the class of the proposed area and refine the bounding box of the proposed area, and the mask module 6717 may be used to generate a pixel-level object mask in the next stage based on the proposed area. The output of the nucleus segmentation region-based CNN 6710 may be an image with segmented nuclei, which may be sent to the user application display device 6410 for further processing and / or display.

[0381] In some embodiments, principal component analysis (PCA) may be used for dimensionality reduction to obtain the most relevant feature representation. Furthermore, alternative methods of loss function calculation may be used, as described above. Due to the nature of the deep data represented by the photon absorption remote sensing PARS signal from the photon absorption remote sensing PARS system 6402, principal component analysis (PCA) may be applied for dimensionality reduction to obtain the most relevant feature representation. In addition to cross-entropy and mean squared error (MSE) losses, the nucleus segmentation region-based CNN 6710 may be implemented and trained using other loss calculation methods; for example, the nucleus segmentation region-based CNN 6710 may be trained using a modified structural similarity index (SSIM) based on overlapping Gaussian sliding windows that take tiled image patches, as well as an earth mover (EM) to take structured representations into account. The output of the nuclear segmentation region-based CNN 6710 may comprise, for example, nuclear type, segmentation, and detection masks, which may be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei, such as epithelial, fibroblast, inflammatory, and others.

[0382] Previous reference cases (images, diagnoses) can be provided to closely match a given case based on a computer-implemented content-based image retrieval (CBIR) system such as CBIR system 7800 (see, e.g., FIG. 78). The output from system 6406 of FIG. 66 can be combined with the output of nuclear segmentation region-based CNN 6710 of FIG. 67 for validation.

[0383] (Classification of tissue malignant tumors (Photon absorption remote sensing PARS images)) In some embodiments, a computer-implemented machine learning architecture is disclosed herein for identifying malignant tumors in tissue. As shown in Figures 68 and 69, a deep learning model 6406 (e.g., a modified convolutional neural network (CNN) model) can be configured to receive local photon absorption remote sensing PARS image features acquired by the image transformation sub-module 6403 by using techniques such as, but not limited to, Contourlet transform (CT) (edge ​​smoothness), histogram (pixel intensity distribution), discrete Fourier transform (DFT) (feature selection using frequency domain information from the image), and local binary pattern (LBP) (texture information). The deep learning model 6406 receives as input the photon absorption remote sensing PARS signals (radioactive, non-radioactive, scatter signals) and photon absorption remote sensing PARS images from the photon absorption remote sensing PARS system 6402, as well as the acquired local photon absorption remote sensing PARS image features from the image transformation sub-module 6403, and may generate one or more inferences that may comprise a classification of tissue malignancy (e.g., benign, malignant, no pathology). The generated inferences may be transmitted to the user application display device 6410 for further processing and / or display.

[0384] (Classification of tissue malignancies (Photon absorption remote sensing PARS signal features)) In some embodiments, a computer-implemented machine learning architecture is disclosed herein for identifying malignant tumors in tissue. As shown in FIG. 70A , a deep learning model 6406 (e.g., a modified CNN) may receive as input a plurality of photon absorption remote sensing PARS signals and features 7015 from a photon absorption remote sensing PARS system 6402. The photon absorption remote sensing PARS signals may comprise radiative and non-radiative signals. The photon absorption remote sensing PARS features may comprise a plurality of extracted features based on processing at least one of the plurality of photon absorption remote sensing PARS signals, where the features provide information of the contrast provided by at least one of the plurality of signals. The deep learning model 6406 may be configured to receive the local photon absorption remote sensing PARS image features acquired by the image transformation sub-module 6403 by using techniques such as, but not limited to, Contourlet transform (CT) (edge ​​smoothness), histogram (pixel intensity distribution), discrete Fourier transform (DFT) (feature selection using frequency domain information from the image), and local binary pattern (LBP) (texture information).

[0385] The deep learning model 6406 receives as input the multi-channel photon absorption remote sensing PARS signals (radiative, non-radiative, scatter signals) and photon absorption remote sensing PARS signals and features 7015 from the photon absorption remote sensing PARS system 6402, as well as the acquired local photon absorption remote sensing PARS image features from the image transformation sub-module 6403, and may generate one or more inferences that may comprise a classification of tissue malignancy (e.g., benign, malignant, no pathology). The generated inferences may be transmitted to the user application display device 6410 for further processing and / or display.

[0386] In some embodiments, the output of the deep learning model 6406 may comprise, for example, nuclei type, segmentation, and detection masks, which may be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei. Nuclei types may include, for example, epithelial, fibroblast, inflammatory, and others. Previous reference cases (images, diagnoses) may be provided to closely match a given case based on a computer-implemented content-based image retrieval (CBIR) system, such as the CBIR system 7800 (see, e.g., FIG. 78).

[0387] Classification of tissue malignancies (simulated stained photon absorption remote sensing PARS images) In some embodiments, another computer-implemented machine learning architecture is disclosed herein for identifying malignant tumors in tissue. As shown in Figure 70B, a deep learning model 6406 (e.g., a modified CNN) can receive simulated stained photon absorption remote sensing PARS image(s) from an image generator 7010 (similar to image generators 3812, 3912), which can generate simulated stained photon absorption remote sensing PARS image(s) based on photon absorption remote sensing PARS signals and features from the photon absorption remote sensing PARS system 6402. The deep learning model 6406 may also be configured to receive local photon absorption remote sensing PARS image features acquired by the image transformation sub-module 6403 by using techniques such as, but not limited to, Contourlet transform (CT) (edge ​​smoothness), histogram (pixel intensity distribution), discrete Fourier transform (DFT) (feature selection using frequency domain information from the image), and local binary pattern (LBP) (texture information).

[0388] The deep learning model 6406 receives as input the simulated stained photon absorption remote sensing PARS image(s) and local photon absorption remote sensing PARS image features obtained from the image transformation sub-module 6403, and may generate one or more inferences that may comprise a classification of tissue malignancy (e.g., benign, malignant, no pathology). The generated inferences may be transmitted to the user application display device 6410 for further processing and / or display.

[0389] In some embodiments, the output of the deep learning model 6406 may comprise, for example, nuclei type, segmentation, and detection masks, which may be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei. Nuclei types may include, for example, epithelial, fibroblast, inflammatory, and others. Previous reference cases (images, diagnoses) may be provided to closely match a given case based on a computer-implemented content-based image retrieval (CBIR) system, such as the CBIR system 7800 (see, e.g., FIG. 78).

[0390] (PARS multidimensional photon absorption remote sensing input data for medical applications) As shown in FIG. 70C , in some embodiments, deep learning model 6406 may receive a set of multidimensional input data. The set of multidimensional input data may be a set of multi-structured input data that may comprise, for example, two or more of: photon absorption remote sensing PARS image 7020, photon absorption remote sensing PARS signal and features 7015, photon absorption remote sensing PARS image features 7023, and simulated stain image 7025 generated from selected photon absorption remote sensing PARS features 7021. Some or all of the multi-structured input data may comprise photon absorption remote sensing PARS data and / or features from photon absorption remote sensing PARS system 6402. Due to the nature of the deep data represented by photon absorption remote sensing PARS signal data from photon absorption remote sensing PARS system 6402, principal component analysis (PCA) may be applied for dimensionality reduction to obtain the most relevant feature representation.

[0391] For example, pixel information (e.g., pixel intensity) in a photon absorption remote sensing PARS image 7020 may be combined with information contained in photon absorption remote sensing PARS signal features 7015 to form multidimensional (deep data) input to the deep learning model 6406. Examples of photon absorption remote sensing PARS signal features 7015 include data values ​​representing mechanical properties (e.g., stiffness, speed of sound, pelvis isolation, thermal conductivity) and chemical properties (e.g., quantum efficiency ratio QER, total absorption, bonding state, viscosity, ion concentration, charge, chemical composition).

[0392] By combining image pixel information and spatial information obtained from the photon absorption remote sensing PARS image 7020 (and optionally the photon absorption remote sensing PARS image features 7023), as well as mechanical, physical, and chemical feature data from the photon absorption remote sensing PARS signal features 7015, results of increased complexity can be generated using input data sent to the deep learning model 6406. The output of the deep learning model 6406 can comprise a tissue malignancy class, a malignancy rating, a cancer prognosis, and a treatment prognosis. The generated inferences can be sent to the user application display 6410.

[0393] In addition to cross-entropy and mean squared error (MSE) loss, the deep learning model 6406 may be implemented and trained using other loss calculation methods, for example, the deep learning model 6406 may be trained using a modified structural similarity index (SSIM) based on overlapping Gaussian sliding windows that take tiled image patches, as well as an earth mover (EM) to take structured representations into account.

[0394] The output of the deep learning model 6406 may comprise, for example, nuclei type, segmentation, and detection masks, which may be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei. Nuclei types may include, for example, epithelial, fibroblast, inflammatory, and others. Past reference cases (images, diagnoses) may be provided for closely matching given cases based on a computer-implemented content-based image retrieval (CBIR) system, such as the CBIR system 7800 (see, e.g., FIG. 78).

[0395] Multi-stain graph fusion for multimodal integration in pathology to predict cancer grading In some embodiments, disclosed herein is a computer-implemented machine learning architecture for predicting a pathology score by performing multi-stain graph fusion for multi-modal integration of a simulated stained photon absorption remote sensing PARS image and multiple unregistered stained histopathology images, as shown in Figure 71. This multi-modal deep learning graph fusion process may use information from the simulated stained photon absorption remote sensing PARS image and multiple unregistered histopathology images 7110 to predict a pathology score.

[0396] The simulated stained photon absorption remote sensing PARS image may be obtained from an image generator 7010 (similar to image generators 3812, 3912), which may generate the simulated stained photon absorption remote sensing PARS image(s) based on the photon absorption remote sensing PARS signals and features from the photon absorption remote sensing PARS system 6402.

[0397] In some embodiments, the deep learning model 6406 may receive a set of multidimensional input data. The set of multidimensional input data may be a set of multi-structured input data that may comprise, for example, two or more of the photon absorption remote sensing PARS image 7020, the photon absorption remote sensing PARS signal and features 7015, the previously unregistered histology image 7110, and the simulated stained image 7025 generated from selected photon absorption remote sensing PARS features. A portion of the multi-structured input data may comprise photon absorption remote sensing PARS data and / or features from the photon absorption remote sensing PARS system 6402.

[0398] The deep learning model 6406 may be implemented to perform pixel-level classification of various stains. The output of the deep learning model 6406 may comprise a heat map 7130 that is used to generate a graph by the graph generator 7130. The graph neural network model 7150 is trained on a plurality of input image graphs. The output of the trained graph neural network model 7150 is enabled to generate inferences representing a tissue malignancy rating and / or probability score. The generated inferences may be transmitted to the user application display device 6410 for further processing and / or display.

[0399] In some embodiments, the output of the deep learning model 6406 may comprise, for example, nuclei type, segmentation, and detection masks, which may be combined into a multi-layered image display containing segmented and / or detected nuclei with annotated types of nuclei. Nuclei types may include, for example, epithelial, fibroblast, inflammatory, and others. Previous reference cases (images, diagnoses) may be provided to closely match a given case based on a computer-implemented content-based image retrieval (CBIR) system, such as the CBIR system 7800 (see, e.g., FIG. 78).

[0400] (Survival analysis - Integrating photon absorption remote sensing (PARS) image data with genomic data) In some embodiments, the computer-implemented machine learning architecture 7200 can be configured to combine photon absorption remote sensing PARS image data with genomic data to obtain an integrated prediction of patient survival. Referring now to FIG. 72, the genomic data 7250 can comprise any signal derived from analysis of DNA or RNA comprising epigenetic features, or mRNA derived from any sequencing or other nuclear analysis technique. The genomic data 7250 can be derived from germline analysis, bulk tumor analysis, single-cell analysis, analysis of malignant cells or subsets thereof, or analysis of benign cells or subsets thereof comprising benign stromal elements. The machine learning architecture 7200 has three parts: photon absorption remote sensing PARS image cluster processing, genomic data processing (e.g., without limitation, mRNA-seq analysis by WGCNA), and multimodality survival analysis.

[0401] For photon-absorbing remote sensing PARS image clustering, photon-absorbing remote sensing PARS images and photon-absorbing remote sensing PARS image features from the photon-absorbing remote sensing PARS system 6402 may be received as input by a patch clustering process 7210, in which patches are clustered into n categories, followed by a patch expansion process 7220 (horizontal flip, vertical flip, and rotation). Each category of patch may serve as input to a deep neural network 7230 (e.g., a multi-instance fully convolutional network (Ml-FCN) composed of multiple sub-networks having the same structure and shared weight parameters). The output from the deep neural network 7230 may undergo attention aggregation to obtain a deep learning risk score for a given patient.

[0402] A deep neural network 7230 may be implemented to receive acquired photon absorption remote sensing PARS image features by using techniques such as Contourlet Transform (CT) (edge ​​smoothness), histogram (pixel intensity distribution), Discrete Fourier Transform (DFT) (feature selection using frequency domain information from the image), and Local Binary Pattern (LBP) (texture information). The output may undergo attention aggregation to obtain a deep learning risk score for the patient.

[0403] From the perspective of genome data processing (not limited to mRNA-seq processing in this case), eigengenes can be obtained by weighted gene co-expression network analysis (WGCNA) 7260. Then, modules can be selected based on the eigengenes by least absolute shrinkage and selection operator (LASSO) processing 7270. After that, the top hub genes of the retained modules can be extracted as risk factors.

[0404] The deep learning risk score and hub genes may then be integrated using a Cox proportional hazards model 7280. Based on the deep learning risk score from the histopathological image, the module hub genes from the genetic data, and clinical characteristics (e.g., age and gender) 7240, a multi-input integrated prognostic machine learning model based on the Cox proportional hazards model 7280 is implemented and trained. The Cox proportional hazards model 7280 is a regression model for examining the association between a patient's survival time and one or more predictor variables. The integrated model estimates survival risk and is enabled to calculate a comprehensive risk score that enables patients to be classified into low-risk or high-risk groups in a survival analysis module 7290.

[0405] (Survival analysis - Integrating photon absorption remote sensing (PARS) signals (extracted features) with genomic data) In some embodiments, the computer-implemented machine learning architecture 7300 can be configured to combine photon absorption remote sensing (PARS) image data with genomic data to obtain an integrated prediction of patient survival. Referring now to FIG. 73, the genomic data 7250 can comprise any signal derived from analysis of DNA or RNA comprising epigenetic features, or mRNA derived from any sequencing or other nuclear analysis technique. The genomic data 7250 can be derived from germline analysis, bulk tumor analysis, single-cell analysis, analysis of malignant cells or subsets thereof, or analysis of benign cells or subsets thereof comprising benign stromal elements. The machine learning architecture 7300 has three parts: photon absorption remote sensing (PARS) data processing, genomic data processing (e.g., without limitation, mRNA-seq analysis by WGCNA), and multimodality survival analysis.

[0406] In terms of photon absorption remote sensing PARS data processing, photon absorption remote sensing PARS features from photon absorption remote sensing PARS system 6402 may be received as input by deep neural network 6406, and output from deep neural network 6406 may undergo attention aggregation to obtain a deep learning risk score for a given patient.

[0407] From the perspective of genome data processing (not limited to mRNA-seq processing in this case), eigengenes can be obtained by weighted gene co-expression network analysis (WGCNA) 7260. Then, modules can be selected based on the eigengenes by least absolute shrinkage and selection operator (LASSO) processing 7270. After that, the top hub genes of the retained modules can be extracted as risk factors.

[0408] The deep learning risk score and hub genes may then be integrated using a Cox proportional hazards model 7280. Based on the deep learning risk score from the deep neural network 6406, the module hub genes from the genetic data, and clinical characteristics (e.g., age and sex) 7240, a multi-input integrated prognostic machine learning model based on the Cox proportional hazards model 7280 is implemented and trained. The Cox proportional hazards model 7280 may be a regression model for examining the association between patient survival time and one or more predictor variables. The integrated model estimates survival risk and is enabled to calculate a comprehensive risk score that is enabled to classify patients into low-risk or high-risk groups in a survival analysis module 7290.

[0409] (Survival analysis based on photon absorption remote sensing PARS image data, previously unaligned tissue structure images, and genomic data) In some embodiments, as shown in FIG. 73 , in addition to photon absorption remote sensing PARS signals and features from a photon absorption remote sensing PARS system 6402, previously unregistered histology images 7110 may also be used as inputs to a deep neural network 6406. In some embodiments, genomic data 7250 may be used as inputs to a deep neural network 6406 to calculate a risk score. The risk score and genomic data 7250 may be integrated using a Cox proportional hazards model 7280, which is a regression model for examining the association between patient survival time and one or more predictor variables. The integrated model estimates survival risk and is enabled to calculate a comprehensive risk score that is enabled to classify patients into low-risk or high-risk groups in a survival analysis module 7290.

[0410] (Multimodal fusion with photon absorption remote sensing PARS images) In some embodiments, depending on the application, the computer-implemented system is implemented to perform image fusion of photon absorption remote sensing PARS image data 7410 with images 7405 from other modalities (e.g., computed tomography (CT), magnetic resonance imaging (MRI)) based on image feature representations and similarity measures, as shown in FIG. 74.

[0411] The system of embodiments may perform registration of photon absorption remote sensing PARS image data with other imaging modalities such as CT and MRI, and dimensional complexity may include image to volume (2D to 3D), image to image (2D to 2D), and volume to volume (3D to 3D).

[0412] An exemplary process 7400 performed by the system of an embodiment is illustrated in Figure 74. Multimodal image fusion can be achieved by an image registration technique that implements an iterative optimization algorithm. At each iteration, a better alignment is achieved based on a pre-defined similarity measure 7450 that calculates the amount of correspondence between the input images.

[0413] The optimization algorithm calculates and may update new transformation / interpolation parameters. The operation continues until an optimal registration is achieved or some predefined criteria is met. The output of the system can be either the transformation parameters 7480 or a final interpolated fused image 7470. The exemplary process 7400 enables the measurement of 3D features and allows spatial localization of histological findings within the local 3D tissue environment. Combining photon absorption remote sensing (PARS) image data with high-resolution 2D / 3D imaging techniques such as micro-CT and micro-MRI (pre-sectioning) provides access to morphological features, can relate histological findings to the 2D / 3D structure of the local tissue environment, and allows for guided sectioning of the tissue.

[0414] (Multimodal fusion of photon absorption remote sensing (PARS) signals / images (deep training)) In some embodiments, multimodal image fusion may be performed by a computer-implemented system configured to execute the exemplary process 7500 shown in FIG. 75. The system performs registration of photon absorption remote sensing (PARS) image data 7510 with images 7505 from other imaging modalities, such as CT and MRI, via a similarity metric 7520 and a deep learning registration model 7530. Dimensional complexity may include image-to-volume (2D to 3D), image-to-image (2D to 2D), and volume-to-volume (3D to 3D). The system output may be a final 2D or 3D fused image, which may be sent to a user application display device 6410 for display. This fusion technique enables measurement of 3D features, spatially localizes the histology of tissue structures within the local 2D / 3D tissue environment, and enables guided sectioning of tissue.

[0415] (Customized Photon Absorption Remote Sensing (PARS) image staining based on user input) According to yet another aspect, customized stain deep learning model 7606 may be implemented as part of machine learning architecture 7600 to perform customized photon absorption remote sensing (PARS) image staining, which may allow a user to control different aspects such as the total number and nature of different stain images displayed on user application display device 6410. The user may mix, modify, and combine stains in real time based on one or more specified criteria, for example, through user input 7610 received by user application display device 6410.

[0416] The customized staining deep learning model 7606 may receive as input one or more photon absorption remote sensing PARS images and photon absorption remote sensing PARS features from the photon absorption remote sensing PARS system 6401 and user input 7610. The user input 7610 may comprise user-defined criteria for generating one or more stained images. The customized staining deep learning model 7606 may generate one or more custom stained images for display on the user application display device 6410 based on the user criteria. Some exemplary user criteria for custom staining that may be included in the user input 7610 may include the following: ● Any nuclear cells smaller / larger than some specified value; • Any nucleus with atypical features; • Any nucleus with prominent nucleoli; • Any nucleus showing mitotic figures; • Any cell membrane that has certain characteristics; • Any cell that has lymphocyte characteristics; • Any cell undergoing apoptosis; • Any cell undergoing cell division; • Any cell adjacent to lymphocytes, • Any smooth muscle cell, • Any cell adjacent to the basement membrane, ● Any connective tissue, ● any non-cellular material; • connective tissues with specific characteristics; ● Any fat cell.

[0417] The customized staining deep learning model 7606 may include a feature extraction mechanism that may determine size, shape, characteristics (e.g., density), and structure to customize the color map.

[0418] A general photon absorption remote sensing (PARS_Al) model for classification, grading, and fusion staining of photon absorption remote sensing (PARS) image data. In some embodiments, the computer-implemented machine learning architecture 7700 is implemented to perform tissue malignancy (cancer) detection, classification, and grading. The machine learning architecture 7700 may include a stain fusion deep learning model 7720 for generating a simulated stained / fused photon absorption remote sensing PARS image 7730 associated with predicted model results for display on the user application display device 6410. The stain fusion deep learning model 7720 may receive multiple simulated stained images (stain 1, stain 2, ... stain n) from the image generator 7710, which is enabled to generate the multiple simulated stained images based on photon absorption remote sensing PARS features from the photon absorption remote sensing PARS system 6402 to generate the associated simulated stained / fused photon absorption remote sensing PARS image 7730.

[0419] The machine learning architecture 7700 includes a diagnostic deep learning model 7706, which may include two deep neural network models for performing classification and grading, respectively, based on photon absorption remote sensing PARS features and images from the photon absorption remote sensing PARS system 6402. The output of the diagnostic deep learning model 7706 and the output of the image generator 7710 (n simulated stained images) may serve as input to a stain fusion deep learning model 7720 to generate an associated simulated stained photon absorption remote sensing PARS image 7730.

[0420] The output 7750 of the diagnostic deep learning model for classification, grading, and staining of photon absorption remote sensing PARS image data 7706 may comprise a predicted class and rating, such as a rating of malignant tumor, and the output of the stain fusion deep learning model 7720 is a simulated stained photon absorption remote sensing PARS image 7730. The two outputs 7730, 7750 may be sent to the user application display device 6410 for further processing (if any) and display.

[0421] (Content-based image retrieval (CBIR) system for assisting pathologists in diagnosis) In some embodiments, a computer-implemented content-based image retrieval (CBIR) system 7800 is implemented to assist a pathologist in making a diagnosis. The system 7800 queries one or more images, which may be, for example, a photon absorption remote sensing PARS image 7802, a simulated stained photon absorption remote sensing PARS image 7805, or a tissue image, and may be configured to acquire similar images 7850 from an image repository 7810 based on the queried images 7802, 7805.

[0422] In some embodiments, images retrieved from the repository 7810 are processed by an image feature extraction module 7820 to obtain semantically significant features (feature vectors), which are then indexed (represented by index features 7840) based on their pairwise differences calculated by a distance measurement 7830. Query images 7802, 7805 are processed by the same feature extraction module 7820 to generate a query image feature vector 7825. The query image feature vector 7825 is then compared by a distance measurement module 7835 with the indexed features 7840 retrieved based on images from the image repository 7810. A final output 7850 is obtained by selecting one or more images from the image repository 7810 that are closest to the queried images 7802, 7805 based on the calculated distances generated by the distance measurement module 7835. For example, if the calculated distance D between the repository image and the queried image is less than a certain threshold, the image from the image repository 7810 is deemed sufficiently similar to the queried image 7802, 7805 to be included as part of the final output 7850 that can be used in a diagnostic report.

[0423] (Interpretable Al_Photon Absorption Remote Sensing PARS Architecture) In some embodiments, a computer-implemented architecture 7900 may be implemented to generate meaningful information for explaining one or more photon absorption remote sensing PARS images from a photon absorption remote sensing PARS system 6402, as shown in FIG. 79. The architecture 7900 may include an explainable Al_photon absorption remote sensing PARS module 7950, which may include multiple different modules. The Al_photon absorption remote sensing PARS module 7950 may receive as input one or more photon absorption remote sensing PARS images from the photon absorption remote sensing PARS system 6402, a tissue structure image 7920, and a user query 7930 to perform diagnostic analysis to assist a pathologist in making a diagnosis. The Al_photon absorption remote sensing PARS module 7950 may include one or more modules and machine learning models representing a class of explanation generation methods, such as a deep learning diagnostic module, a deep learning saliency map generator, a concept attribution generator, a prototype generator, a counterfactual generator, a confidence score generator, and a user query interpreter.

[0424] Within the Al_Photon Absorption Remote Sensing PARS module 7950, the deep learning diagnostic module is enabled to generate diagnostic predictions based on photon absorption remote sensing PARS images. Global and local saliency maps from the deep learning saliency map generator are enabled to explain model predictions by providing visualizations. The concept attribution generator is enabled to provide explanations of model predictions using synthetically generated visualizations and / or superordinate concepts equipped with domain-relevant natural language. The prototype generator is enabled to generate explanations of the inner workings of the model. These explanations are provided through real or synthetically generated examples, such as prototypical instances of particular categories or features. The counterfactual generator is enabled to generate counterfactuals used to explain model results by presenting the outcomes of other possible scenarios that lead to different outcomes. Counterfactual examples are synthetically generated visualizations or real data. The confidence score generator is enabled to generate confidence scores or measures indicating the reliability of the model's predictions and results. The user query interpreter analyzes the user's input query (e.g., for a given patient, visualize a specific part of tissue, a specific substructure, an indication of a specific type of cancer, the number of nuclei of a specific size, etc.).

[0425] The output of the Al_Photon Absorption Remote Sensing PARS module 7950 is a generated report in the form of a collection of images, quantitative measures, presentation of similar cases, and domain-relevant natural language, which can be sent to the User Application Display Device 6410 for display to the user.

[0426] In some embodiments, one or more simulated stained images from the image generator 7910 may be used as input to an Al_photon absorption remote sensing PARS module 7950 to perform diagnostic analysis to assist a pathologist in making a diagnosis. The one or more simulated stained images may also be sent to a user application display device 6410 for display to a user along with output from the Al_photon absorption remote sensing PARS module 7950.

[0427] (Example Uses) The embodiments disclosed herein may include non-radioactive (heat and pressure) and radiative (fluorescence is one possible signal) signals in a sample. The embodiments disclosed herein may also include collecting radiative and non-radiative relaxation due to optical absorption, as well as scattering from both excitation and detection. Collected signals and / or raw data may be used to directly generate and color images of a sample, such as H&E (hematoxylin and eosin) histology images, without staining the sample. H&E histology images may be directly generated and colored using methods disclosed herein (e.g., based on comparison of non-radioactive and radiative signals, quantum efficiency ratio (QER), signal lifetime or evolution, and / or clustering algorithms) and by using features of raw photon absorption remote sensing (PARS) signals. The embodiments disclosed herein may be used to determine or measure mechanical properties, such as the speed of sound and / or temperature properties, of a sample using a photon absorption remote sensing system or photon absorption remote sensing (PARS). These features or characteristics can be measured using a small or pinpointed area of ​​a sample (e.g., the size of a focused laser or light beam). The embodiments disclosed herein can extract more than the amplitude or scalar amplitude of a signal in a sample. For example, two targets may have the same or similar optical absorption, but may have slightly different other properties, such as different speeds of sound, which can result in different evolution and / or shape of the signal. The embodiments disclosed herein can be used to determine or add new molecular information to photon absorption remote sensing (PARS) images.

[0428] It will be apparent that other examples can be designed with different fiber-based or free-space components to achieve similar results. Other alternatives may include sources of various coherence lengths, the use of balanced photodetectors, interrogation beam modulation, the incorporation of optical amplifiers in the return signal path, etc.

[0429] During in vivo imaging experiments, no drugs or ultrasound coupling media are required. However, the target can be prepared with any liquid, such as water or oil, prior to the non-contact imaging session. Similarly, in some instances, an intermediate window, such as a coverslip or glass window, can be placed between the imaging system and the sample.

[0430] The embodiments disclosed herein may use a combination of a photon absorption remote sensing (PARS) device alongside optical coherence tomography (OCT). Optical coherence tomography (OCT) is a complementary imaging modality to photon absorption remote sensing (PARS) devices. Optical coherence tomography (OCT) measurements can be performed using various approaches, either time-domain optical coherence tomography (TD-OCT) or frequency-domain optical coherence tomography (FD-OCT), as described in US2010 / 0265511 and US2014 / 0125952. In optical coherence tomography (OCT) systems, multiple A-scans are typically acquired while a sample beam is scanned laterally across the tissue surface, constructing a two-dimensional map of reflectivity over depth and lateral range, typically referred to as a B-scan. The lateral resolution of the B-scan is approximated by the confocal resolution of the sample arm optics, which is typically given by the size of the focused optical spot within the tissue.

[0431] All light sources, including but not limited to photon absorption remote sensing (PARS) excitation, photon absorption remote sensing (PARS) detection, photon absorption remote sensing (PARS) signal enhancement, and optical coherence tomography (OCT) sources, can be implemented as continuous beams, modulated continuous beams, or short pulsed lasers with pulse widths ranging from attoseconds to milliseconds. They can be set to any wavelength suitable for exploiting the sample's optical (or other electromagnetic) properties, such as scattering and absorption. Wavelengths can also be selected to intentionally enhance or suppress detection or excitation photons from different absorbers. Wavelengths can range from nanometers to microns. Continuous wave beam power can be set to any suitable power range, from attowatts to watts. Pulsed light sources can use pulse energies appropriate for the particular sample under test, such as in the attojoule to joule range. Various coherence lengths can be implemented to exploit interference effects. These coherence lengths can range from nanometers to kilometers. Similarly, pulsed light sources can use any repetition rate deemed appropriate for the sample under test, from continuous wave to gigahertz regimes. The light source may be adjustable, monochromatic or polychromatic.

[0432] The total absorption photon absorption remote sensing (TA-PARS), multi-path photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time domain photon absorption remote sensing (TD-PARS) subsystems may include interferometers such as Michelson, Fizeau, Ramsey, Fabry-Perot, Mach-Zehnder, or optical quadrature detection. The interferometers may be free-space, fiber-based, or some combination. The basic principle is that phase and amplitude oscillations of the probe-receiver beam can be detected interferometrically, enabled by a variety of detectors at AC, RF, or ultrasonic frequencies.

[0433] The total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time domain-photon absorption remote sensing (TD-PARS) subsystems may detect amplitude modulation in a signal using and implementing non-interferometric detection designs. Non-interferometric detection systems may be free-space or fiber-based, or some combination thereof.

[0434] The total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time domain-photon absorption remote sensing (TD-PARS) subsystems may use a variety of optical fibers, such as photonic crystal fiber, image guide fiber, and double-clad fiber.

[0435] The photon absorption remote sensing PARS subsystem can be implemented as a conventional photoacoustic remote sensing system, non-interferometric photoacoustic remote sensing (NI-PARS), camera-based photoacoustic remote sensing (C-PARS), coherence-gated photoacoustic remote sensing (CG-PARS), single-source photoacoustic remote sensing (SS-PARS), or extensions thereof.

[0436] In one example, all beams can be combined and scanned. In this way, PARS excitations can be detected in the same area where they are generated and where they are maximal. Optical coherence tomography (OCT) detection can also be performed in the same location as PARS to aid in alignment. Other arrangements can be used, including keeping one or more of the beams fixed while scanning the other beams, or vice versa. Optical scanning can be performed by galvanometer mirrors, MEMS mirrors, polygon scanners, stepper / DC motors, etc. Mechanical scanning of the sample can be performed by stepper stages, DC motor stages, linear drive stages, piezo drive stages, piezo stages, etc.

[0437] Both optical and mechanical scanning approaches can be utilized to generate one-, two-, or three-dimensional scans of a sample. Adaptive optics, such as TAG lenses and deformable mirrors, can be used to perform axial scans within a sample. Both optical and mechanical scanning can be combined to form a hybrid scanner. This hybrid scanner can use one or two optical axes to capture large areas or strips in a short time. The mirrors can potentially be controlled with custom control hardware to have customized scan patterns, improving scanning efficiency in terms of speed and quality. For example, one optical axis can be used to rapidly scan while one mechanical axis simultaneously moves the sample. This can render a ramp-like scan pattern that can then be interpolated. Another example is using custom control hardware to step the mechanical stage only when the fast axis has finished moving, producing a Cartesian-like grid that may not require interpolation.

[0438] Photon absorption remote sensing (PARS) can provide 3D imaging by optical or mechanical scanning of the beam, or mechanical scanning of the sample or imaging head, or a combination of mechanical and optical scanning of the beam, optics, and sample, which can enable rapid structural and functional en face or 3D imaging.

[0439] One or more pinholes can be used to filter out optical or mechanical scanning of the beam, or mechanical scanning of the sample or imaging head, or a combination of mechanical and optical scanning of the beam, optics, and sample, which can improve the signal-to-noise ratio of the resulting image.

[0440] The beam combiner may be implemented using dichroic mirrors, prisms, beam splitters, polarizing beam splitters, WDMs, and the like. Beam paths may focus on the sample using different optical paths. Single or multiple photon absorption remote sensing (PARS) excitation, detection, signal enhancement, etc. paths and optical coherence tomography (OCT) paths may each use independent focusing elements on the sample, or all may share a single (only one or exactly one) path or any combination. Beam paths may return from the sample using unique optical paths that are different from those used to focus on the sample. These unique optical paths may interact with the sample at normal incidence or at an angle where the central beam axis forms an angle with the sample surface ranging from 5 degrees to 90 degrees.

[0441] In some applications, such as ophthalmic imaging, the imaging head may not implement any primary focusing elements, such as an objective lens, to tightly focus the light onto the sample. Instead, the beam may be collimated or loosely focused (to create a spot size much larger than the optical diffraction limit) while being directed at the sample. For example, ophthalmic imaging devices direct a collimated beam into the eye, allowing the eye's lens to focus the beam onto the retina.

[0442] The imaging head can focus the beam into the sample to a depth of at least 50 nm. The imaging head can focus the beam into the sample to a depth of up to 10 mm. The added depth over previous photon absorption remote sensing (PARS) comes from the novel use of deep-penetrating detection wavelengths as described above.

[0443] The light may be amplified by an optical amplifier before interacting with the sample or before detection. The light may be collected by a photodiode, an avalanche photodiode, a phototube, a photomultiplier, a CMOS camera, a CCD camera (including EM-CCD, intensified CCD, back-thinned and cooled CCD), a spectrometer, etc. The detected signal may be amplified by an RF amplifier, a lock-in amplifier, a transimpedance amplifier, or other amplifier configuration.

[0444] The modalities may be used for A-, B-, or C-scan imaging of in vivo, ex vivo, or phantom studies. The total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time-domain-photon absorption remote sensing (TD-PARS) subsystems may take the form of any embodiment common to microscopy and bioimaging technologies. Some of these may include, but are not limited to, devices implemented as tabletop microscopes, inverted microscopes, handheld microscopes, surgical microscopes, endoscopes, or ophthalmic devices. They may be constructed based on principles known in the art.

[0445] Total absorption-photon absorption remote sensing (TA-PARS), multi-pass photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time-domain photon absorption remote sensing (TD-PARS) subsystems can be optimized to utilize multifocal designs to improve the depth of focus for 2D and 3D imaging. The chromatic aberration of the collimating lens and objective lens pair can be used to refocus light from the fiber onto the object so that each wavelength is focused at a slightly different depth. These chromatic aberrations can be used to encode depth information into the recovered photon absorption remote sensing (PARS) signal, which can then be recovered using a wavelength-specific analysis approach. Simultaneous use of these wavelengths can also be used to improve the depth of field and signal-to-noise ratio (SNR) of photon absorption remote sensing (PARS) images. During imaging, depth scanning by wavelength tuning can be performed.

[0446] Photon absorption remote sensing (PARS) methods can provide lateral or axial discrimination on a sample by spatially encoding the detection area, such as by using several pinholes, or by the spectral content of a broadband beam.

[0447] Total absorption-photon absorption remote sensing (TA-PARS), multi-pass photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time-domain photon absorption remote sensing (TD-PARS) subsystems can be combined with other imaging modalities, such as stimulated Raman microscopy, fluorescence microscopy, two-photon and confocal fluorescence microscopy, coherent anti-Raman-Stokes microscopy, Raman microscopy, other photoacoustic and ultrasound systems, enabling simultaneous imaging of microcirculation, blood oxygenation parameters, and other molecular targets—potentially important tasks that are difficult to perform. Multi-wavelength visible laser sources can also be implemented to generate photon absorption signals for functional or structural imaging.

[0448] A polarization analyzer may be used to separate the detected light into its polarization states. The light detected in each polarization state may provide information about the sample. A phase analyzer may be used to separate the detected light into its phase components. This may provide information about the sample.

[0449] The total absorption-photon absorption remote sensing (TA-PARS), multi-path-photon absorption remote sensing (MP-PARS), multiphoton excitation photon absorption remote sensing (PARS), quantum efficiency ratio (QER), lifetime photon absorption remote sensing (PARS), and time domain-photon absorption remote sensing (TD-PARS) subsystems may detect generated signals in the detection beam(s) returning from the sample. These perturbations may comprise, but are not limited to, changes in intensity, polarization, frequency, phase, absorption, nonlinear scattering, and nonlinear absorption, and may be caused by various factors such as pressure, thermal effects, etc.

[0450] Analog-based signal extraction may be performed along the electrical signal path. Some examples of such analog devices may include, but are not limited to, lock-in amplifiers, peak detection circuits, etc.

[0451] The photon absorption remote sensing (PARS) subsystem can detect temporal information encoded in the back-reflected detection beam. This information identifies chromophores and can be used to enhance contrast and improve signal extraction. This temporal information can be extracted using analog and digital processing techniques. These can include, but are not limited to, the use of lock-in amplifiers, Fourier transforms, wavelet transforms, and intelligent algorithmic extraction. In one example, lock-in detection can be utilized to extract photon absorption remote sensing (PARS) signals that resemble known expected signals for the extraction of specific chromophores, such as DNA, cytochromes, and red blood cells.

[0452] The imaging head of the system may include closed-loop or open-loop adaptive optics for wavefront and aberration correction, including, but not limited to, wavefront sensors, deformable mirrors, TAG lenses, etc. Aberrations may include defocus, astigmatism, coma, distortion, third-order effects, etc. Signal-enhancing beams may also be used to suppress signals from undesired chromophores by intentionally inducing saturation effects such as photobleaching.

[0453] Various types of optics can be utilized to exploit their respective advantages. For example, an axicon can be used primarily to generate a Bessel beam with a greater depth of focus than that available with standard Gaussian beam optics. Such optics can also be used elsewhere in the beam path as deemed appropriate. Catoptric systems can also be substituted for each refractive element, such as the use of a reflective objective rather than a standard compound objective.

[0454] The optical path may include nonlinear optical elements for various related purposes such as wavelength generation and wavelength shifting. The beam foci may overlap at the sample but may be laterally and axially offset from one another by small amounts, if appropriate.

[0455] The total absorption-photon absorption remote sensing TA-PARS, multi-path-photon absorption remote sensing MP-PARS, multi-photon excitation photon absorption remote sensing PARS, quantum efficiency ratio QER, lifetime photon absorption remote sensing PARS, and time domain-photon absorption remote sensing TD-PARS subsystems may be used as spectrometers for sample analysis.

[0456] Other advantages inherent in the structure will be apparent to those skilled in the art. The embodiments described herein are illustrative and are not intended to limit the scope of the claims, which are to be interpreted in light of the specification as a whole.

[0457] It will be appreciated that the systems described herein can be used in a variety of ways, such as for those purposes described in the prior art, and in other ways to take advantage of the above-described aspects. A non-exhaustive list of uses is discussed below.

[0458] The system can be used for imaging angiogenesis in different preclinical tumor models. The system may be used to demix targets (e.g., detect, separate, or otherwise discretize constituent species and / or subspecies) based on their absorption, scattering, or frequency content by utilizing different wavelengths, different pulse widths, different coherence lengths, repetition rates, exposure times, different evolutions or lifetimes of signals, quantum efficiency ratios, and / or other comparisons of non-radiative and radiative signals, etc.

[0459] The system can be used to image images with resolution up to and beyond the diffraction limit. The system can be used to image anything that absorbs light, including exogenous and endogenous targets and biomarkers.

[0460] The system may have several surgical applications, such as functional and structural imaging during brain surgery, use for assessment of internal bleeding and ablation verification, imaging organs and organ transplants for perfusion adequacy, imaging angiogenesis around pancreatic islet transplants, imaging skin grafts, imaging tissue scaffolds and biomaterials to assess angiogenesis and immune rejection, imaging to assist in microsurgery, and guidance to avoid cutting critical blood vessels and nerves.

[0461] The system may also have several gastroenterological applications, such as imaging the vascular bed and depth of invasion in Barrett's esophagus and colorectal cancer. In at least some embodiments, depth of invasion is important for prognosis and metabolic potential. This may be used for virtual biopsies, monitoring Crohn's disease and IBS, and carotid artery examinations. Gastroenterological applications may be combined with or piggybacked onto clinical endoscopes, and the miniaturized photon absorption remote sensing PARS system may be designed either as a standalone endoscope or to fit within the accessory channel of a clinical endoscope.

[0462] The system may also be used for clinical imaging of macro- and microcirculatory cells and pigment cells, which may find use in applications such as: (1) the eye, with the potential to extend or replace fluorescence angiography; (2) imaging of skin lesions, with imaging of melanoma, basal cell carcinoma, hemangioma, psoriasis, eczema, dermatitis, Mohs surgery, and imaging to verify tumor margin resection; (3) peripheral vascular disease; (4) diabetes and pressure ulcers; (5) burn imaging; (6) plastic surgery and microsurgery; (7) imaging of circulating tumor cells, particularly melanoma cells; (8) imaging of lymph node angiogenesis; (9) imaging of response to photodynamic therapy with a vascular ablation mechanism; (10) imaging of response to chemotherapy with anti-angiogenic drugs; and (11) imaging of response to radiotherapy.

[0463] The system can also be used for several histopathology imaging applications, such as frozen pathology, generating H&E stain-like images from tissue samples, and virtual biopsies. The system can be implemented to generate virtual staining and other types of images for various tissue preparations, such as formalin-fixed paraffin-embedded (FFPE) tissue blocks, formalin-fixed paraffin-embedded (FFPE) tissue slides, formalin-fixed paraffin-embedded FFPE tissue sections, frozen pathology sections, formalin-fixed tissue, freshly excised unprocessed tissue, and freshly excised specimens. Visualization of macromolecules, such as DNA, RNA, cytochromes, lipids, and proteins, can be performed within these tissue samples.

[0464] The generated stains or images may be used in one or more histopathology imaging applications for different diseases, including, but not limited to, wound healing, angiogenesis and tissue regeneration, hypersensitivity, infection, inflammation, autoimmunity, scarring and fibrosis.

[0465] The system may be useful for estimating oxygen saturation using multi-wavelength photon absorption remote sensing (PARS) excitation in applications including: (1) estimating venous oxygen saturation where pulse oximetry cannot be used, including estimating cerebral venous oxygen saturation and central venous oxygen saturation, which may replace catheter placement procedures that can be dangerous, especially in children and infants.

[0466] Oxygen flux and oxygen consumption can be estimated using photon absorption remote sensing PARS imaging to estimate oxygen saturation, as well as by estimating blood flow in the blood vessels entering and leaving a region of tissue.

[0467] The system can be useful for separating prominent histological chromophores such as cell nuclei from the surrounding cytoplasm by exploiting their respective absorption spectra. The system can be used to unmix targets using absorption content, scattering, phase, polarization, or frequency content by utilizing different wavelengths, different pulse widths, different coherence lengths, repetition rates, flow rates, exposure times, etc.

[0468] Other example applications include imaging of contrast agents in clinical or preclinical applications, identification of sentinel lymph nodes, non-invasive or minimally invasive identification of tumors in lymph nodes, non-destructive testing of materials, imaging of genetically encoded reporters such as tyrosinase, chromoproteins, fluorescent proteins, etc. for preclinical or clinical molecular imaging applications, imaging of actively or passively targeted optically absorbing nanoparticles for molecular imaging, and imaging of thrombi and potentially staging the age of the thrombus.

[0469] Other examples of applications may include clinical and preclinical ophthalmology applications, oximetry and retinal metabolic rate measurements in diseases such as age-related macular degeneration, diabetic retinopathy and glaucoma, corneal vasculature and stem cell imaging, corneal nerve and neovascularization imaging, evaluation of Schlemm's canal changes in glaucoma patients, choroidal neovascularization imaging, anterior and posterior segment blood flow imaging and blood flow status, wound healing, angiogenesis and tissue regeneration, hypersensitivity, infection, inflammation, autoimmunity, and scarring and fibrosis.

[0470] The system can be used to measure and estimate metabolism in biological samples, leveraging the capabilities of both photon absorption remote sensing (PARS) and optical coherence tomography (OCT). In this example, optical coherence tomography (OCT) can be used to estimate volumetric blood flow in a region of interest, and a photon absorption remote sensing (PARS) system can be used to measure oxygen saturation in the blood vessels of interest. The combination of these measurements can then provide an estimate of metabolism in the region.

[0471] The system may be used to examine the vasculature of stroke patients to help identify head and neck cancer types and skin cancer types, functional brain activity, location of blood clots, monitoring changes in neurological and brain function / development as a result of changes in atherosclerotic plaques, gut bacterial composition, oxygen sufficiency after flap reconstruction, monitoring sufficiency after cosmetic or cosmetic surgery, and imaging cosmetic injectables.

[0472] The system can be used to track the topology of surface deformations. For example, optical coherence tomography (OCT) can be used to track the location of the sample surface. Corrections can then be applied to a tightly focused photon absorption remote sensing (PARS) device using mechanisms su...

Claims

1. 1. A computer-implemented method for analyzing a sample, the computer-implemented method comprising: receiving a plurality of signals from the sample, the signals comprising radioactive and non-radioactive signals; extracting a plurality of features based on processing at least one of the plurality of signals, the plurality of features providing information of the contrast provided by the at least one of the plurality of signals; applying the features to a machine learning architecture to generate inferences about the samples; A computer-implemented method comprising:

2. The step of processing a plurality of said signals comprises: exciting the sample at an excitation location with an excitation beam, the excitation beam being focused at or below the sample; interrogating the sample with an interrogation beam directed towards the excitation location of the sample, the interrogation beam being focused at or below the sample. The computer-implemented method of claim 1 .

3. extracting the plurality of features comprises processing both radioactive and non-radioactive signals. The computer-implemented method of claim 1 .

4. the plurality of signals comprises absorption spectrum signals; The computer-implemented method of claim 1 .

5. the plurality of signals comprises scattered signals; The computer-implemented method of claim 1 .

6. The sample is an in vivo or in situ sample. The computer-implemented method of claim 1 .

7. The sample is unstained, The computer-implemented method of claim 1 .

8. The sample is stained. The computer-implemented method of claim 1 .

9. the plurality of features being complemented by at least one informative feature of image data acquired from a complementary modality; The computer-implemented method of claim 1 .

10. The complementary modalities are: at least one image from ultrasound imaging, a positron emission tomography (PET) scan, a computed tomography (CT) scan, and a magnetic resonance imaging (MRI); one or more photoactive labels for contrasting or highlighting specific areas within at least one of said images; Equipped with 10. The computer-implemented method of claim 9.

11. the plurality of features being complemented by at least one informative feature of patient information; The computer-implemented method of claim 1 .

12. processing at least one of the plurality of signals comprises converting the at least one of the plurality of signals into at least one image. The computer-implemented method of claim 1 .

13. and converting to at least one of the images comprises applying a simulation stain.

13. The computer-implemented method of claim 12.

14. The simulated staining comprises at least one of hematoxylin and eosin (H&E) staining, Jones staining (MPAS), PAS and GMS staining, toluidine blue, Congo red, Masson's trichrome staining, Lilly's trichrome, and Verhoeff's staining, immunohistochemistry (IHC), histochemical staining, and in situ hybridization (ISH); 14. The computer-implemented method of claim 13.

15. The simulation staining can be applied to frozen tissue sections, archived tissue samples, or fresh, unprocessed tissue.

15. The computer-implemented method of claim 14.

16. The step of converting into at least one of the images comprises: converting the image into at least two images; applying a different simulation stain to each of said images; Equipped with 13. The computer-implemented method of claim 12.

17. converting to at least one of the images comprises applying a colorized machine learning architecture.

13. The computer-implemented method of claim 12.

18. The colored machine learning architecture comprises at least one generative adversarial network (GAN).

18. The computer-implemented method of claim 17.

19. The generative adversarial network (GAN) comprises one of a cycle-consistent generative adversarial network (CycleGAN) and a conditional generative adversarial network (cGAN).

20. The computer-implemented method of claim 18.

20. the inference comprises at least one of survival time, drug response, drug resistance, phenotypic characteristics, molecular characteristics, mutational burden, tumor molecular characteristics, parasites, toxicity, inflammation, transcriptomic features, protein expression features, patient clinical outcome, suspicious signals, biomarker location or value, cancer grade, cancer subtype, tumor border area, and grouping of cancer cells based on cell size and shape; The computer-implemented method of claim 1 .

21. The method further comprises generating a signal to cause rendering, at a display device, a user interface (UI) showing the visualization of the inference. The computer-implemented method of claim 1 .

22. the contrast comprises one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast; The computer-implemented method of claim 1 .

23. the inference comprises a probability of disease for at least one region within the sample; the probability of the disease is determined based on a plurality of the features and a complementary data stream received by the machine learning architecture. The computer-implemented method of claim 1 .

24. The reasoning further states: a heat map identifying one or more regions of the sample; a corresponding probability of disease for each of one or more of said regions of said sample; and Equipped with 24. The computer-implemented method of claim 23.

25. the corresponding probability of disease for each of the one or more regions of the sample is indicated by a corresponding intensity of color shown in each of the regions in the heat map.

25. The computer-implemented method of claim 24.

26. the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal. The computer-implemented method of claim 1 .

27. the radioactive signals comprise one or more autofluorescence signals; The computer-implemented method of claim 1 .

28. the plurality of signals comprises radiative and non-radiative absorption relaxation signals; The computer-implemented method of claim 1 .

29. 1. A computer-implemented method for training a machine learning architecture to generate simulated stain images, the machine learning architecture comprising a neural network having a plurality of nodes and weights stored in a memory device, the computer-implemented method comprising, at each training iteration: acquiring a true total absorption TA image; generating a simulated stained image based on the true total absorption TA image; generating a false total absorption TA image based on the generated simulated stained image; calculating a first loss based on the generated false total absorption TA image and the generated true total absorption TA image; acquiring a labeled stained image; Calculating a second loss based on the generated simulated stained image and the labeled stained image; updating weights of the neural network based on at least one of the first loss and the second loss; A computer-implemented method comprising:

30. the simulated stained image is generated by a second neural network comprising a second set of nodes and weights; the second set of weights are updated during each iteration based on at least one of the first loss and the second loss.

30. The computer-implemented method of claim 29.

31. the fake total absorption TA image is generated by a third neural network comprising a second set of nodes and weights; the third set of weights is updated during each iteration based on at least one of the first loss and the second loss.

30. The computer-implemented method of claim 29.

32. The step of calculating the second loss based on the generated simulated stained image and the labeled stained image includes: processing the generated simulated stained image through a first identifier network; processing the labeled stained image through a second identifier network; and calculating the second loss based on respective outputs from each of the first and second identifier networks.

30. The computer-implemented method of claim 29.

33. The computer-implemented method further comprises processing the respective outputs from each of the first and second identifier networks through a respective classification matrix before calculating the second loss.

33. The computer-implemented method of claim 32.

34. The machine learning architecture comprises at least one generative adversarial network (GAN).

30. The computer-implemented method of claim 29.

35. The generative adversarial network (GAN) comprises one of a cycle-consistent generative adversarial network (CycleGAN) and a conditional generative adversarial network (cGAN).

35. The computer-implemented method of claim 34.

36. The labeled stained image is a labeled photon absorption remote sensing (PARS) image.

30. The computer-implemented method of claim 29.

37. The labeled photon absorption remote sensing PARS images are automatically labeled based on unlabeled photon absorption remote sensing PARS images prior to training of the neural network.

37. The computer-implemented method of claim 36.

38. automatically labeling the unlabeled photon absorption remote sensing PARS image comprises labeling the unlabeled photon absorption remote sensing PARS image based on existing labeled stained images from a database; The existing labeled stain image and the unlabeled photon absorption remote sensing (PARS) image share structural similarities.

38. The computer-implemented method of claim 37.

39. 1. A system as a computer system for analyzing a sample, comprising: The system includes a processor operating in conjunction with computer memory and non-transitory computer-readable storage, the processor comprising: receiving a plurality of signals from the sample, the signals comprising radioactive and non-radioactive signals; extracting a plurality of features based on processing at least one of the plurality of signals, the plurality of features providing information of the contrast provided by the at least one of the plurality of signals; applying the features to a machine learning architecture to generate inferences about the samples; The system is configured to run

40. The step of processing a plurality of said signals comprises: exciting the sample at an excitation location with an excitation beam, the excitation beam being focused at or below the sample; interrogating the sample with an interrogation beam directed towards the excitation location of the sample, the interrogation beam being focused at or below the sample; Equipped with 40. The system of claim 39.

41. extracting the plurality of features comprises processing both radioactive and non-radioactive signals.

40. The system of claim 39.

42. the plurality of signals comprises absorption spectrum signals; 40. The system of claim 39.

43. the plurality of signals comprises scattered signals; 40. The system of claim 39.

44. The sample is an in vivo or in situ sample.

40. The system of claim 39.

45. The sample is unstained, 40. The system of claim 39.

46. The sample is stained.

40. The system of claim 39.

47. the plurality of features being complemented by at least one informative feature of image data acquired from a complementary modality; 40. The system of claim 39.

48. the plurality of signals comprises radiative and non-radiative absorption relaxation signals; 40. The system of claim 39.

49. The complementary modalities are: at least one image from ultrasound imaging, a positron emission tomography (PET) scan, a computed tomography (CT) scan, and a magnetic resonance imaging (MRI); one or more photoactive labels for contrasting or highlighting specific areas within at least one of said images; Equipped with 48. The system of claim 47.

50. the plurality of features being complemented by at least one informative feature of patient information; 40. The system of claim 39.

51. processing at least one of the plurality of signals comprises converting the at least one of the plurality of signals into at least one image.

40. The system of claim 39.

52. and converting to at least one of the images comprises applying a simulation stain.

52. The system of claim 51.

53. The simulated staining comprises at least one of hematoxylin and eosin (H&E) staining, Jones staining (MPAS), PAS and GMS staining, toluidine blue, Congo red, Masson's trichrome staining, Lilly's trichrome, and Verhoeff's staining, immunohistochemistry (IHC), histochemical staining, and in situ hybridization (ISH); 53. The system of claim 52.

54. The simulation staining can be applied to archived tissue samples, frozen tissue sections, or fresh, unprocessed tissue.

54. The system of claim 53.

55. The step of converting into at least one of the images comprises: converting the image into at least two images; applying a different simulation stain to each of said images; Equipped with 52. The system of claim 51.

56. converting to at least one of the images comprises applying a colorized machine learning architecture.

52. The system of claim 51.

57. The colored machine learning architecture comprises at least one generative adversarial network (GAN).

57. The system of claim 56.

58. The generative adversarial network (GAN) comprises one of a cycle-consistent generative adversarial network (CycleGAN) and a conditional generative adversarial network (cGAN).

58. The system of claim 57.

59. the inference comprises at least one of survival time, drug response, drug resistance, phenotypic characteristics, molecular characteristics, mutational burden, tumor molecular characteristics, parasites, toxicity, inflammation, transcriptomic features, protein expression features, patient clinical outcome, suspicious signals, biomarker location or value, cancer grade, cancer subtype, tumor border area, and grouping of cancer cells based on cell size and shape; 40. The system of claim 39.

60. the processor is further configured to generate a signal to cause rendering, on a display device, a user interface (UI) showing a visualization of the inference.

40. The system of claim 39.

61. the contrast comprises one of absorption contrast, scattering contrast, attenuation contrast, amplitude contrast, phase contrast, decay rate contrast, and lifetime decay rate contrast; 40. The system of claim 39.

62. the inference comprises a probability of disease for at least one region within the sample; the probability of the disease is determined based on a plurality of the features and a complementary data stream received by the machine learning architecture.

40. The system of claim 39.

63. The reasoning further states: a heat map identifying one or more regions of the sample; a corresponding probability of disease for each of one or more of said regions of said sample; and Equipped with 63. The system of claim 62.

64. the corresponding probability of disease for each of the one or more regions of the sample is indicated by a corresponding intensity of color shown in the each region within the heatmap.

64. The system of claim 63.

65. the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal.

40. The system of claim 39.

66. the radioactive signals comprise one or more autofluorescence signals; 40. The system of claim 39.

67. 1. A non-transitory computer-readable medium storing a set of machine-interpretable instructions that, when executed by a processor, cause the processor to: receiving a plurality of signals from the sample, the signals comprising radioactive and non-radioactive signals; extracting a plurality of features based on processing at least one of the plurality of signals, the plurality of features providing information of the contrast provided by the at least one of the plurality of signals; applying the features to a machine learning architecture to generate inferences about the samples; A non-transitory computer-readable medium that causes

68. the inference comprises a probability of disease for at least one region within the sample; the probability of the disease is determined based on a plurality of the features and a complementary data stream received by the machine learning architecture.

68. The non-transitory computer-readable medium of claim 67.

69. The reasoning further states: a heat map identifying one or more regions of the sample; a corresponding probability of disease for each of one or more of said regions of said sample; and Equipped with 69. The non-transitory computer-readable medium of claim 68.

70. the corresponding probability of disease for each of the one or more regions of the sample is indicated by a corresponding intensity of color shown in the each region within the heatmap.

70. The non-transitory computer-readable medium of claim 69.

71. the non-radiative signal comprises at least one of a photothermal signal and a photoacoustic signal.

68. The non-transitory computer-readable medium of claim 67.

72. the radioactive signals comprise one or more autofluorescence signals; 68. The non-transitory computer-readable medium of claim 67.

73. 1. A system as a computer system for training a machine learning architecture, said system comprising: a processor operating in conjunction with computer memory and non-transitory computer readable storage, the processor performing, in each training iteration: instantiating a machine learning architecture comprising a neural network having a plurality of nodes and weights stored in a memory device; acquiring a true total absorption TA image; generating a simulated stained image based on the true total absorption TA image; generating a false total absorption TA image based on the generated simulated stained image; calculating a first loss based on the generated false total absorption TA image and the true total absorption TA image; acquiring a labeled stained image; Calculating a second loss based on the generated simulated stained image and the labeled stained image; updating weights of the neural network based on at least one of the first loss and the second loss; The system is configured to run

Images