Method and system for raman spectroscopy

Abstract

To provide a method for analyzing spectral data obtained by Raman spectroscopy.SOLUTION: A spectral sequence is generated based on photons acquired in response to irradiating a position of a sample with a plurality of light pulses. The spectral sequence is processed by a trained 2D neural network to generate a Raman spectrum with reduced fluorescence.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] FIELD OF THE INVENTION This specification relates generally to methods and systems for analyzing spectral data, and more particularly to analyzing spectral data acquired by Raman spectroscopy. [Background technology]

[0002] Raman spectroscopy is a spectroscopic technique that can be used to characterize and determine the composition of a sample. Fluorescence background is a common problem in Raman spectroscopy. When a sample is illuminated with light, it can produce both Raman scattering and fluorescence. Because the probability of Raman scattering is much lower than the probability of fluorescence, the detected fluorescence signal can be strong enough to overwhelm the weaker Raman signal (detected Raman scattering). This strong fluorescence background reduces the signal-to-noise ratio (SNR) of the Raman signal, introducing errors into the identification and quantitative measurement of substances.

[0003] There are several methods for reducing the fluorescence background in Raman spectroscopy. For example, light with a longer wavelength can be used for illumination / excitation. However, Raman signal intensity decreases as the wavelength increases. Alternatively, the fluorescence background can be reduced by collecting scattered light from the sample via time-correlated single-photon counting (TCSPC), such as by using a single-photon avalanche diode (SPAD) array detector. SPADs are an emerging technology that promises to match or exceed the sensitivity of mature technologies such as CCDs (charge-coupled devices) and CMOS (complementary metal-oxide semiconductors), while still providing the arrival time of each detected photon on the picosecond scale. By using a SPAD array as a detector, spectral data can be acquired with zero read noise. Furthermore, time-resolved detection offers the opportunity for fluorescence suppression and / or fluorescence lifetime detection. In addition to SPADs, other detectors such as photomultiplier tubes (PMTs), hybrid photodetectors (HPDs), intensified CCDs, gated CMOS, and Kerr-gate detectors can be used for TCSPC. Summary of the Invention

[0004] A method for Raman spectroscopy includes illuminating a sample location with a plurality of light pulses and acquiring photons from the sample, where acquiring the photons includes recording the arrival time of each photon; generating a spectral series based on the acquired photons, where the spectral series includes a plurality of spectra, each of the spectra constructed from photons having substantially the same arrival time from a corresponding illumination with the light pulses; processing the spectral series with a trained 2D neural network; and generating a Raman spectrum from the 2D neural network.

[0005] A method for Raman spectroscopy includes receiving a spectral sequence including a plurality of spectra acquired at a sample location, each spectrum of the plurality of spectra corresponding to a different arrival time, the plurality of spectra acquired by illuminating the sample location with a plurality of light pulses and acquiring photons from the sample, wherein acquiring the photons includes recording the arrival time of each of the photons; processing the spectral sequence with a trained 2D neural network; and generating a Raman spectrum from the trained 2D neural network.

[0006] The microscope system includes a detector, a pulsed laser for generating light pulses, a sample holder for positioning a sample, a processor, and a controller including a non-transitory memory for storing computer-readable instructions, wherein by executing the instructions in the processor, the microscope system is configured to: irradiate a location of the sample with a plurality of light pulses with the laser; acquire photons from the sample with the detector; wherein acquiring the photons includes recording the arrival time of each photon from the corresponding irradiation with the light pulse; generate a spectral series based on the acquired photons, the spectral series including a plurality of spectra, each of the spectra constructed from photons having substantially the same arrival time; process the spectral series using a trained 2D neural network; and generate a Raman spectrum from the 2D neural network.

[0007] It should be understood that the foregoing summary is provided to introduce a selection of concepts in a simplified form that are further described in the detailed description. It is not intended to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Moreover, the claimed subject matter is not limited to implementations that solve any disadvantages discussed above or in any part of this disclosure. [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates a system configuration for Raman spectroscopy, according to some embodiments. [Figure 2] 2 illustrates the wavelength-selective element and detector of FIG. 1 according to some embodiments. [Figure 3] FIG. 10 is a diagram showing the time characteristics of a laser pulse, a fluorescent signal, and a Raman signal. [Figure 4] 1 shows an exemplary spectral sequence. [Figure 5A] 1 shows the Raman components of the training data. [Figure 5B] Fluorescence components of the training data are shown. [Figure 6]1 is a flowchart for processing a spectral sequence using a 2D neural network, according to some embodiments. [Figure 7] The measured spectrum and the output spectrum from the trained neural network are shown. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following description relates to systems and methods for Raman spectroscopy, and more particularly to using neural networks to remove fluorescence background in Raman spectroscopy.

[0010] In some embodiments, the method includes receiving a spectral sequence including a plurality of spectra, each spectrum of the spectral sequence corresponding to a different arrival time. The arrival time of a photon at the detector is related to the generation of a corresponding light pulse that causes the emission of a photon from the sample. The occurrence of the light pulse, or zero arrival time, can be the time when the detector detects a laser pulse or the time when an electrical signal related to the occurrence of the light pulse is received by the detector. For example, the electrical signal can be an electrical trigger pulse for generating the light pulse or an electrical synchronization pulse generated by the detector electronics to trigger the laser. To improve the signal-to-noise ratio, the spectral sequence can be obtained by integrating sample light detected in response to illuminating the sample location with multiple light pulses. In one embodiment, the spectral sequence at one sample location is 10 4 ~10 10 The spectral sequence is acquired by illuminating the sample with light pulses. The spectral sequence is processed with a trained 2D neural network. A Raman spectrum is generated from the trained 2D neural network. The term Raman spectrum refers to the spectrum of the Raman signal containing the sample-specific Raman signature.

[0011] In another embodiment, a method for Raman spectroscopy includes illuminating a location on the sample with a plurality of light pulses and acquiring photons from the sample in response to the illumination, where acquiring the photons includes recording arrival times of the photons; generating a spectral sequence based on the acquired photons, where the spectral sequence includes a plurality of spectra, each of the spectra constructed from photons having substantially the same arrival time; processing the spectral sequence with a trained 2D neural network; and generating a Raman spectrum from the trained 2D neural network.

[0012] The light beam can be a light pulse having a pulse duration of less than a nanosecond. In some embodiments, the light pulse is provided by a laser, and the light pulse has a duration of less than 100 picoseconds. In some embodiments, the light pulse is generated by a picosecond pulsed laser.

[0013] A spectral sequence includes multiple spectra, each corresponding to a different arrival time of a photon at a detector. A spectral sequence is a two-dimensional data set. For example, a spectral sequence can be represented by an M×N data array, where M is the number of time points or time bins, and N is the number of spectral points (or wavelengths / wavenumbers) in each spectrum (or the length of the spectrum). Figure 4 shows one spectral sequence. Arrival times belonging to the same time point or time bin are considered to have the same arrival time in this specification. The time point or time bin corresponds to the time resolution for acquiring the sample light. The time resolution can be less than 50 picoseconds.

[0014] The multiple spectra in the spectral sequence are generated from detected photons scattered or emitted from the same sample location in response to illumination of the sample with a light pulse. The photons are detected and recorded over a period of time following illumination of the sample location with each light pulse, which is shorter than the duration between successive light pulses. The spectral sequence can include a first portion of the spectrum having an earlier arrival time and a second portion of the spectrum having a later arrival time. The first portion can include both fluorescence and Raman signals, and the second portion can include only fluorescence signals.

[0015] The spectral sequence is processed using a trained 2D neural network (NN) to remove the fluorescence signal (i.e., the fluorescence background) from the detected signal. The trained 2D NN takes in one spectral sequence and outputs / generates a Raman spectrum with reduced fluorescence background. The 2D NN may be a convolutional neural network (CNN). The 2D NN may also be an autoencoder. The Raman spectrum is a one-dimensional signal, and the length of the Raman spectrum is the same as at least one dimension of the spectral sequence. For example, the output Raman spectrum is one-dimensional data with a length of N. Therefore, the trained 2D NN does not change the resolution or spectral range of the spectral data during processing. The trained 2D NN acts as a filter to remove the fluorescence background from the Raman signal.

[0016] The trained 2D NN can be generated by training the 2D NN using either simulated data and / or experimental data. The training data includes a plurality of training spectral sequences and corresponding Raman spectra. The training spectral sequence can be generated from a pure Raman spectrum, a fluorescence background, and a temporal profile of a light pulse. The pure Raman spectrum is a Raman spectrum of a known substance without the fluorescence background. The pure Raman spectrum can be obtained from a library. The fluorescence background (or pure fluorescence spectrum) is a fluorescence spectrum of a known substance without the Raman spectrum. The fluorescence background can be generated based on the fluorescence lifetime of a sample. The fluorescence time of the sample can be known or measured. The fluorescence background can be generated using an exponential decay model and fluorescence time. The temporal profile of the laser pulse is the intensity of the laser pulse over time, which can be measured experimentally.

[0017] In some embodiments, the trained 2D NN is retrained with a training spectral sequence generated based on a measured time profile of the laser. The trained 2D NN can be retrained to compensate for parameter drift of the Raman microscope system over time.

[0018] The composition of a sample can be analyzed based on the Raman spectrum output from the trained neural network. For example, the composition of a sample can be determined by comparing the Raman spectrum of the sample with reference spectra of known substances. Furthermore, the substances present in the sample can be quantified based on the Raman spectrum.

[0019] In some embodiments, a microscope system for Raman spectroscopy comprises: a detector; a pulsed laser for generating light pulses; a sample holder for positioning a sample; a processor; and a controller including a non-transitory memory for storing computer-readable instructions, wherein by executing the instructions in the processor, the microscope system is configured to: irradiate a location of the sample with a plurality of light pulses with the laser; acquire photons from the sample with the detector; wherein acquiring the photons includes recording the arrival time of each photon from the corresponding irradiation with the light pulse; generate a spectral series based on the acquired photons, the spectral series comprising a plurality of spectra, each of the spectra constructed from photons having substantially the same arrival time; process the spectral series using a trained 2D neural network; and generate a Raman spectrum from the 2D neural network.

[0020] For example, light from the sample is spatially separated based on wavelength via a spectrometer before reaching the detector. The detector can be an array detector that can detect photons arriving at each pixel of the array, which correlates to the spectral information of the light. The detector also records the arrival time of the photon relative to the light pulse that illuminated the sample. The detector can be a SPAD detector.

[0021] The microscope system may further include a scanner for directing the light beam to different sample locations and acquiring at least one spectral series at each of the sample locations, and an image indicative of the compositional distribution in the sample region may be generated based on the Raman spectra from the multiple sample locations.

[0022] 1 shows a high-level configuration of a microscope system for performing Raman spectroscopy, such as Raman system 100. It will be understood that different optical architectures for Raman spectroscopy are known in the art, and therefore the example of FIG. 1 should not be considered limiting.

[0023] The Raman system 100 includes a light source 101 that generates a light beam 107, which may include a laser source. The laser source may be a pulsed laser that generates light pulses. The light pulses may have a pulse duration of less than 1 nanosecond. The laser source may be a picosecond laser. The light beam is directed toward a sample 103 by sequentially passing through a dichroic mirror 104 and an optical assembly 102. The signal 103 may include any type of signal of interest to a user, which may include a substantially dry signal (e.g., powder, solid material), a substantially fluid signal (e.g., liquid, gas), or some combination thereof (e.g., gel).

[0024] The optical assembly 102 may include a scanner and at least one objective lens. The scanner scans a light beam over the sample so that the sample area can be imaged. The objective lens focuses the light beam to a point on the sample. In response to light illuminating the sample 103, light is scattered or emitted from the sample. The sample may be held by a sample stage (not shown). Different sample positions may be scanned by adjusting the light beam and / or the sample stage. Light 108 scattered or emitted from the sample (e.g., including the Raman, fluorescent, and Rayleigh portions of the scattered light) is directed through the optical assembly 102 toward a dichroic mirror 104. The dichroic mirror 104 reflects the light 108 from the sample toward a wavelength-selective element 105. The wavelength-selective element selects a wavelength range of light to be detected by a detector 106. The detector 106 may be a line sensor or an array sensor. The wavelength-selective element may include one or more of a filter or a monochromator. The wavelength-selective element may include a spectrometer for separating the incident light based on its spectral components, different wavelengths, or colors, and projecting each component onto the detector 106. The detector may be a line detector or an array detector capable of performing time-correlated single-photon counting. The detector may be a SPAD detector. In some embodiments, the detector may be a photomultiplier tube (PMT), a hybrid photodetector (HPD), a multiplied CCD, a gated CMOS, and a Kerr gate detector. One embodiment of the wavelength-selective element 105 and the detector 106 is shown in FIG. 2.

[0025] Signal processing and / or digitization of signals related to scattered light received by detector 106 can be handled by an electronic signal processor or controller 113. In some embodiments, controller 113 can be a suitably programmed microprocessor or application specific integrated circuit including any known type of read-only or read-write memory that holds computer-readable instructions and data for spectrometer operation as described herein. The controller can further control one or more of light source 101, one or more components of optical assembly 102, and detector 106. Input / output devices can be connected to the controller for receiving user instructions and / or displaying data to a user.

[0026] 2 shows one embodiment of the wavelength-selective element 105 and the detector 106. The wavelength-selective element includes a spectrometer 202, such as a volume holographic grating. The wavelength-selective element 105 may further include a lens 208 for focusing spatially separated light from the spectrometer 202 onto a detection element of the detector 106. The wavelength-selective element 105 may include a beam-limiting device 206, such as an optical fiber or an input splitter, for directing light 214 from the sample to the spectrometer 202. If the beam-limiting device is an optical fiber, it also functions as a light guide for directing the light to the spectrometer. A lens 204 may be positioned between the beam-limiting device 206 and the spectrometer 202 to direct the light to the spectrometer 202.

[0027] Figure 3 shows the timing of the fluorescence signal 302 and the Raman signal 306 relative to the laser pulse 304. The laser pulse 304 is the Rayleigh scattering of a light pulse irradiated onto a sample. The x-axis is time in units of σ, which is the full width at half maximum (FWHM) of the temporal profile of the laser pulse 304. The y-axis is the normalized intensity of the various signals. The origin of the x-axis (t=0) is when the peak of the laser pulse hits the detector, or the time when the detector detects the laser pulse. Thus, the x-axis represents the arrival time.

[0028] After irradiating the sample with a light pulse, both the Raman signal 306 and the fluorescence signal 302 are generated substantially instantaneously and thus overlap in time with the laser pulse 304. The temporal profile of the Raman signal 306 substantially matches the shape of the temporal profile of the laser pulse 304. For example, as shown in FIG. 3, a Gaussian temporal profile of the laser pulse generates an overlapping Gaussian Raman signal. The fluorescence signal 302 decays exponentially with a sample-dependent lifetime τ; therefore, some fluorescence emission overlaps in time with the Rayleigh scattering and Raman signal of the laser pulse, but the majority of the fluorescence emission occurs afterward. Here, a Gaussian temporal profile of the laser pulse is shown as an example. The temporal profile of the laser pulse can have other shapes. In some embodiments, the laser's temporal profile may change over time and therefore need to be recalibrated or measured.

[0029] A first portion of the spectral sequence, e.g., 0-4σ, contains both the fluorescence signal and the Raman signal, and most of the first portion of the spectral sequence may be the Raman signal. A second, later portion of the spectral sequence, e.g., 4σ-30σ, contains most of the fluorescence signal and a small portion of the Raman signal. The fluorescence signal 302 needs to be removed from the detected spectral data to obtain the Raman signal 306.

[0030] One way to separate the fluorescence signal from the Raman signal is through time gating, where the detector has a defined cutoff time 308, typically tens or hundreds of picoseconds after the laser pulse. All photons before the cutoff are retained as signal photons, and all photons after the cutoff are discarded as background photons.

[0031] Another method for separating the fluorescence signal from the Raman signal is disclosed herein, in which the arrival time of each photon is measured and recorded with a time resolution of Δt using TCSPC. After measuring the emission from multiple laser pulses, the TCSPC detector generates a plot of emission versus time (arrival time). Classical methods for separating TCSPC data into Raman and fluorescence components include deconvolution and principal component analysis. Here, the fluorescence signal is separated from the Raman signal using a trained 2D neural network. By using a neural network, the fluorescence background can be more effectively or completely removed from the detected signal without reducing the SNR of the Raman signal.

[0032] FIG. 4 shows an example of one spectral sequence. The y-axis is time (or arrival time), and the origin (t=0) corresponds to the time when the laser pulse arrives / is received at the detector. The x-axis is wavelength (or wavenumber / energy). The x-axis can alternatively be represented as a number of detector pixels. Each time along the y-axis corresponds to a spectrum along the x-axis. The spectral sequence contains spectra acquired at a time resolution determined by the detector. The amplitude of each pixel in FIG. 4 corresponds to the photon count (or signal intensity) received at a particular detector pixel (or particular wavelength). At arrival times near t=0, three peaks are presented, corresponding to the peaks of the fluorescence signal, the Raman signal, and the scattered laser light.

[0033] A spectral sequence at a sample location is acquired after illuminating the sample location with multiple light pulses. Photons scattered or emitted from the sample are collected by a pixelated detector. Each detector pixel can measure one photon per light pulse. The detector records the pixel location that received the photon as well as the time the photon arrived at the detector. The arrival time of each detected photon relative to the corresponding laser pulse can then be calculated and used to stack the spectral data collected from the sample location into a spectral sequence.

[0034] After acquiring spectral data at one sample position, the relative position of the sample and the light beam can be adjusted (by scanning the beam and / or translating the sample) to acquire spectral data from another sample position.

[0035] 5A and 5B show some aspects of training data for training a 2D NN (such as the 2D NN shown in FIG. 6). The training data includes a plurality of spectral sequences and corresponding Raman spectra. The training data may be simulated data, experimental data, or data generated based on both simulated and experimental data.

[0036] In one embodiment, training data can be generated from simulations based on pure Raman and pure fluorescence spectra. A pure Raman spectrum is a spectrum from Raman scattering alone, without any fluorescence components. A pure Raman spectrum can be obtained from a library of known Raman spectra (or Raman peaks). A pure fluorescence spectrum is a spectrum from fluorescence alone, without any Raman components. A pure fluorescence spectrum can be generated from known fluorescence lifetimes. Figure 5A shows a simulated pure Raman spectrum containing three Raman peaks. Figure 5B shows a pure fluorescence spectrum. Compared to the Raman spectrum, the fluorescence spectrum has a broad peak and a long decay after the peak.

[0037] The training spectral series can be reconstructed from pure Raman spectra, pure fluorescence spectra, and laser pulse characteristics. The laser pulse characteristics include the temporal signature of the laser pulse. For example, the laser pulse characteristics include one or more of the profile of the temporal profile of the laser pulse, the pulse shape (i.e., the spatial distribution of the laser pulse), and the pulse duration. The simulated training data is constructed to closely resemble the actual data, including matching the laser pulse temporal profile and detector noise / baseline, as well as the shape of the Raman peak.

[0038] In one embodiment, the spectral sequence can be generated via simulation based on a known pure Raman spectrum, the time profile of the laser pulse, and a known fluorescence lifetime. For example, if the laser pulse profile in time has a Gaussian shape,

number

number

number

[0039] Fluorescence is expressed as Equation 2

number

number

number

number

[0040] In some embodiments, training data can be generated from experimental data, and acquired spectral sequences can be processed with other fluorescence reduction methods to obtain pure Raman and pure fluorescence spectra for training. A 2D NN can be trained at the factory using the training data. The trained NN can then be stored in the Raman spectroscopy system for processing user-acquired spectral sequences.

[0041] In one embodiment, a 2D NN can be trained using both simulated and experimental data. A trained 2D NN can be first generated using simulated data, as described above. Transfer learning can then be used. In transfer learning, a previously trained general model is retrained with a second, smaller set of target training data compared to the simulated training data. The second training data is acquired based on measured data from a sample. In one embodiment, the second training data is experimental label data, and the labels (pure Raman spectra) are generated by a classical fitting method (such as background subtraction or fitting). In another embodiment, the second training data can be acquired by exciting a signal with multiple laser colors, with one wavelength causing fluorescence from the signal and one wavelength not. In another embodiment, the second training data is acquired using another fluorescence reduction technique, such as Kerr gating or stimulated emission depletion. In yet another embodiment, the second training data is generated using measurements from a pure signal for Raman spectra and measurements from the same signal with a fluorescent substance added. Retraining of a trained 2D NN can be done by the user with samples that have a similar composition to the sample of interest.

[0042] 6 illustrates an exemplary procedure for processing a spectral sequence using a trained neural network 612. The neural network 612 is a 2D neural network that operates on the two-dimensional data input from 602. The 2D neural network includes a 2D convolutional layer 604, a flattening layer 606, and a dense layer 608.

[0043] The input data is processed by 2D convolutional layers 604. These are the initial layers of the network, accepting M×N inputs (M and N represent the dimensions of the input data). The input data is a single spectral sequence, as shown in FIG. 4. In one embodiment, M is time and N is wavenumber. Each convolutional layer may use a set of learnable filters that are small spatially (along width and height) but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing dot products between the filter's entries and the input, generating a 2D activation map. These maps are then stacked along the depth dimension to generate the output volume.

[0044] The data output from the 2D convolutional layer 604 is input to a flattening layer 606. After passing through one or more 2D convolutional layers, the output is a 3D tensor. The 3D tensor is flattened into a 1D tensor before it can be passed to a dense layer 608. The flattening operation takes a 3D tensor and reshapes it into a 1D tensor by aligning the elements along one axis.

[0045] The 1D data output from the flattened layer is then input to a dense layer 608, also known as a fully connected layer. The dense layer connects every neuron in one layer to every neuron in the next layer. The first dense layer has an input size equal to the number of elements in the flattened tensor. The final dense layer has a size of N, which represents the desired output of the network. Each neuron in the dense layer calculates a dot product of the input with its weight, adds a bias, and then applies an activation function. In step 610, the final dense layer is output as the result of the 2D neural network.

[0046] In some embodiments, the spectral sequence may be a 2D vector of size 128x22, with 128 spectral bins and 22 time bins, that forms the input to a convolutional neural network (CNN) model. The first layer of the 2D CNN model may consist of a convolutional layer including 32 filters of size 3x2 with a rectified linear (ReLU) activation function, the second layer may consist of a convolutional layer including 32 filters of size 4x3 with a ReLU activation function, the third layer may consist of a convolutional layer including 64 filters of size 6x3 with a ReLU activation function, the fourth layer may consist of a flattening layer, the fifth layer may consist of a dense layer with 128 nodes with a ReLU activation function, and the last layer may consist of a dense layer with 128 nodes of dimensions the same as the spectral dimensions of the input. The model can be trained using an Adam learning schedule with a learning rate of 10^-4 and a loss function of mean squared error over 100 epochs of training with training data split into batches of 32 spectral sequences.

[0047] Thus, the 2D convolutional layer serves to extract one or more features from the input data, the flattening layer reshapes the 3D output of the convolutional layer into a 1D tensor, and the dense layer makes a final prediction based on one or more of these. The dimension (N) of the output of the trained neural network is the same as the dimension of the spectral sequence representing wavenumbers. The neural network 612 is trained based on the training data so that the parameters of the 2D convolutional layer 604 and the dense layer 608 are identified. By processing the spectral sequence using the trained neural network, the fluorescence signal can be removed while preserving the Raman signal in the detected spectral data received by the detector.

[0048] The trained NNs presented here are CNNs, but 2D NNs with other architectures such as encoder and decoder or autoencoder networks can be used.

[0049] Figure 7 shows the spectral sequence and the output of a trained 2D NN. The x-axis is the number of detector pixels, which can be converted to wavelength. The y-axis is the signal intensity in arbitrary units. The trained 2D NN in this example was generated by training the 2D CNN shown in Figure 6 using 64,000 simulated spectral sequences. The spectral sequence dimensions were 22 x 64, corresponding to 22 time points (y-axis in Figure 4) and 64 pixels (x-axis in Figure 4). Plot 704 is the simulated measured spectral sequence summed over time. The simulated measured spectral sequence was not used to train the 2D NN. Plot 706 is the 1D output (Raman spectrum) of the trained NN generated after processing the simulated measured spectral sequence and the theoretical Raman spectrum. The NN output closely matches the theoretical Raman spectrum 708. The baseline of plot 706 is a nearly flat line close to zero intensity, indicating that the fluorescent component was successfully removed from the spectral data.

[0050] The methods and systems provided herein provide a way to remove fluorescence from Raman signals in Raman spectroscopy. Compared to existing time gating, background subtraction, or classical fitting methods, using a trained 2D neural network can potentially outperform these methods. Unlike time gating, the neural network method can learn the details of the different time dependencies of Raman and fluorescence signals and use these dependencies to separate the signals. This can result in processed Raman spectra with higher fidelity. Furthermore, in addition to removing the fluorescence background, the trained 2D neural network can also remove time skew errors in the detected spectral data. Time skew errors are caused by different pixel time clocks in pixelated detectors. Time skew errors cannot be removed using other methods.

[0051] Example 1: A method for Raman spectroscopy, the method comprising: illuminating a signal location with a plurality of light pulses and acquiring photons from the sample, where acquiring the photons includes recording the arrival time of each of the photons; generating a spectral series based on the acquired photons, where the spectral series includes a plurality of spectra, each spectrum of the plurality of spectra constructed from photons having the same arrival time from a corresponding illumination with the light pulses; processing the spectral series with a trained 2D neural network; and generating a Raman spectrum from the trained 2D neural network.

[0052] Example 2: The method of example 1, wherein a portion of the plurality of spectra acquired at earlier arrival times includes both Raman and fluorescence signals, and a portion of the plurality of spectra acquired at later arrival times does not include a Raman signal.

[0053] Example 3: The method of any of Examples 1-2, wherein processing the spectral sequence with a trained 2D neural network and generating a Raman spectrum from the trained 2D neural network comprises removing a fluorescence signal from the spectral sequence and generating a Raman spectrum with the trained 2D neural network.

[0054] Example 4: The method of any of Examples 1-3, further comprising generating a trained 2D neural network by training the 2D neural network with training data comprising a plurality of training spectral sequences.

[0055] Example 5: The method of any of Examples 1-4, further comprising generating a training series of spectra based on pure Raman spectra and pure fluorescence spectra.

[0056] Example 6: The method of any of Examples 1-5, further comprising generating a training spectral sequence further based on one or more characteristics of the light pulses.

[0057] Example 7: The method of any of Examples 1-6, wherein the one or more characteristics of the light pulse include a temporal profile of the intensity of the light pulse.

[0058] Example 8: The method of any of Examples 1-7, wherein generating the trained 2D neural network further comprises first training the 2D neural network with a simulated training spectral sequence, and then training the 2D neural network with an experimentally obtained training spectral sequence.

[0059] Example 9: The method of any of Examples 1-8, further comprising determining the composition of the sample based on the Raman spectrum.

[0060] Example 10: The method of any of Examples 1-9, wherein determining the composition of the sample based on the Raman spectrum includes determining the composition using a second neural network.

[0061] Example 11: The method of any of Examples 1 to 10, wherein the trained 2D neural network is a convolutional neural network or an autoencoder.

[0062] Example 12: The method of any of Examples 1 to 11, wherein the spectral series is a 2D data set and the length of the Raman spectrum is the same as at least one dimension of the spectral series.

[0063] Example 13: The method of any of Examples 1-12, wherein acquiring photons from the sample includes acquiring photons via time-correlated single photon counting.

[0064] Example 14: The method described in any of Examples 1 to 3, further comprising obtaining a 2D neural network trained using training data and generating the training data based on the temporal profile of the intensity of the light pulse.

[0065] Example 15: The method of any of Examples 1-14, further comprising generating training data further based on pure Raman spectra and fluorescence lifetimes from a library.

[0066] Example 16: The method of any of Examples 1 to 15, further comprising obtaining a temporal profile of the intensity of the light pulse and retraining the trained 2D neural network based on the temporal profile.

[0067] Example 17: A method for Raman spectroscopy, the method comprising: receiving a spectral sequence including a plurality of spectra acquired at a sample location, each spectrum of the plurality of spectra corresponding to a different arrival time, the plurality of spectra being acquired by illuminating the sample location with a plurality of light pulses and acquiring photons from the sample, wherein acquiring the photons comprises recording the arrival time of each of the photons; processing the spectral sequence with a trained 2D neural network; and generating a Raman spectrum from the trained 2D neural network.

[0068] Example 18: A microscope system comprising: a detector; a pulsed laser for generating light pulses; a sample holder for positioning a sample; a processor; and a controller including a non-transitory memory for storing computer readable instructions, wherein by executing the instructions in the processor, the microscope system is configured to: laser irradiate a location of the sample with a plurality of light pulses via the pulsed laser; acquire photons from the sample via the detector, wherein acquiring the photons includes recording the arrival time of each of the photons from the corresponding irradiation with the light pulse; generate a spectral series based on the acquired photons, wherein the spectral series comprises a plurality of spectra, each spectrum of the plurality of spectra being constructed from photons having the same arrival time; process the spectral series using a trained 2D neural network; and generate a Raman spectrum from the trained 2D neural network.

[0069] Example 19: The microscope system of example 18, wherein the detector is a single-photon avalanche diode detector.

[0070] Example 20: The microscope system of example 18 or example 19, further comprising a scanner for directing light pulses to different sample locations on the sample.

Claims

1. 1. A method for Raman spectroscopy, comprising: illuminating a sample location with a plurality of light pulses and acquiring photons from the sample, wherein acquiring the photons includes recording an arrival time of each of the photons; generating a spectral sequence based on the acquired photons, the spectral sequence including a plurality of spectra, each spectrum of the plurality of spectra being constructed from photons having the same arrival time from a corresponding illumination of the light pulse; processing the spectral sequence using a trained 2D neural network; and generating a Raman spectrum from the trained 2D neural network.

2. 2. The method of claim 1, wherein a portion of the plurality of spectra acquired at earlier arrival times includes both Raman and fluorescence signals, and a portion of the plurality of spectra acquired at later arrival times does not include the Raman signal.

3. 3. The method of claim 2, wherein processing the spectral sequence with the trained 2D neural network and generating the Raman spectrum from the trained 2D neural network comprises removing the fluorescence signal from the spectral sequence and generating the Raman spectrum with the trained 2D neural network.

4. The method of claim 1 , further comprising generating the trained 2D neural network by training the 2D neural network with training data comprising a plurality of training spectral sequences.

5. The method of claim 4 , further comprising generating the training spectral sequence based on a pure Raman spectrum, a pure fluorescence spectrum, and one or more characteristics of the light pulses.

6. The method of claim 5 , wherein the one or more characteristics of the light pulse include a temporal profile of the intensity of the light pulse.

7. 5. The method of claim 4, wherein generating the trained 2D neural network further comprises first training the 2D neural network with a simulated training spectral sequence and then training the 2D neural network with an experimentally obtained training spectral sequence.

8. The method of any one of claims 1 to 7, further comprising determining a composition of the sample based on the Raman spectrum.

9. 9. The method of claim 8, wherein determining the composition of the sample based on the Raman spectrum comprises determining the composition using a second neural network.

10. The method of any one of claims 1 to 7, wherein the trained 2D neural network is a convolutional neural network or an autoencoder.

11. The method of any one of claims 1 to 7, wherein the spectral series is a 2D data set and the length of the Raman spectrum is the same as at least one dimension of the spectral series.

12. The method of any one of claims 1 to 7, wherein acquiring photons from the sample comprises acquiring the photons via time-correlated single photon counting.

13. 10. The method of claim 1, further comprising: obtaining the trained 2D neural network using training data; and generating the training data based on a temporal profile of the intensity of the light pulse.

14. 1. A microscope system, comprising: A detector; a pulsed laser for generating light pulses; a sample holder for positioning the sample; a controller including a processor and a non-transitory memory for storing computer readable instructions; By executing the computer readable instructions on the processor, the microscope system: irradiating the sample location with a plurality of light pulses via the pulsed laser and acquiring photons from the sample via the detector, wherein acquiring the photons includes recording the arrival time of each of the photons from a corresponding irradiation of the light pulse; generating a spectral sequence based on the acquired photons, the spectral sequence including a plurality of spectra, each spectrum of the plurality of spectra being constructed from photons having the same arrival time; processing the spectral sequence using a trained 2D neural network; generating a Raman spectrum from the trained 2D neural network.

15. The microscope system of claim 14 , wherein the detector is a single-photon avalanche diode detector.

Images