Dynamic assessment of cellular metabolism through optical imaging and artificial intelligence techniques

Abstract

Systems and methods are provided for predicting characteristics of one or more cells of a tissue sample based on intracellular dynamic activity information. An optical imaging device is used in conjunction with a high-speed camera to capture a set of images. The set of images are then processed to determine frequency data associated with one or more cells in the images. One or more machine learning (ML) / deep learning (DL) algorithm is applied to the set of images to predict characteristics of one or more cells in the tissue sample. In some embodiments, a semantic segmentation algorithm is used to identify intracellular structures within each cell and the ML / DL algorithms are trained to utilize a semantic map of each cell to predict the characteristics.

Description

DYNAMIC ASSESSMENT OF CELLULAR METABOLISMTHROUGH OPTICAL IMAGING AND ARTIFICIALINTELLIGENCE TECHNIQUESSTATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0001] This invention was made with Government support under Project Number Z01- HD000261 awarded by the National Institutes of Health. The Government has certain rights in this invention.CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Ser. No. 63 / 556,756, fded on February 22, 2024, which is hereby incorporated by reference herein.BACKGROUND

[0003] A technique for analyzing the intracellular dynamic activity of biological samples has valuable implications in fields such as biology and drug development. It also holds potential for practical applications in clinical trials, such as real-time determination of removal areas during cancer cell resection. Cytology research, including cell therapy and cell biopsy, is emerging as a promising technology for early diagnosis and treatment of serious diseases, such as cancer, neurodegenerative disease, and heart disease. In cancer treatment, for example, treatment is performed by inducing cancer cell death or making an incision to remove the cancer. If cancer cells are then detected after the treatment through a cell viability assessment, the process may be repeated to prevent metastasis and recurrence. Therefore, in order to improve the effectiveness of a cytology treatment / diagnostic method, a rapid and accurate method for examining cell viability, verifiable through measurements of intracellular dynamic activity, is required.

[0004] There are a number of current techniques for observing intracellular dynamic activity. One such technique is referred to as time-lapse microscopic observation. This approach involves monitoring the dynamic activity of cells based on structural changes, such as cell proliferation and division, over time. While this technique is relatively straightforward, the inherent translucency of biological samples, such as cells, poses challenges in achieving optimal visibilityand resolution through imaging techniques well-known in the art, such as bright-field microscopy. Another technique may be generally categorized as a fluorescence / dye-based assay, which offers greater visibility compared to the common microscopy imaging modality. Nevertheless, the use of fluorescence / dye-based methods might affect the long-term stability of the cells under observation. Moreover, the introduction of foreign substances, such as fluorescence / dye agents, can pose toxicity risks. Furthermore, the incorporation of foreign substances makes it challenging to reuse these test cells within the same sample over time. An alternative approach involves the use of electron microscopy or atomic force microscopy, requiring preliminary sample treatments like immobilization, staining, or coating to visualize the sample’s structure. It is important to note that these methods do not analyze living cells; instead, the methods “fix” samples for observation. Consequently, they are not suitable methods for conducting experiments over an extended period of time. All of the above techniques require biopsies of samples from a tissue for analysis and are time consuming to prepare and process.

[0005] A more recent technique has demonstrated the effectiveness of dynamic full-field optical coherence microscopy (DFFOCM) to perform quantitative measurement of cellular dynamic activities. This is an observational method based on optical interferometry that enables tomographic images of optical scattering media to be observed in a real-time, label-free, non- invasive, and non-destructive manner with an axial and transverse resolution of several micrometers. These techniques monitor the frequency and magnitude of cellular activity to predict a label of cellular status (e g., viable / non-viable). By utilizing a dye-free approach and a low-power consistent light source, these methods enable the examination of samples over extended periods. This allows for the thorough evaluation of the efficacy of various substances, such as specific drug candidates. However, even this technique may be improved to increase the accuracy of the prediction. Thus, there is a need for addressing these issues and / or other issues associated with the prior art.SUMMARY

[0006] Embodiments of the present disclosure relate to techniques for predicting intracellular dynamic activity using artificial intelligence models and dynamic optical imaging techniques.

[0007] In accordance with a first aspect of the present disclosure, a system for observing intracellular dynamic activity of a tissue sample is provided. The system includes: an opticaldevice including a high-speed camera for capturing a set of images of the tissue sample; and a controller in communication with the high-speed camera. The controller comprises at least one processor to: process the set of images using at least one machine learning (ML) / deep learning (DL) algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

[0008] In at least one embodiment of the first aspect, the optical device comprises an interferometry imaging device. In at least one other embodiment of the first aspect, the optical device comprises an optical microscope.

[0009] In at least one embodiment of the first aspect, the processing the set of images using the at least one ML / DL algorithm comprises: dividing the set of images into one or more segments, each segment of the one or more images including AT images; generating, for each segment in the one or more segments, frequency data for each pixel of the AT images in the segment; and concatenating the set of images and the frequency data for processing by the at least one ML / DL algorithm.

[0010] In at least one embodiment of the first aspect, the processing the set of images using the at least one ML / DL algorithm further comprises: processing at least one image in each segment by a semantic segmentation model to generate a semantic map that indicates at least two different intracellular structures in each of one or more cells; and concatenating the semantic map with at least one of the set of images or the frequency data for processing by the at least one ML / DL algorithm.

[0011] In at least one embodiment of the first aspect, the at least one ML / DL algorithm comprises a convolutional neural network (CNN) that is trained to generate at least one output image and / or quantitative value that indicates, for each cell of one or more cells depicted in the set of images, a level of intracellular dynamic activity for the cell.

[0012] In at least one embodiment of the first aspect, the semantic segmentation model comprises a CNN.

[0013] In at least one embodiment of the first aspect, a sampling frequency of the high-speed camera is at least 50 frames per second.

[0014] In at least one embodiment of the first aspect, generating the frequency data comprises, for each pixel location in the set of images: processing M pixel values for the pixel location within each segment using a Fast Fourier Transform (FFT) algorithm to generate afrequency spectrum for the pixel location; and calculating a mean frequency parameter in accordance with the following equation based on the frequency spectrum:where is the size of the segment; is the frequency of the frequency spectrum at bin z of M; and Pi is the power density of the frequency spectrum at bin i ofM.

[0015] In at least one embodiment of the first aspect, generating the frequency data further comprises, for each pixel location in the set of images: calculating a magnitude parameter in accordance with the following equation based on the frequency spectrum: mag = ^=1P .

[0016] In at least one embodiment of the first aspect, the tissue sample and optical device are enclosed within an environmental enclosure that includes at least one of temperature control, humidity control, or atmospheric control.

[0017] In at least one embodiment of the first aspect, the tissue sample is located within a sample holder that includes at least one temperature sensor and a gas port for delivering a gas proximate the sample.

[0018] In at least one embodiment of the first aspect, a feedback signal from the at least one temperature sensor is used to adjust a temperature of the gas provided to the tissue sample to heat or cool the tissue sample to a desired temperature.

[0019] In accordance with a second aspect of the present disclosure, a method for classifying intracellular dynamic activity of a tissue sample is provided. The method includes: capturing, using a high-speed camera associated with an optical device, a set of images of the tissue sample; and processing the set of images using at least one ML / DL algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

[0020] In at least one embodiment of the second aspect, the processing the set of images using the at least one ML / DL algorithm comprises: dividing the set of images into one or more segments, each segment of the one or more images including M images; generating, for each segment in the one or more segments, frequency data for each pixel of the M images in the segment; and concatenating the set of images and the frequency data for processing by the at least one ML / DL algorithm.

[0021] In at least one embodiment of the second aspect, the processing the set of images using the at least one ML / DL algorithm further comprises: processing at least one image in each segment by a semantic segmentation model to generate a semantic map that indicates at least two different intracellular structures in each of one or more cells; and concatenating the semantic map with at least one of the set of images or the frequency data for processing by at least one additional ML / DL algorithm.

[0022] In at least one embodiment of the second aspect, the at least one ML / DL algorithm comprises a CNN that is trained to generate at least one output image and / or quantitative value that indicates, for each cell of one or more cells depicted in the set of images, a level of intracellular dynamic activity for the cell.

[0023] In at least one embodiment of the second aspect, generating the frequency data comprises, for each pixel location in the set of images comprises: processing M pixel values for the pixel location within each segment using a FFT algorithm to generate a frequency spectrum for the pixel location; and calculating a mean frequency parameter in accordance with the following equation based on the frequency spectrum: / fmean - ZJ I=11P iJf i! / ZJ J=11P- where is the size of the segment; is the frequency of the frequency spectrum at bin iand Pj is the power density of the frequency spectrum at bin i oiM.

[0024] In at least one embodiment of the second aspect, generating the frequency data further comprises, for each pixel location in the set of images: calculating a magnitude parameter in accordance with the following equation based on the frequency spectrum:

[0025] In accordance with a third aspect of the present disclosure, a remote computing device in communication with an optical device including a high-speed camera for capturing a set of images of a tissue sample is provided. The remote computing device comprises a memory for storing the set of images and parameters for at least one ML / DL algorithm; and at least one processor in communication with the memory. The at least one processor is configured to: process the set of images using the at least one ML / DL algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

[0026] In at least one embodiment of the third aspect, the processing the set of images using the at least one ML / DL algorithm comprises: dividing the set of images into one or more segments, each segment of the one or more images including M images; generating, for each segment in the one or more segments, frequency data for each pixel of the M images in the segment; and concatenating the set of images and the frequency data for processing by the at least one ML / DL algorithm.

[0027] In at least one embodiment of the third aspect, the processing the set of images using the at least one ML / DL algorithm further comprises: processing at least one image in each segment by a semantic segmentation model to generate a semantic map that indicates at least two different intracellular structures in each of one or more cells; and concatenating the semantic map with at least one of the set of images or the frequency data for processing by at least one additional ML / DL model.

[0028] In at least one embodiment of the third aspect, the at least one ML / DL algorithm comprises a CNN that is trained to generate at least one output image and / or quantitative value that indicates, for each cell of one or more cells depicted in the set of images, a level of intracellular dynamic activity for the cell.

[0029] In at least one embodiment of the third aspect, generating the frequency data comprises, for each pixel location in the set of images comprises: processing M pixel values for the pixel location within each segment using a FFT algorithm to generate a frequency spectrum for the pixel location; and calculating a mean frequency parameter in accordance with the following equation based on the frequency spectrum:where M is the size of the segment; ftis the frequency of the frequency spectrum at bin z of A / ; and Pi is the power density of the frequency spectrum at bin i ofM.

[0030] In at least one embodiment of the third aspect, generating the frequency data further comprises, for each pixel location in the set of images: calculating a magnitude parameter in accordance with the following equation based on the frequency spectrum: mag = ^=1P .BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present systems and methods for classifying intracellular dynamic activity of one or more cells in a tissue sample are described in detail below with reference to the attached drawing figures, wherein:

[0032] FIG. 1A illustrates an optical device, in accordance with some embodiments;

[0033] FIG. IB illustrates an optical device, in accordance with some other embodiments;

[0034] FIG. 1C illustrates an optical device, in accordance with some embodiments;

[0035] FIG. ID is an example of a hand-held probe, in accordance with some embodiments;

[0036] FIG. IE is an example structure of a lensed optical fiber (LOF) for an optical probe, in accordance with some embodiments;

[0037] FIG. IF shows example LOFs, in accordance with some embodiments;

[0038] FIG. 2 illustrates a system for classifying a sample, in accordance with some embodiments;

[0039] FIGS. 3A & 3B illustrate a sample holder, in accordance with at least one embodiment;

[0040] FIGS. 4A-4D illustrate the conceptual operation of the system, in accordance with some embodiments;

[0041] FIG. 5 is an example plot of cell viability assessment using the cell dynamic activity observation techniques described above, in accordance with an embodiment;

[0042] FIG. 6 shows a comparison of dynamic activity for cells in seven tissue samples, where oxygen concentration is adjusted for each sample, in accordance with an embodiment;

[0043] FIG. 7 is an output image showing the classification of one or more cells in a tissue sample that indicates a predicted characteristic of each cell, in accordance with some embodiments;

[0044] FIGS. 8A-8C shows a ML / DL framework for predicting a characteristic of cells in a tissue sample based on intracellular dynamic activity, in accordance with some embodiments;

[0045] FIG. 9 is a flowchart of a method for predicting characteristics of a tissue sample using ML / DL techniques, in accordance with some embodiments; and

[0046] FIG. 10 illustrates an exemplary computer system, in accordance with some embodiments.DETAILED DESCRIPTION

[0047] The observation of dynamic activity in cells or tissue is critical for detecting changes in their metabolic activity and viability, with significant implications for modem biology, drug development, toxicology, and cell biology research. Monitoring cell dynamic activity and accessing viability are fundamental aspects of contemporary biological research. Accurately determining the metabolic variations or viability of cells is essential for evaluating the effects of different treatments or environmental factors on cell health and selecting viable cells for further experimentation. Therefore, these techniques play a crucial role in advancing our understanding of cellular biology and developing new therapies.

[0048] Observing cells in real-time can provide insight into the complex signaling pathways and cellular interactions that drive physiological and pathological processes. Traditional methods used for observing the intracellular activity or viability of cells include fluorescence imaging or dye analysis (e g., as Trypan blue), calcium imaging, patch-clamp electrophysiology, RNA sequencing, etc. However, these methods have limitations that hinder their efficacy. For instance, fluorescence imaging, dye analysis, and calcium imaging techniques require the introduction of foreign substances into the cells, which can alter the intrinsic properties of the cell and potentially produce false results. Additionally, some of these substances could be harmful to the cellular environment, further complicating the research process. Patch-clamp electrophysiology, on the other hand, is a highly precise experimental system that allows researchers to study the electrical activity of individual cells. However, this technique is invasive and requires specialized knowledge and equipment, making it challenging for researchers to implement. Similarly, RNA sequencing requires extensive pre- and post-production processes that can limit the real-time observation of cellular activity. These processes can also introduce variability, making it difficult to compare results across experiments.

[0049] Systems and methods are disclosed for classifying intracellular dynamic activity of cellular metabolism using high-speed imaging techniques. In at least one embodiment, an interferometry imaging system is coupled with a high-speed camera to capture a set of images of a tissue sample. The set of images may be processed to generate frequency data that indicates parameters related to the dynamic activity of the cells visible in the tissue sample within the set of images. Frequency data can be assessed on a per-pixel level. In at least some embodiments, a first ML / DL model referred to as a semantic segmentation algorithm is implemented and trainedto identify different intracellular structures within each visible cell. A second ML / DL model is then trained to use at least one of the set of high-speed images, the per-pixel frequency data, and a semantic map to classify an intracellular dynamic activity level of one or more of the cells visible in the set of images to predict characteristics of the tissue samples, such as whether each of the cells is viable or non-viable, a induced stress, intracellular dynamic activity changes, metabolic fractionation, or the like. The output of the second ML / DL model can be an image and / or a quantitative value that indicates the measured characteristics of each of the one or more cells. Evaluation of the output of the second ML / DL model can then be performed to determine whether the tissue sample is suitable for further study or use within a clinical setting or trial.

[0050] FIG. 1A illustrates an optical device 100, in accordance with some embodiments. The device 100 may be referred to as an optical interferometer or, specifically, a Linnik interferometry system (similar to Michelson interferometry). Interferometry uses the interference of superimposed waves to extract information about a sample 102. Interferometry is useful for measuring microscopic displacements, refractive index changes, and / or surface irregularities of the sample 102. By comparing two images of the same beam path-length taken at two points in time, a difference in pixel intensity can provide information about relative changes in any of the parameters listed above.

[0051] As shown in FIG. 1 A, the device 100 includes a light source 116 and a beam splitter 114 that splits the light from the light source into two separate paths. In a first path, light from the light source 116 is directed toward an objective lens 122 and then a reflector 124. The light is reflected back through the lens 122 and onto the beam splitter 114. In a second path, light from the light source 116 is directed toward an objective lens 118 and then the sample 102, which is held in a sample holder 120. Light is reflected and / or partially transmitted through the sample 102 and reflected back through the objective lens 118 and onto the beam splitter 114. Light from both paths is then combined by the beam splitter 114 and directed onto the lens 112 and a highspeed camera 110, which can sample the light to generate an observed image 104.

[0052] Incorporating the high-speed camera 110 for capturing a set of high-speed images over a particular time period with the interferometer device 100 is useful for determining a classification of cellular metabolism. Comparing the portions of the images corresponding to a particular cell over time can provide a signal that reflects both a frequency and magnitude of dynamic cellular activity. Furthermore, although the techniques described herein may bedescribed in relation to the interferometer 100 described above, the techniques may also be implemented with other types of interferometers, such as a Mach-Zehnder interferometry system.

[0053] In an embodiment, the high-speed camera 110 is a digital camera including an image sensor (e.g., CMOS image sensor) that is capable of sampling at a sampling rate of at least 50 frames per second, preferably at least 150 frames per second and, in some embodiments, at least 75 frames per second. In some cases, it has been shown that intracellular dynamic activity may exhibit internal movement having frequencies up to around 25 Hz. Thus, a minimum sampling frequency of at least twice the expected dynamic activity frequency is preferred.

[0054] FIG. IB illustrates an optical device 150, in accordance with other embodiments. The device 150 may be referred to as a microscope, optical microscope, or optical transmission microscope (e.g., when the tissue sample is at least partially transparent). Unlike the device 100, the device 150 does not measure interference patterns based on different path lengths, but instead measures direct light absorbance / reflectance / transmission characteristics of a sample. The device 150 is suitable for acquiring the two-dimensional structure of an at least partially transparent sample and analyzing the internal scattering. The device 150 is also simpler than the device 100, which may be sensitive to noise caused by, e.g., vibration and / or temperature changes. Thus, device 150 may be operated more stably in a smaller form factor or harsher environment.

[0055] As shown in FIG. IB, the device 150 also includes a light source 116 and beam splitter 114. However, the beam splitter 114 in FIG. IB directs light from the light source 116 only toward the sample 102 through the objective lens 118. Light is reflected and / or partially transmitted through the sample 102 and reflected back through the objective lens 118 to the beam splitter 114. A portion of that light is transmitted through the beam splitter 114 through the lens 112 and to the high-speed camera 110 to generate the observed image 104.

[0056] It will be appreciated that, in another embodiment, the light source 116 may also be positioned below the sample so that light is transmitted up through the sample from below. In such an embodiment, the beam splitter 114 may be omitted from the device 150. Although the device 150 is shown using a single lens 112 and single camera 110, in some embodiments, the device 150 may be a stereo microscope that uses a pair of offset lenses and a pair of cameras to generate stereo images of the sample.

[0057] FIG. 1C illustrates an optical device 150, in accordance with other embodiments. The optical device 150 shown in FIG. 1C is similar to the optical device 150 shown in FIG. IB, but abeam shape transformer 180 is included in the light path between the beam splitter 114 and the objective lens 118. The beam shape transformer 180 is configured to control the amount of light reflected from the lens surface. The beam shape transformer 180 can be a unit including several lenses that plays a role in altering the shape of light by spreading or focusing the light.

[0058] In some embodiments, the beam shape transformer 180 and the objective lens 118 can be housed in a hand-held probe 190. The hand-held probe 190 can then be applied directly to a live person, without the need for a sample, sample holder, or specialized environmental enclosure for imaging samples (as described below in FIG. 2). Using a hand-held probe 190 offers significant potential for various in-vivo clinical applications. It enables high-resolution, real-time imaging of tissue changes, making it useful for monitoring wound healing, tracking tumor growth, and assessing organ transplant rejection. In oncology, the hand-held probe 190 can monitor tumor behavior and response to treatment, while in cardiovascular care, it provides detailed imaging of blood vessels and blood flow, aiding in early detection of vascular diseases and improving surgical planning. It is also valuable for cellular imaging in cancer treatment and regenerative medicine, tracking cell behavior at the sub-cellular level. Moreover, the hand-held probe 190 can support minimally invasive procedures by providing real-time feedback during surgeries, improving precision, and reducing risks to surrounding healthy tissue.

[0059] FIG. ID is an example of a hand-held probe 190, according to one embodiment. The hand-held probe 190 may be implemented with a galvanometer 191, a magnifier lens (not shown), relay lens 192, and charge-coupled device (CCD) camera 193. The hand-held probe 190 may be designed to be attached to long optical fiber and electrical wires, allowing a clinician to diagnose patients while keeping them comfortable. Light from the beam splitter 114 is propagated from the galvanometer 191, relay lens 192, focal lens 194, and specula 195. The light focuses on and then scans the sample.

[0060] In some embodiments, the beam shape transformer 180 in the hand-held probe 190 provides tunability of the light beam. In some implementations, the beam shape transformer 180 may be based on a lensed optical fiber (LOF). An example structure of an LOF for an optical probe is shown in FIG. IE. The input beam from a single-mode fiber (SMF) is expanded before being focused by the lens to provide an effective light-gathering power to the fiber lens. As the core of SMF is very small (<10 pm), without being expanded, the beam cannot be focused at a reasonable working distance in general. In order to fabricate the beam expansion region, a shortpiece of coreless silica fiber (CSF) is spliced in between, where the beam is expanded according to the numerical aperture (NA) of the SMF.

[0061] In FIG. IE, Lcis a length of coreless silica fiber (CSF), A / is a working distance or focal length, Lb is a focal length of a reflected beam, R is a radius of lens curvature, wo is half of the mode-field diameter (MFD) of single-mode fiber (SMF), w is a beam spot at each interesting position, and n is a refractive index of the medium.

[0062] The working distance A and the beam waist woi are determined by the length of the beam expansion region L and the radius of lens curvature R. However, at the lens surface, a part of the beam is reflected and could be back-coupled to the lead-in SMF. For this case, the diameter and the propagation angle of the reflected beam at the SMF is important for calculating the back- coupling efficiency.

[0063] In some embodiments, a Michelson-type optical interferometer system has a separate reference arm. Therefore, it is desirable to minimize the back-coupling from the lens surface of the LOF probe to minimize light loss. Anti-reflection coating on the lens surface is one solution, since it removes the Fresnel reflection itself. However, from a practical point of view, coating on the fiber lens with a long piece of lead-in fiber is neither easy nor cost-effective. Therefore, another approach to handle the inevitable Fresnel reflection is to minimize the mode-coupling to the lead-in SMF by adjusting the structure parameters of the LOF.

[0064] FIG. IF shows three types of typical LOFs. The LOF with a sufficiently large R (almost flat surface) is often a good choice, owing to the large beam size at the SMF. However, as shown in (a) in FIG. IF, its performance as a focuser is low due to the low lens power. FIG. IF at (c) shows the opposite case, whereby using a small R and a short Lc we can have a good focuser. Moreover, in this case, the size of the beam reflected at the SMF is rather small, so that it gives fairly high back-coupling to the SMF. In one implementation, the LOF optimized for a Michelson-type probe is shown in (b) in FIG. IF. The probe has a long Lcand a large R, so that it can act as a good focuser but has a low back-coupling efficiency.

[0065] In some embodiments, the shape of the light output from the beam shape transformer 180 is tunable by applying electric current to the adjust the parameters of the optical lenses within the beam shape transformer 180.

[0066] FIG. 2 illustrates a system 200 for classifying a sample, in accordance with some embodiments. As shown in FIG. 2, the system 200 includes an environmental enclosure 210 thathouses an optical imaging device 100 (or alternatively optical imaging device 150) and a sample in a sample holder. The environmental enclosure 210 may comprise a number of walls, which may be, e.g., a transparent Lexan, glass, acrylic, or other material suitable for use. The walls can be supported by a frame, which may be, e.g., constructed from an aluminum extrusion, sheet metal, molded plastic, or other suitable materials. In an embodiment, the materials may be selected to be inert or non-corrosive when in contact with any gases flowed into or out of the environmental enclosure 210.

[0067] The environmental enclosure 210 may have one or more doors to provide access to the sample and / or sample holder. In some embodiments, a number of samples may be stored in the enclosure, and a robotic or other automated system may be provided to move a selected sample under the objective lens of the optical image device 100 for analysis. The robotic system can be controlled to choose a sample within the enclosure for analysis without opening the enclosure and exposing the contents of the enclosure to the external atmosphere. In some embodiments, the environmental enclosure 210 may include multiple compartments to, e.g., isolate samples in one compartment from samples or an environment of another enclosure. In this manner, the environmental enclosure 210 may include a first chamber that can act as an airlock to help isolate an external environment from an internal environment in a second chamber of the enclosure. For example, a sample from outside the enclosure can be placed in the first chamber and sealed therein by, e.g., closing an access door. Gas in the first chamber can be at least partially evacuated using, e.g., a negative vacuum pressure. Gas of the desired environment can then be flowed into the first chamber to normalize the environment of the first chamber with an environment of a second chamber. Finally, an internal access door can be opened and the sample can be automatically moved from the first chamber to the second chamber for storage, analysis, and / or further processing / testing.

[0068] In an embodiment, the environmental enclosure 210 can implement one or more of a temperature control, humidity control, or environmental control (e.g., gas regulation) within the enclosed space of the environmental enclosure 210. A temperature control unit 220 can be provided to regulate a temperature in the enclosed space. In an embodiment, a heater element (e.g., a resistive heating element, radiative heating element, etc.) can be placed in the environmental enclosure 210 or contacting one or more outside walls of the environmental enclosure 210 to maintain a set temperature within the environmental enclosure 210. The heatingelement may be controlled based on a temperature sensor signal 202, which provides feedback about the temperature of a sample in the sample holder. For example, a resistive heating element can be placed proximate a fan to heat up the air in the environmental enclosure 210 to a desired temperature. The temperature control unit 220 may receive the temperature sensor signal 202 from one or more temperature sensors (not explicitly shown in FIG. 2) placed in the environmental enclosure 210 and / or placed in the sample holder. The temperature sensor input signals 202 can be used to turn on or turn off the fan and / or heating element in order to raise or lower the temperature in the environment surrounding one or more tissue samples.

[0069] In another embodiment, a sample holder may include one or more temperature sensors for detecting the temperature of one or more of the sample, the sample holder (in one or more locations proximate the sample), and / or the gas proximate the sample. In one exemplary embodiment, the sample holder includes an outlet port for flowing gas over the sample. The gas can be provided via a gas line 204 (e.g., a distensible plastic or silicone tube) connected to the outlet port of the sample holder on one end and connected to an external gas control unit 230 on the other end. The gas control unit 230 can regulate a flow volume and / or flow pressure of gas from a gas canister 240 to the sample holder and may comprise one of a regulator, needle valve, solenoid valve, or the like for controlling pressure and / or flow of gas. In one embodiment, the gas flowing out of the gas control unit 230 may be passed through a heating element or heat exchanger within a temperature control unit 220. The heating element or heat exchanger heats and / or cools the gas flowing through the temperature control unit 220 prior to the gas being flowed onto the sample by exiting the outlet port of the sample holder. In some embodiments, the outlet port of the sample holder can include a nozzle or diffuser to direct and / or control the flow of gas around the sample from the outlet port.

[0070] In an embodiment, the sample holder can include at least two temperature sensors. A first temperature sensor can be placed on or in a surface of the sample holder proximate the sample. For example, the first temperature sensor can be molded into a top surface of the sample holder that is designed to hold or otherwise support a tissue sample. The first temperature sensor is used to measure a temperature of the sample. A second temperature sensor can be placed proximate the outlet port of the sample holder and is designed to measure an outlet temperature of the gas flowing over the sample. In this manner, the temperature control unit 220 can monitorboth the temperature of the sample and a temperature of the gas being used to heat (or cool) the sample.

[0071] In other embodiments, the sample holder can include additional sensors, such as additional temperature sensors, gas sensors (e.g., CO2 sensors, O2 sensors, etc.), liquid sensors, humidity sensors, or the like.

[0072] The environmental enclosure 210 may also be sealed so that gas flow into or out of the environmental enclosure 210 can be controlled through one or more inlet or outlet ports. In other embodiments, the enclosure may not be sealed, but a positive pressure (e.g., greater than atmospheric pressure) may be maintained inside the environmental enclosure 210 so that gas may flow out of the environmental enclosure 210 but gas from the external environment cannot flow into the environmental enclosure 210.

[0073] It will be appreciated that although FIG. 2 only illustrates a single gas line 204 and a single temperature sensor signal 202, the system 200 may implement any number of individual gas lines, manifolds, solenoid valves, pressure regulators, heating elements / heat exchangers, and / or temperature sensors. For example, each of a plurality of sample holders may include one or more temperature sensors to measure a temperature corresponding to a particular sample. Each sample holder may also be connected to a separate gas line used to flow different gases at the same or different flow rates over each of the samples. Each gas line may also flow through a separate heat exchanger to separate regulate a temperature of each gas flowed over each particular sample. Finally, in addition to the fine scale temperature and gas control, a temperature of the overall environment within the environmental enclosure 210 can be separately regulated using a separate one or more heating elements / fans located in the enclosure but not associated with any particular sample holder.

[0074] In an embodiment, a control system 250 may be provided to control operation of the temperature control unit 220 and / or gas control unit 230. The temperature control unit 220, the gas control unit 230, and / or the control system 250 can include one or more processors, logic, circuitry, or other means for controlling the temperature and flow of gas into or out of the environmental enclosure 210. In an embodiment, the control system 250 may be connected to, through either a wired or wireless interface, a remote computing system 260. The remote computing system can provide a display and a user interface (e.g., a graphic user interface (GUI)) for controlling operation of the system 200. In other embodiments, the control system250 is included within the remote computing system 260 and can be connected to the temperature control unit 220 and / or the gas control unit 230 via a wired or wireless interface. In an embodiment, the components of the system may each be connected to a network (e.g., a wireless local area network (WLAN)) and can be configured to share data over the network. For example, the temperature control unit 220 and / or the control system 250, located external to the environmental enclosure 210, can be configured to communicate wirelessly with a transceiver in a sample holder to read out measurement data for one or more temperature sensors in the sample holder. The wireless connection can either be indirect, e.g., through an access point or other wireless hub, or direct, e.g., through the use of Near Field Communication (NFC), Bluetooth, or some other point-to-point wireless communications protocol.

[0075] FIGS. 3A & 3B illustrate a sample holder 300, in accordance with at least one embodiment. As shown in FIG. 3A, the sample holder 300 is a substantially cylindrical assembly that includes a number of parts that fit together. A base of the sample holder 300 is a sample container 302 which may be a cup having a recess formed in a top surface for holding a sample 102. The sample container 302 may be made of metal, plastic, or any other suitable material for holding the sample tissue. A top outer edge of the sample container 302 may have threads formed thereon to be screwed into a media container 306 that is disposed on top of the sample container 302. In some embodiments, a window 304 (e.g., a circular piece of glass or plastic) may be placed between the sample container 302 and the media container 306, and sealed with a gasket 314 disposed in the media container 306 and made of, e.g., a rubber material.

[0076] In an embodiment, the media container 306 includes at least one gas port 308, which may be, e.g., a barbed connection for connecting to a distensible tubing used as a gas line for providing gas to the media container. Although not shown explicitly in FIG. 3A, the gas port 308 may be connected to a chamber or manifold formed inside the media container 306 that is fluidly connected to the recess in the sample container 302 when the sample holder 300 is assembled. In this manner, gas from a gas line connected to the gas port 308 can be flowed over the sample 102, thereby controlling an atmospheric environment surrounding the sample 102.

[0077] The media container 306 and / or the sample container 302 may also include connections for one or more temperature sensors, such as temperature feedback sensor 310 and / or temperature sensor 312. In one embodiment, the temperature feedback sensor 310 provides a temperature measurement to the temperature control unit 220 in order to cause thetemperature control unit 220 to adjust a temperature of the gas being provided to the sample holder 300. The temperature sensor 312 may also take a separate temperature measurement, which may be provided directly to the control unit 250 and / or the temperature control unit 220. In at least one embodiment, the temperature sensor 312 may be incorporated into the sample container 302 to measure a temperature of the sample 102, and the temperature feedback sensor 310 may be incorporated into the media container 306 to measure a temperature of the gas input to the sample holder 300.

[0078] Although only shown with a single gas port, in some other embodiments, additional ports for gases and / or liquids may be provided on the media container 306 to provide separate gas inlet ports or liquid inlet ports. Liquid inlet ports may be used, e.g., to deliver certain drug candidates to the sample under observation. It will also be appreciated that, although not shown, channels or some other bypass mechanism may be incorporated into the sample container and / or media container so that gas and / or liquid may flow from the media container 306 to the recess in the sample container 302. In other words, the window 304 and seal thereof can be designed in a way to allow for gas or liquid to flow between the media container 306 and / or the sample container 302 and does not create a completely sealed environment in the recess of the sample container 302.

[0079] Although not shown explicitly, sample container may also include an outlet port to allow gas to escape from the recess in the sample container 302. The outlet port can include a check valve or the like to prevent flow of gas into the recess via the outlet port.

[0080] FIG. 3B shows a side view of the exploded assembly. The stack of parts includes at least the sample container 302, the window304, and the media container 306.

[0081] FIGS. 4A-4D illustrate the conceptual operation of the system 200, in accordance with some embodiments. Images captured by the optical imaging device 100 or optical imaging device 150 using the high-speed camera 110 can be used to determine the dynamic activity of biological tissue samples containing one or more cells. In one embodiment, the sampling rate of the camera 110 should be at least twice a frequency of typical cellular activity rates. As shown in FIG. 4A, a frame 410 captured by the camera 110 shows a number of individual cells captured using a dynamic full-field optical interferometry imaging system, which may include an optical device such as optical device 100 as described in FIG. 1A or optical device 150 as described in FIG. IB and FIG. 1C. Intracellular motion is quantitatively analyzed through frequency analysis(e.g., using a Fast Fourier Transform (FFT) or Welch’s method) of each pixel location in image based on a set of images captured over a time period by the high-speed camera 110. For example, the camera 110 may capture, e.g., 1000 images over the course of a number of seconds. The images can then me arranged in a stack of images.

[0082] As shown in FIG. 4B, the stack of images 420 is divided into a number of segments 422 of width M. In other words, each segment 422 comprises M images captured sequentially by the camera in the stack of images 420. The parameter M may be selected appropriately to provide a sufficient range of frequency bins for the output of the frequency analysis. In certain embodiments, may be selected at 100 images; however, alternative segment lengths are also encompassed by the present disclosure. The choice of M may be contingent upon the structural resolution required for observing the target object. A smaller M is capable of capturing minute intracellular movements, albeit with a compromise in overall resolution. Conversely, a larger M faces challenges in capturing subtle intracellular movements, but excels in obtaining a clear representation of the overall structure. In certain embodiments, a single segment 422 can be designed with a default overlap of 50% for each segment within the stack of images 420. However, analysis of the images can be conducted without overlap to detect subtle intracellular movements. This approach holds the potential for achieving a more precise observation. The superposition of individual segments 422 is carried out to achieve several objectives, including noise reduction, signal smoothing, preservation of important signal features, mitigation of edge effects, and enhancement of overall accuracy. For example, if A / =100, each segment 422 may be shifted by 25 or 50 images in the time sequence of images. In yet other embodiments, segments 422 can be selected from the subset of images in accordance with different sampling frequencies. For example, a first segment 422 can comprise 100 images at a first sampling frequency, a second segment 422 can comprise 100 images at a second sampling frequency that is half of the first sampling frequency, a third segment 422 can comprise 100 images at a third sampling frequency that is half of the second sampling frequency and a quarter of the first sampling frequency, and so forth. Once the segments 422 of images are defined, the frequency analysis is performed to generate, for each pixel 424 in the image, frequency data that indicates intracellular activity for a cell covered by the pixel.

[0083] Each image can be semantically segmented through a semantic segmentation algorithm to identify the individual cells in the image. The semantic segmentation algorithm canbe selected from one of a variety of algorithms known in the art including, but not limited to, edge detection algorithms and machine learning algorithms. In one embodiment, each identified cell can be indicated using a bounding box (defined as upper left and lower right pixel coordinates in the image, for example) that encloses all pixels in the image that are included in the cell. In some embodiments, a mask can also be generated that indicates which pixels within the bounding box are associated with a particular cell. This may be utilized when multiple cells may be spaced very close together such that activity in a neighboring cell that falls within the bounding box of a different cell does not contribute to a calculation that defines a level of dynamic activity of the other cell.

[0084] In an embodiment, a first image in each segment 422 is used to identify the bounding boxes and / or masks for each cell of the one or more cells, and then those bounding boxes and / or masks are used for all images in the segment 422. In other embodiments, the bounding boxes and / or masks may be adjusted for each image in the segment 422 to account for any cell motion between images in the segment 422. In some embodiments, bounding boxes and / or masks in one segment 422 may be correlated with bounding boxes and masks in other segments 422 to identify the same cell across different segments 422 in the set of images 420. This can be useful if there is any vibration in the system during image capture such that the location of the cells shifts in the image frame across the set of images 420 over time during the time period.

[0085] In an embodiment, observation of a tissue sample using the interferometry based imaging system (e.g., optical device 100) is performed to obtain dynamic activity data for each cell within the field of view. In an embodiment, the frequency data calculated from the set of images 420 includes a major parameter referred to as mean frequency, fmean, which is calculated for each pixel of a cell as follows:where M is the size of the segment (which is also the size of the FFT); ft is the frequency of the spectrum at bin i of M; and Pt is the power density of the spectrum at bin z of M. Again, this is a per cell, per pixel 424 calculation, where each cell is associated with more than one pixel 424. For example, a particular cell might overlap hundreds or thousands of pixels 424 in a single 1+ megapixel image. In other embodiments, other methods for calculating frequency data may be used that modify the combination of frequency and power spectrum information as shown in Eq. 1.

[0086] In addition to the mean frequency parameter, a magnitude parameter, mag, can also be calculated for each pixel. The mag parameter is defined as: mag = ^=1Ph(Eq. 2) where mag is the sum of the power density over the set of frequency bins. Like Eq. 1 above, this calculation is a per cell, per pixel calculation. In other embodiments, other methods for calculating magnitude data may be used that modify the combination of power spectrum information as shown in Eq. 2.

[0087] An image 430 of the structural elements of a single cell is shown in FIG. 4C, and a corresponding image 440 showing the dynamic activity of the cell is shown in FIG. 4D. The image 430 is selected from at least a portion of one image in the set of images 410. The image 440 can be generated in accordance with the frequency data calculated for each pixel of the cell.

[0088] FIG. 5 is an example plot of cell viability assessment using the cell dynamic activity observation techniques described above, in accordance with an embodiment. The plot charts the mean frequency parameter fmeanon the x-axis and the magnitude parameter on the y-axis. It is apparent from the division of viable (e.g., live cells) and non-viable (e.g., dead cells) cells that viable cells exhibit lower frequency dynamic activity with higher magnitude and non-viable cells exhibit higher frequency dynamic activity at lower magnitudes. Thus, it can be shown that dynamic activity information associated with each cell can be used to predict whether a cell is viable. However, it can also be shown that looking at dynamic activity associated with different structures within a single cell may provide additional information about viability when compared to simply taking an average dynamic activity level over the whole cell. For example, by focusing on the dynamic activity of specific structures, such as mitochondria, nucleus, chloroplasts, or the like, may provide additional insight into whether a cell is viable or non-viable.

[0089] FIG. 6 shows a comparison of dynamic activity for cells in seven tissue samples, where oxygen concentration is adjusted for each sample, in accordance with an embodiment. Depending on the concentration of oxygen provided to different cells, the dynamic momentum of the cell may appear differently. Oxygen is an essential component of cellular respiration, a process by which cells generate energy to perform their functions. When there is not enough available oxygen, the energy-generating ability of cells is reduced, which can affect the cell’s dynamic activity. It can be observed that human cell line (HeLa) cells exhibit more dynamic activity when approximately 12.5% oxygen (compared to 20% oxygen in normal atmosphere) issupplied to the cells. On the other hand, when a limited amount of oxygen is supplied to the cells, the relative decrease in dynamic activity of the cells can be numerically confirmed.

[0090] High-speed optical imaging techniques provide not only the morphological information of the cells (e.g., allowing for labeling of different structures or organelles within each cell), but also provides dynamic activity information about intracellular movements such that the viability of the cells can be evaluated in a non-invasive manner. ML techniques can be utilized to predict the viability of cells based on a provided set of high-speed optical images of the cells and the frequency data generated therefrom. Each cell depicted in the set of images can be labeled by the ML algorithm according to a binary class: viable / non-viable.

[0091] In an embodiment, a training dataset comprising high-speed interferometry images can be generated by imaging known samples of cells, both viable and non-viable, as well as mixtures of both viable and non-viable cells. Each set of images is then paired with a groundtruth output that indicates whether each cell in the images is viable of non-viable. The groundtruth outputs in the training data set can be manually generated by viewing the cells in the sample and labeling each cell using known techniques. The training dataset can be used to train various artificial intelligence models to classify the cells as either viable or non-viable.Additional test sets of images that are not in the training dataset can also be collected and used to assess the accuracy of the models.

[0092] In some embodiments, the machine learning algorithm trained using the training dataset may be selected from one of a logistic regression algorithm, a random forest algorithm, a support vector machine, or a Gaussian Naive Bayes algorithm. During testing, each of these algorithms was found to be between 93% and 95% accurate, as shown in the table below:Table ILogistic Random Forest SVM Gaussian NaiveRegression BayesBalanced 93.49 95.00 94.19 93.03Accuracy (%)

[0093] FIG. 7 is an output image 700 showing the classification of one or more cells in a tissue sample that indicates a predicted characteristic of each cell, in accordance with some embodiments. The image may be selected from the set of images 420 as a representative imageof a particular segment 422. In certain embodiments, the image 700 may be generated by the one or more ML / DL algorithms. As shown in FIG. 7, the image 700 has replaced the color of each cell with a color that indicates a level of the characteristic predicted for that cell or a color that corresponds with a state of the predicted characteristic.

[0094] For example, where the predicted characteristic is viability of each cell based on the measured intracellular dynamic activity, then viable cells may be shown in one color (e.g., green) and non-viable cells may be shown in another color (e.g., red). In the image 700, which is shown in black and white or greyscale, black cells may be the non-viable cells and gray cells may be the viable cells. Of course, any two (or more) colors may be used for the predicted characteristic in other embodiments. In alternative embodiments, locations of each cell may be replaced with a shape (e.g., square / triangle / circle / etc.) to indicate a level or state of the predicted characteristic. In yet other embodiments, when the predicted characteristic has a range of values, a gradient between two colors may be used to indicate a location within the range. For example, a gradient may be used to indicate a confidence of the predicted viability.

[0095] In yet other embodiments, the output of the ML / DL models may take an alternative form to the image, such as an array of values, each value in the array corresponding to a particular cell identified in the image 700. Other forms of output for indicating a characteristic of the tissue sample based on the intracellular dynamic activity are contemplated as being within the scope of the present disclosure.

[0096] FIG. 8A shows a ML / DL framework 800 for predicting a characteristic of cells in a tissue sample based on intracellular dynamic activity, in accordance with an embodiment. As shown in FIG. 8A, the ML / DL framework 800 receives a set of high-speed images 420 captured using the optical device 100 or 150. A pre-processor 810 may use the set of high-speed images 420 to generate frequency data in accordance with the techniques discussed in FIGS. 4A-4D. The frequency data can be appended to the set of images 420 in an output 802 of the pre-processor. In at least one embodiment, the frequency data may include a magnitude and mean-frequency parameter for each pixel location included in the set of images 420, for each of one or more segments 422 of the set of images 420, and may be appended as a new image or images to the set of images. For example, a first image comprising mean-frequency values and a second image comprising magnitude values can be concatenated to the, e.g., M images that make up each segment 422 in the set of images 420. The pre-processor 810 is not referred to as an ML / DLalgorithm as the frequency data can be calculated in accordance with conventional algorithms such as an FFT algorithm.

[0097] In an embodiment, the output 802 of the pre-processor 810, which includes the set of images 420 and the frequency data, is then provided to a semantic segmentation model 820. The semantic segmentation model 820 is designed to incorporate not only the standard structural image data like in some conventional segmentation models, but also to leverage numerical data related to intracellular dynamic activity, such as movement frequency and power density, to provide additional context to the segmentation process. In an embodiment, the semantic segmentation model 820 is a ML algorithm that generates a bit map for each cell in the set of images 420. The value of each pixel in the bit map corresponds to a label for a class associated with that pixel location in the set of images 420. The classes represent the different structural components of a cell. For example, a first class may represent a cell nucleus, a second class may represent mitochondria, a third class may represent a Golgi apparatus, a fourth class may represent cytoplasm, and so forth. A value of zero may indicate that the pixel location does not belong to any cell in the image (i.e., associated with a background between cells). Thus, the semantic segmentation model 820 generates a semantic map 804 that indicates, for each pixel location in the set of images 420, which part of a cell (e.g., class) that pixel location belongs to, for each of the one or more cells identified in the set of images 420. In an embodiment, the semantic segmentation model 820 can be trained using both the structural features extracted from the set of images 420, similar to conventional semantic segmentation models, but also using the features extracted from the frequency data, which provides additional context about scatter movement within each cell that can provide additional context for predicting the classification of different intracellular structures within each cell. In one embodiment, the output of the semantic segmentation model 820 includes both the semantic map 804 and the frequency data 806 for each of the organelles within the cell(s), which can be extracted from the frequency data portion of the input 802.

[0098] In an embodiment, another ML / DL model 830 is then used to process the semantic map 804 and the frequency data 806 to predict a characteristic for each cell based on the intracellular dynamic activity. Again, the predicted characteristic can be, e.g., a cell viability classification, a stress level (e.g., a level of stress of each cell indicative of changes in oxygen concentration or nutrients available to each cell), dynamic activity changes, and / or metabolicfractionation (among other potential cellular characteristics). In an embodiment, the output 840 of the ML / DL model 830 is an image that indicates a value of state of the predicted characteristic for each cell of the one or more cells identified in the set of images 420. For example, the image may be essentially a copy of one of the images in the set of images 420, but with the pixels associated with each cell changed to a particular color to represent whether the cell is viable (e.g., green) or non-viable (e.g., red). In other embodiments, the output 840 of the ML / DL model 830 may take other forms, such as a list of cells identified by a cell index assigned to each identified cell (e.g., a value from 0 to n-1), and a corresponding label for each of the n cells indicating the value of state of the predicted characteristic.

[0099] In an embodiment, the semantic segmentation model 820 may comprise one of a convolutional neural network (CNN), a variational autoencoder (VAE), a U-Net or the like. The semantic segmentation model 820 may be trained to minimize an LI loss of the classification of cells in the images.

[0100] In an embodiment, the ML / DL model 830 is a deep CNN (e.g., includes 36, 50, or more hidden layers) trained to predict the viability of each cell based on the semantic map 804 and frequency data 802 associated with the intracellular structures within each cell. In an embodiment, training of the ML / DL model 830 can be performed using biopsy data 808 as ground truth data to be used comparatively with the predicted output to train the ML / DL model 830. For example, biopsy data 808 can be used to identify, via laboratory results, assays, or other well-known means for testing tissue samples, the predicted characteristic for a tissue sample. This ground truth data can be compared against the output 840 of the ML / DL model 830 during training to adjust the parameters of the model using, e.g., back propagation with gradient descent or other well-known training techniques.

[0101] In other embodiments, the semantic segmentation model 820 and / or the ML / DL model 830 may be implemented as one or more of a convolutional neural network (CNN), a recurrent neural network (RNN), a long-short term memory (LSTM), an encoder-decoder, a generative adversarial network (GAN), and / or multi-scale or pyramid-based networks. Examples include, but are not limited to, DeConvNet, HRNet, U-Net, Feature Pyramid Network (FPN), Mask R-CNN, ReSeg, and others.

[0102] As used herein, the terms model and algorithm may be used interchangeably within the context of the description of the pre-processor 810, semantic segmentation model 820, and / orML / DL model 830. These terms may be used to describe a set of instructions that may be executed by one or more processors to perform some desired function. For example, in the case of the ML / DL model 830, a set of instructions may be executed by one or more processors, including, e.g., a parallel processor, to implement a CNN.

[0103] FIG. 8B illustrates one architecture of the semantic segmentation model 820, in accordance with some embodiments. As shown in FIG. 8B, the semantic segmentation model 820 can be divided into two branches. A first branch processes the frequency data 850 in the input 802, and a second branch processes the set of images 420 including the cell structure information in parallel with the first branch. The first branch extracts a first set of features from the frequency data using a feature extraction model 852 and generates a first semantic map using an image segmentation model 854 based only on the frequency data 850. The second branch extracts a second set of features from the set of images 420 using a feature extraction model 856 and generates a second semantic map using an image segmentation model 858 based only on the set of images 420. The first semantic map and the second semantic map are then merged via a merging algorithm 860 to generate a final semantic map 804 provided to the ML / DL model 830.

[0104] In one embodiment, the feature extraction models 852 / 856 and the image segmentation models 854 / 858 are CNNs. For example, a first CNN is used to extract features from the frequency data 850 or set of images 420 to generate a set of feature maps, and a second CNN is used to process the set of feature maps to generate the first and second semantic maps, respectively. In another embodiment, the feature extraction models 852 / 856 may be CNNs used to extract the feature maps, but the image segmentation models 854 / 858 may implement, e.g., a k-means clustering algorithm to predict the class of each pixel location based on the vector of extracted features for that pixel. Other types of ML / DL algorithms may be implemented for either the feature extraction or image segmentation steps.

[0105] In an embodiment, the extracted features can include signal values (e.g., meanfrequency, magnitude, etc.), thresholds, histograms, and edges, for example. Various ML / DL techniques for estimating the predicted class based on the extracted features include, but are not limited to, region growing, k-means clustering, watershed method, active contours, and graph cuts in lieu of the CNN algorithms discussed herein. Additionally, more advanced algorithms such as Markov random field and / or sparsity-based methods may be implemented.

[0106] In an embodiment, each of the branches may implement a separate CNN for generating the first and second semantic map, respectively. In other words, the two CNNs may be trained separately and use different parameters. The first and second semantic maps are then merged via a merging algorithm 860, as discussed in more detail below. Alternatively, the merging operation can be performed by a separate ML / DL model trained to take two semantic maps as input and predict the final semantic map 804. In other words, the additional ML / DL model is trained to predict which class value to trust in the case of a mis-match between the predicted class value of the two branches. In one embodiment, the additional ML / DL model may be a relatively simple CNN with a small number of hidden layers.

[0107] In an embodiment, the merging algorithm 860 includes determining, for each pixel location, whether the value of the class in the first semantic map matches the value of the class in the second semantic map. If the values match, then the matching value is used in the final semantic map 804 for that pixel location. If the values do not match, then a class value for that pixel location in the final semantic map 804 can be selected using an alternative manner. In an embodiment, the alternative manner can be that the class value is defaulted to zero (e.g., the background), as the mismatched values from the two branches indicate a low confidence in an accurate prediction. In another embodiment, the alternative manner can be that the class value is selected based on one or more neighboring pixels in the final semantic map 804. For example, by sampling the 8 neighboring pixels immediately adjacent to the current pixel, and determining a class value in the neighboring pixels that occurs the most, the class value for the current pixel can be selected based on the most common class of the neighboring pixels. In yet another embodiment, the alternative manner can be that the class value is selected by randomly selecting the class value from the semantic map from one of the two branches. In other words, even though there is a mismatch between the class predicted by the two branches for a given pixel location, one of the predicted class values is more likely to be correct compared to a random class, so the merging algorithm 860 can randomly select from the two predicted options.

[0108] FIG. 8C illustrates another architecture of the semantic segmentation model 820, in accordance with some embodiments. As shown in FIG. 8C, the semantic segmentation model 820 is again divided into two branches. However, in this architecture, features are extracted from both the frequency data 850 and the set of images 420 included in the input 802, via the feature extraction models 852 / 856, and then the extracted features are merged via merging algorithm862 before the final semantic map 804 is predicted via image segmentation model 864 based on the merged set of features.

[0109] In an embodiment, the merging algorithm 862 simply concatenates the two sets of feature maps, which are then processed by the image segmentation model 864 to generate the final semantic map 804. The image segmentation model 864 may be similar to the image segmentation models 854 / 858 discussed above and may implement one of a CNN, RNN, k- means clustering algorithm, or the like.

[0110] Although the above description of the ML framework 800 shows one exemplary embodiment where the ML / DL model 830 processes only the semantic map 804 and the frequency data 806, in other embodiments, the ML / DL model 830 can also be configured to process the set of images 420 in addition to the semantic map 804 and frequency data 806.

[0111] FIG. 9 is a flowchart of a method 900 for predicting characteristics of a tissue sample using machine learning techniques, in accordance with some embodiments. Although the method is described in conjunction with the optical device 100 and / or the optical device 150, it will be appreciated that the method 900 can be performed using a number of different optical imaging devices capable of imaging intracellular structures in conjunction with a high-speed camera. Furthermore, the steps described herein may be performed, at least in part, using one or more processors of a computing device. The steps may be implemented by executing instructions in any combination of hardware, firmware, and / or software.

[0112] At 902, a set of images of a tissue sample are captured using a high speed camera. In an embodiment, an interferometry imaging system is utilized to capture the set of high-speed images of the tissue sample. In another embodiment, an optical microscopy imaging device (e.g., non-interferometry based) is utilized to capture the set of high-speed images of the tissue sample. The tissue sample may be placed in a sample holder that provides some manner of gas and / or temperature control for the environment proximate the tissue sample. For example, in one embodiment, a sample holder includes a means to deliver temperature controlled and flow- regulated gas to a diffusion port located proximate the tissue sample. In another embodiment, a hand-held probe 190 may be used for in vivo applications that do not necessitate extracting a sample tissue and the hand-held probe 190 can be applied directly to a test subject (e.g., human body).

[0113] At 904, the set of images may be pre-processed to generate frequency data. The frequency data can include, for each pixel location associated with a cell in the image(s), a mean frequency parameter and / or a magnitude parameter that reflect intracellular dynamic activity of the cell or of the various intracellular structures (e.g., organelles). In an embodiment, step 904 is optional as the features related to the frequency data can be derived directly by a machine learning model by processing the high-speed images.

[0114] At 906, the set of images are processed by a semantic segmentation model to identify different intracellular structures within each cell. The semantic segmentation model can generate a semantic map that indicates which pixel locations corresponding to each cell are classified as which types of intracellular structures.

[0115] At 908, the frequency data and / or semantic map are processed by a ML / DL model to generate an output that indicates a predicted characteristic of the tissue sample, or of each cell of the tissue sample. In an embodiment, viability of the tissue sample may be indicated by identifying which cells in the images are predicted to be viable versus non-viable. In another embodiment, a stress level of each cell is predicted. In yet other embodiment, dynamic activity changes or metabolic fractionation is predicted.

[0116] At 910, the output is utilized to select tissue samples for further processing or use. In at least one embodiment, the output can be analyzed by a processor to automatically determine whether the tissue sample should be selected for a particular use. For example, tissue samples that are used to test drug candidates can be evaluated against a threshold criteria to determine whether a threshold number or percentage of cells in the imaged tissue sample are viable. If the number or percentage of cells predicted to be viable using the techniques described above are above the threshold criteria, then the tissue sample or drug candidate may be selected as a target candidate for further study or use in clinical trials. In some embodiments, a successful target candidate can be used to generate additional drug candidates for further testing.

[0117] In yet another embodiments, the predicted characteristic(s) for specific cells can be analyzed to evaluate the threshold criteria. For example, a tissue sample may include cells of a first type and cells of a second type (e.g., cancer cells). The threshold criteria can be evaluated by determining whether a first threshold of cells of the first type are viable and a second threshold of cells of the second type are non-viable. In other words, when evaluating a particular drug candidate, the success of the candidate can be determined based on whether the candidate issuccessful at eliminating viability of one type of cell while maintaining viability of another type of cell.

[0118] An example system suitable for use in implementing some embodiments of the present disclosure is set forth below. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and / or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. Furthermore, persons of ordinary skill in the art will understand that any system that performs the operations of the method is within the scope and spirit of embodiments of the present disclosure.

[0119] FIG. 10 illustrates an exemplary computer system 1000, in accordance with some embodiments. The computer system 1000 includes a processor 1002, a non-volatile memory 1004, and a network interface controller (NIC) 1020. The processor 1002 can execute instructions that cause the computer system 1000 to implement the functionality various elements of the system 200 described above. For example, the control system 250 and / or the remote computing device can each take the form of the computer system 1000.

[0120] Each of the components 1002, 1004, and 1020 can be interconnected, for example, using a system bus to enable communications between the components. The processor 1002 is capable of processing instructions for execution within the system 1000. The processor 1002 can be a single-threaded processor, a multi-threaded processor, a vector processor or parallel processor that implements a single-instruction, multiple data (SIMD) architecture, or the like. The processor 1002 is capable of processing instructions stored in the volatile memory 1004. In some embodiments, the volatile memory 1004 is a dynamic random access memory (DRAM). The instructions can be loaded into the volatile memory 1004 from a non-volatile storage, such as a Hard Disk Drive (HDD) or a solid state drive (not explicitly shown), or received via the network. In an embodiment, the volatile memory 1004 can include instructions for an operating system 1006 as well as one or more applications 1008. It will be appreciated that theapplication(s) can be configured to provide the functionality of one or more components of the system 200, as described above. A network interface card (NIC) 1020 enables the computer system 1000 to communicate with other devices over a network, including a local area network (LAN) or a wide area network (WAN) such as the Internet.

[0121] It will be appreciated that the computer system 1000 is merely one exemplary computer architecture and that the processing devices implemented in the system 200 can include various modifications such as additional components in lieu of or in addition to the components shown in FIG. 10. For example, in some embodiments, the computer system 1000 can be implemented as a system-on-chip (SoC) that includes a primary integrated circuit die containing one or more CPU cores, one or more GPU cores, a memory management unit, analog domain logic and the like coupled to a volatile memory such as one or more SDRAM integrated circuit dies stacked on top of the primary integrated circuit dies and connected via wire bonds, micro ball arrays, and the like in a single package (e.g., chip). In another embodiment, the computer system 1000 can include a printed circuit board with a number of components soldered thereto, as well as one or more expansion cards coupled to an interface such as a peripheral component interconnect (PCI) express (PCIe), or the like. In yet another embodiment, the computer system 1000 can be implemented as a server device, which can, in some embodiments, execute a hypervisor and one or more virtual machines that share the hardware resources of the server device.

[0122] It is noted that the techniques described herein may be embodied in executable instructions stored in a computer readable medium for use by or in connection with a processorbased instruction execution machine, system, apparatus, or device. It will be appreciated by those skilled in the art that, for some embodiments, various types of computer-readable media can be included for storing data. As used herein, a "computer-readable medium" includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer-readable medium and execute the instructions for carrying out the described embodiments. Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format. A non-exhaustive list of conventional exemplary computer-readable medium includes: a portable computer diskette; a random-access memory (RAM); a read-only memory (ROM); an erasable programmable read only memory (EPROM); aflash memory device; and optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), and the like.

[0123] It should be understood that the arrangement of components illustrated in the attached Figures are for illustrative purposes and that other arrangements are possible. For example, one or more of the elements described herein may be realized, in whole or in part, as an electronic hardware component. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0124] To facilitate an understanding of the subject matter described herein, many aspects are described in terms of sequences of actions. It will be recognized by those skilled in the art that the various actions may be performed by specialized circuits or circuitry, by program instructions being executed by one or more processors, or by a combination of both. The description herein of any sequence of actions is not intended to imply that the specific order described for performing that sequence must be followed. All methods described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.

[0125] The use of the terms "a" and "an" and "the" and similar references in the context of describing the subject matter (particularly in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term “at least one” followed by a list of one or more items (for example, “at least one of A and B”) is to be construed to mean one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by context. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation, as the scope of protection sought is defined by the claims as set forth hereinafter together with any equivalents thereof. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illustrate the subject matter and does not pose a limitation on the scope of the subject matter unless otherwise claimed. The use of the term“based on” and other like phrases indicating a condition for bringing about a result, both in the claims and in the written description, is not intended to foreclose any other conditions that bring about that result. No language in the specification should be construed as indicating any nonclaimed element as essential to the practice of the invention as claimed.

Claims

CLAIMSWhat is claimed is:

1. A system for predicting characteristics of a tissue sample, the system comprising: an optical device including a high-speed camera for capturing a set of images of the tissue sample; a controller in communication with the high-speed camera, the controller comprising at least one processor to: process the set of images using at least one machine learning (ML) / deep learning (DL) algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

2. The system of claim 1, wherein the optical device comprises an interferometry imaging device or an optical microscope.

3. The system of claim 1, wherein the optical device comprises a hand-held probe including a beam shape transformer.

4. The system of claim 1, wherein the processing the set of images using the at least one ML / DL algorithm comprises: dividing the set of images into one or more segments, each segment of the one or more images including AT images; generating, for each segment in the one or more segments, frequency data for each pixel of the AT images in the segment; and concatenating the set of images and the frequency data for processing by the at least one ML / DL algorithm.

5. The system of claim 4, wherein the processing the set of images using the at least one ML / DL algorithm further comprises:processing at least one image in each segment by a semantic segmentation model to generate a semantic map that indicates at least two different intracellular structures in each of one or more cells; and concatenating the semantic map with at least one of the set of images or the frequency data for processing by the at least one ML / DL algorithm.

6. The system of claim 5, wherein the at least one ML / DL algorithm comprises a convolutional neural network (CNN) that is trained to generate at least one output image that indicates, for each cell of one or more cells depicted in the set of images, a predicted characteristic for the cell.

7. The system of claim 5, wherein the semantic segmentation model comprises at least one convolutional neural network (CNN).

8. The system of claim 4, wherein a sampling frequency of the high-speed camera is at least 50 frames per second.

9. The system of claim 4, wherein generating the frequency data comprises, for each pixel location in the set of images: processing M pixel values for the pixel location within each segment using a Fast Fourier Transform (FFT) algorithm to generate a frequency spectrum for the pixel location; and calculating a mean frequency parameter in accordance with the following equation based on the frequency spectrum:J fmean £-11=1 ‘ P i-Jf 1. / / -n=l1p i. where AL is the size of the segment;is the frequency of the frequency spectrum at bin z of AL; and Pi is the power density of the frequency spectrum at bin i oiM.

10. The system of claim 9, wherein generating the frequency data further comprises, for each pixel location in the set of images: calculating a magnitude parameter in accordance with the following equation based on the frequency spectrum:mag = ^=i P .

11. The system of claim 1, wherein the tissue sample and optical device are enclosed within an environmental enclosure that includes at least one of temperature control, humidity control, or atmospheric control.

12. The system of claim 11, wherein the tissue sample is located within a sample holder that includes at least one temperature sensor and a gas port for delivering a gas proximate the sample.

13. The system of claim 12, wherein a feedback signal from the at least one temperature sensor is used to adjust a temperature of the gas provided to the tissue sample to heat or cool the tissue sample to a desired temperature.

14. A method for predicting viability of a tissue sample, the method comprising: capturing, using a high-speed camera associated with an optical device, a set of images of the tissue sample; and processing the set of images using at least one machine learning (ML) / deep learning (DL) algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

15. The method of claim 14, wherein the processing the set of images using the at least one ML / DL algorithm comprises: dividing the set of images into one or more segments, each segment of the one or more images including A images; generating, for each segment in the one or more segments, frequency data for each pixel of the AY images in the segment; and concatenating the set of images and the frequency data for processing by the at least one ML / DL algorithm.

16. The method of claim 15, wherein the processing the set of images using the at least one ML / DL algorithm further comprises: processing at least one image in each segment by a semantic segmentation model to generate a semantic map that indicates at least two different intracellular structures in each of one or more cells; and concatenating the semantic map with at least one of the set of images or the frequency data for processing by at least one additional ML / DL algorithm.

17. The method of claim 16, wherein the at least one ML / DL algorithm comprises a convolutional neural network (CNN) that is trained to generate at least one output image that indicates, for each cell of one or more cells depicted in the set of images, a predicted characteristic of the cell.

18. The method of claim 15, wherein generating the frequency data comprises, for each pixel location in the set of images comprises: processing M pixel values for the pixel location within each segment using a Fast Fourier Transform (FFT) algorithm to generate a frequency spectrum for the pixel location; and calculating a mean frequency parameter in accordance with the following equation based on the frequency spectrum:where AL is the size of the segment; ftis the frequency of the frequency spectrum at bin i ofand Pi is the power density of the frequency spectrum at bin z of L.

19. The method of claim 18, wherein generating the frequency data further comprises, for each pixel location in the set of images: calculating a magnitude parameter in accordance with the following equation based on the frequency spectrum:

20. A remote computing device in communication with an optical device including a high-speed camera for capturing a set of images of a tissue sample, wherein the remote computing device comprises: a memory for storing the set of images and parameters for at least one machine learning algorithm; and at least one processor in communication with the memory to: process the set of images using the at least one machine learning (ML) / deep learning (DL) algorithm to predict characteristics of one or more cells in the tissue sample based, at least in part, on intracellular dynamic activity information.

Images