System and method for detecting and classifying living microorganisms using a thin film transistor (TFT) image sensor and deep learning

Abstract

A bacterial colony-forming unit (CFU) detection system is disclosed that utilizes a thin-film transistor (TFT)-based image sensor array to save approximately 12 hours compared to Environmental Protection Agency (EPA) approved methods. A lens-free imaging modality was constructed using a TFT image sensor with a sample field of view of approximately 10 cm 2 was used. Time-lapse images of bacterial colonies cultured on chromogenic agar plates were automatically collected at 5-minute intervals. Two deep neural networks were used to detect and count the growing colonies and identify their species. When blindly tested using 265 colonies of Escherichia coli and other coliform bacteria (i.e., Citrobacter and Klebsiella pneumoniae), the system reached an average CFU detection rate of 97.3% with 9 hours of incubation and an average recovery rate of 91.6% in approximately 12 hours. This TFT-based sensor can be applied to various microbiological detection methods. The imaging field of view of this platform can be cost-effectively increased to >100 cm 2 can be increased cost-effectively.

Description

Technical Field

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 338,972, filed May 6, 2022, which is incorporated herein by reference. Priority is claimed in accordance with 35 U.S.C. § 119 and any other applicable statutes.

[0002] This technical field generally relates to early screening and detection methods for the detection and / or identification of living microorganisms such as cells (prokaryotic or eukaryotic), viruses, fungi, bacteria, yeast, and multicellular organisms. More particularly, this technical field relates to systems and methods that periodically capture holographic microscope images of bacterial growth on a growth plate and automatically analyze these spatio-temporal patterns or holograms over time using multiple deep neural networks for rapid detection and / or classification of the corresponding microbial species.

Background Art

[0003] Bacterial infections are a major cause of millions of deaths every year in both developed and developing countries. The costs associated with treating bacterial infections exceed $4 billion per year in the United States (US) alone. Therefore, rapid and accurate detection of pathogenic bacteria is very important for human health, for example, when preventing such infectious diseases caused by food and drinking water contamination. Among those pathogenic bacteria, Escherichia coli (E. coli) and other coliform bacteria are one of the most common ones, and they indicate fecal contamination in food and water samples. The most basic and frequently used method for detecting E. coli and total coliform bacteria involves culturing samples on solid agar plates or in liquid media according to US Environmental Protection Agency (EPA)-approved protocols (for example, methods EPA1103.1 and EPA1604). However, these traditional culture-based methods usually require ≥24 hours for the final readout and need visual recognition and counting of colony-forming units (CFUs) by microbiology experts. Various nucleic acid-based molecular detection techniques have been developed for rapid bacterial detection with results ready in less than a few hours, but they generally show lower sensitivity and have problems in differentiating live bacteria from dead bacteria, and in fact, there is no EPA-approved nucleic acid-based E. coli detection method that can be used for screening water samples.

[0004] A variety of other techniques have been developed to provide high sensitivity and specificity for the detection of bacteria based on different methods, such as fluorescence measurement, solid-phase hemocytometry, fluorescence microscopy, Raman spectroscopy, etc. However, these systems generally do not operate at large sample volumes (e.g., >0.1 L). As another alternative, Wang et al. demonstrated a complementary metal-oxide semiconductor (CMOS) image sensor-based time-lapse imaging platform for the early detection and classification of coliform bacteria. See Wang, H. et al., A. Early Detection and Classification of Live Bacteria Using Time-Lapse Coherent Imaging and Deep Learning. Light Sci. Appl. 2020, 9 (1), 118. https: / / doi.org / 10.1038 / s41377-020-00358-9. This method achieved a savings in detection time of over 12 hours and provided species classification with >80% accuracy within 12 hours of incubation. The field of view (FOV) of the CMOS image sensor in this design is <0.3 cm 2 and, therefore, mechanical scanning of the Petri dish area was required to obtain images of the entire FOV of the cultured sample. This is time-consuming, requires additional sample scanning hardware, and it also introduces some additional digital processing burden for image alignment and stitching. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] Recently, due to the rapid development of thin-film transistors (TFTs), TFT technology has been widely used in the fields of flexible displays, radio frequency identification tags, ultra-thin electronic circuits, and large-scale sensors, thanks to its high scalability, low-cost mass production (such as with roll-to-roll manufacturing), low power consumption, and low heat generation. TFT technology has also been applied in the field of biosensing, for example, to detect pathogens by converting antibody-antigen binding, enzyme-substrate catalytic activity, or DNA hybridization into electrical signals. For example, a low-cost TFT nanoribbon sensor was developed by Hu et al. to detect gene copies of Escherichia coli and Klebsiella pneumoniae (K. pneumoniae) within minutes by using pH changes due to DNA amplification. See Hu, C. et al., Ultra-Fast Electronic Detection of Antimicrobial Resistance Genes Using Isothermal Amplification and Thin Film Transistor Sensors, Biosens. Bioelectron. 2017, 96, 281-287. https: / / doi.org / 10.1016 / j.bios.2017.05.016. As another example, Salinas et al. implemented an aZnO TFT biosensor with a recyclable plastic substrate for real-time Escherichia coli detection. See Salinas, R. A. et al., Biosensors Based on Zinc Oxide Thin-Film Transistors Using Recyclable Plastic Substrates as an Alternative for Real-Time Pathogen Detection. Talanta 2022, 237, 122970. However, these TFT-based biosensing methods were unable to distinguish between live and dead bacteria and did not provide quantification of the CFU concentration of the sample under test.

Means for Solving the Problem

[0006] Here, the TFT-based image sensor is used to construct a real-time CFU detection system for automatically counting bacterial colonies and rapidly identifying their species using deep learning. The large FOV (about 10 cm 2 or more) of the TFT image sensor eliminates the need for mechanical scanning of agar plates, which enables the creation of a field-portable and cost-effective lens-free CFU detector as shown in FIGS. 2A-2C. This compact system includes continuously switched red, green, and blue light-emitting diodes (LEDs) that periodically illuminate the cultured samples (Escherichia coli, Citrobacter, and Klebsiella pneumoniae) as shown in FIG. 2C, and the spatio-temporal pattern of the samples is collected by the TFT image sensor over an imaging period of 5 minutes. Two deep learning-based classifiers were trained to detect bacterial colonies and then classify them into E. coli and total coliform bacteria. Blind-tested on a dataset populated with 265 colonies (85 E. coli CFUs, 66 Citrobacter CFUs, and 114 Klebsiella pneumoniae CFUs), the TFT-based system was able to detect the presence of colonies as early as about 6 hours during the incubation period, achieving an average CFU detection rate of 97.3% with 9-hour incubation and saving more than 12 hours compared to the EPA-approved culture-based CFU detection method. For the classification of the detected bacterial colonies, an average recovery rate of 91.6% was achieved with about 12-hour incubation.

[0007] This TFT-based portable in-situ CFU detection system can be further scaled up to achieve even lower costs using much larger FOVs based on, for example, roll-to-roll manufacturing methods commonly used in the flexible display industry, and is cost-effective and significantly benefited from the ultra-large FOV of the TFT image sensor. In some embodiments, the TFT image sensor can be integrated with each agar plate to be tested, disposed of after determination of the CFU count, and can open up various new opportunities for microbiological instrumentation in laboratory and in-situ settings.

[0008] In one embodiment, a system for the detection and classification of live microorganisms and / or their colonies in a sample using time-lapse imaging is disclosed. The system includes a light source and a thin-film transistor (TFT)-based image sensor disposed along an optical path generated from the light source. A growth plate containing a growth medium and the sample is inserted along the optical path and disposed adjacent to the TFT-based image sensor. A microcontroller or other circuitry in the system is configured to periodically illuminate the growth plate using light from the light source and capture time-lapse images of the microorganisms and / or their colonies on the growth plate using the TFT-based image sensor. The system includes a computing device configured to execute image processing software for processing and analyzing the time-lapse images of the microorganisms and / or their colonies on the growth plate and detecting candidate microorganisms and / or their colonies in the time-lapse images.

[0009] In another embodiment, a method is disclosed for detecting and classifying living microorganisms and / or colonies thereof using time-lapse imaging. The method includes providing a growth plate containing an agar medium thereon and containing a sample, periodically illuminating the growth plate with illumination light of at least one spectral band from a light source, capturing a time-lapse image of the microorganisms and / or colonies thereof on the growth plate using a TFT-based image sensor, a first trained deep neural network trained to detect true microorganisms and / or colonies thereof from non-microbial objects, and a second trained deep neural network that receives at least one time-lapse image or digitally processed time-lapse image as input and outputs a species classification associated with the detected true microorganisms and / or colonies thereof, and detecting candidate microorganisms and / or colonies thereof in the time-lapse image using image processing software including the second trained deep neural network.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0011] FIG. 1 illustrates a system 10 for the early detection and classification of living microorganisms and / or their colonies in a sample 110 using time-lapse imaging and deep learning, according to one embodiment. The microorganisms include prokaryotic cells, eukaryotic cells (e.g., stem cells), fungi, bacteria, viruses, multicellular organisms (e.g., parasites), or clusters or films or colonies thereof. The system 10 includes a holographic imager device 12 (see also FIGS. 2A-2C) used to obtain time-lapse images 70h of microorganism growth occurring on one or more growth plates 14 (e.g., a solid growth medium supplemented with nutrients used to culture microorganisms, or a Petri dish containing chromogenic agar as another growth medium suitable for the type of microorganism). The images 70h include the spatio-temporal pattern (e.g., hologram) of the sample 110.

[0012] The holographic imager device 12 includes a light source 18 that is used to direct light onto the sample 110. The light source 18 may include a three - color LED module that sequentially switches between red, green, and blue light - emitting diodes (LEDs), as described herein. Other alternatively actuated spectral bands may be used in the light source 18 in alternative embodiments. The holographic imager device 12 further includes a TFT - based image sensor 20 disposed along the optical path of the light emitted from the light source 18. As seen in FIGS. 2A and 2C, the holographic imager device 12 includes a frame or housing 13 in which the light source 18 is disposed at one end (e.g., the upper end) and the TFT - based image sensor 20 is disposed at the opposing end (e.g., the lower end). And a growth plate 14 containing the sample 110 is inserted into the optical path between the light source 18 and the TFT - based image sensor 20. In some embodiments, the growth plate 14 may be placed directly above the TFT - based image sensor 20. In other embodiments, the growth plate 14 may contain the TFT - based image sensor 20 directly above or within the growth plate 14. The TFT - based image sensor 20 may be reusable or, in some embodiments, disposable. Optional lenses or sets of lenses (not shown) may be disposed along the optical path and used to magnify or reduce the hologram captured using the TFT - based image sensor 20. The distance between the light source 18 and the sample 110 (in other words, the z 1 distance) is significantly greater (>>) than the distance between the sample 110 and the TFT sensor 20 (z 2 ). For example, in one embodiment, the z 1 distance is about 15.5 cm and the z 2 distance is about 5 mm.

[0013] In some embodiments, the holographic imager device 12 may include an incubator 16 for heating one or more growth plates 14 and / or maintaining the temperature at a setpoint temperature or temperature range optimal for microbial growth. A separate incubator 16 may be used with the holographic imager device 12. The incubator 16 may include, in one embodiment, an optically transparent plate or substrate that contains heating elements used to adjust the temperature of one or more growth plates 14 therein. The incubator 16 may also include a fully or partially enclosed housing that houses the holographic imager device 12 together with one or more growth plates 14. The holographic imager device 12 may also include one or more optional humidity control units 17 used to maintain one or more growth plates 14 at a setpoint humidity level or range. The humidity control unit 17 may be integrated with the incubator 16, the holographic imager device 12, or a separate component.

[0014] A series of time-lapse images 70h of microorganisms and / or their colonies on the growth plate 14 are used to identify microorganism colony candidates based on the time-lapse differential images (in other words, time-lapse images) obtained over time. The differential images (images 70h obtained at different times) include images of growing microorganisms and / or colonies, but also include non-microbial objects such as dust, agar bubbles themselves or surface movement, and other artifacts. Image processing software 80 executed on a computing device 82 having one or more processors 84 is used to perform image preprocessing, differential analysis, colony mask segmentation, and candidate position localization, and cropping of the videos of colony candidates. However, some of these videos of colony candidates may represent non-biological objects or artifacts such as bubbles, dust, etc. that are not true microorganism colonies and need to be masked or excluded. As described herein, a first trained deep neural network (DNN) 90 is used by the image processing software 80 to detect actual microorganisms and / or colonies and ignore non-microbial objects. When "true" microorganisms and / or colonies are selected, one or more of the time-lapse images and / or at least one digitally processed time-lapse image (e.g., a re-normalized image generated by the division-based normalization described herein) are sent to a second trained deep neural network (DNN) 92 that is used to classify the species of the microorganisms and / or colonies or to classify and separate specific species.

[0015] System 10 implements a holographic imager device 12 that includes a holographic imaging system for capturing holographic images of growing microorganisms and / or colonies. A light source 18 (e.g., an illumination module including a three-color light-emitting diode (LED)) illuminates the microorganisms and / or their colonies on one or more growth plates 14 (incubated using an incubator 16), and a holographic image 70h of the microorganisms and / or their colonies is captured using at least one TFT-based image sensor 20. The light source 18 preferably emits one or more illumination spectral bands that can be actuated (e.g., turned on / off) on demand. This can be achieved through different spectral bands emitted by the light source 18 or through the use of filters that allow the passage of different spectral bands. The holographic imager device 12 can be placed inside a separate incubator 16 or the holographic imager device 12 can be integrated with the incubator 16. In particular, there is no need to scan one or more growth plates 14. A large field of view (FOV) is captured by the TFT-based image sensor 20. In some embodiments, a lens or set of lenses is used to capture an even larger FOV of one or more growth plates 14. Alternatively, a larger-sized TFT-based image sensor 20 can be used. In a preferred embodiment, the captured FOV is at least 10 cm 2 or more. An even larger FOV including a FOV of 100 cm 2 or more is contemplated.

[0016] A microcontroller or control circuit 26 is provided for controlling the illumination of light source 18, the incubator 16, the humidity control unit 17, and the capture of images using the TFT-based image sensor 20. The microcontroller or control circuit 26 may also communicate with a computing device 82 to receive instructions and / or send data, for example, using a program 28 executed by the computing device 82. The microcontroller or control circuit 26 may include one or more microprocessors, drivers, etc. disposed on a printed circuit board (PCB) that are used to operate various subsystems and transfer data. This includes the timing and sequence of illumination using the light source 18, image acquisition from the TFT-based image sensor 20, etc. The microcontroller or control circuit 26 may also be used to control the setpoint temperature or temperature range of the incubator 16. The control circuit 26 may also be used to control the setpoint humidity level or humidity range of the incubator 16 using the humidity control unit 17. The microcontroller or control circuit 26 may be disposed external to the frame 13 as seen in FIG. 2A, or alternatively, the microcontroller or control circuit 26 may be included within the frame 13.

[0017] System 10 includes at least one computing device 82 (e.g., a personal computer, laptop, tablet PC, server, etc.) having one or more processors 84 therein for executing image processing software 80 to process an image 70h obtained from a TFT-based image sensor 20. The computing device 82 can be arranged together with the holographic imager device 12 (e.g., in a local implementation), or the computing device 82 can be arranged remotely from the holographic imager device 12 (e.g., a remote computing device such as a server). In other embodiments, a plurality of such computing devices 82 can be used (e.g., one for controlling the holographic imager device 12 and another for processing the image 70h). Additionally, in some embodiments, the computing device 82 can control various aspects of the operation of the holographic imager device 12 using a microcontroller or control circuit 26. For example, using a graphical user interface (GUI) 94 viewable on a display 83, a user can control aspects of the system 10 (e.g., the periodicity or timing of an image scan, the operation of the TFT-based image sensor 20, the temperature control of the incubator 16, the transfer of the image file 70h from the TFT-based image sensor 20 to the computing device 82, etc.). The GUI 94 can also be used to display videos, sorted colonies 102, colony counts, and to display a colony growth map for viewing / interaction.

[0018] Computing device 82 executes image processing software 80 that includes a microbial and / or colony detection deep neural network 90 for identifying true microorganisms and / or colonies from other non-microbial artifacts (e.g., dust, bubbles, speckles, etc.). Computing device 82 also executes a separate classification deep neural network 92 in image processing software 80 that classifies the specific species classification or actual species of the microorganisms and / or colonies. In an alternative embodiment, the functions of the first and second trained deep neural networks 90, 92 are combined into a single trained deep neural network (e.g., deep neural network 90). Multiple different species of microorganisms and / or colonies may be identified in a single sample. In a particular embodiment, system 10 enables the rapid detection of Escherichia coli and total coliform bacteria (in other words, Klebsiella aerogenes and Klebsiella pneumoniae subsp. pneumoniae) in a water sample. This automated, cost-effective live microorganism detection system 10 can be adapted for a wide range of applications in microbiology by significantly reducing the detection time and automating the identification of microorganisms and / or colonies without labeling or the need for an expert.

[0019] To use system 10, sample 110 is obtained and optionally undergoes a signal amplification operation, in which the sample is pre-incubated using growth medium 112 (Figure 1) for a period of time at an elevated temperature, after which a filtering process follows, for example using a filter membrane. Sample 110 is generally a fluid and can include, for example, a water sample (although sample 110 can be a food sample, a biological or other fluid sample). The filter membrane is then placed in physical contact with one or more growth plates 14 (e.g., the agar surface of growth plate 14) under a light pressure for a period of time to transfer microorganisms (e.g., bacteria) to the agar growth medium in growth plate 14 and is then removed. However, in other embodiments, sample 110 can be placed directly on growth plate 14 and spread, for example, using the L-shaped spreader disclosed herein. One or more growth plates 14 are then covered (e.g., upside down so that the agar surface faces TFT-based image sensor 20) in / on incubator 16 and placed in holographic imager device 12. Growth plate 14 with sample 110 is then allowed to incubate for several hours and is periodically imaged by TFT-based image sensor 20. In some embodiments, a single growth plate 14 is imaged by TFT-based image sensor 20. In other embodiments, multiple growth plates 14 are imaged by TFT-based image sensor 20. In later embodiments, multiple TFT-based image sensors 20 can be used. In some embodiments, the TFT-based image sensor 20 is separate from growth plate 14. In other embodiments, growth plate 14 can be integrated with TFT-based image sensor 20. This can be arranged on or within growth plate 14.

[0020] In a particular embodiment, a method of detecting and classifying living microorganisms and / or colonies thereof using time-lapse imaging involves loading a growth plate 14 containing a sample 110 into or onto an incubator 16. The one or more growth plates 14 are then illuminated with light of different spectral bands (e.g., colors) from a light source 18. In particular, the growth plate 14 is periodically illuminated with illumination light of different spectral bands (e.g., color LEDs) in a continuous manner. An image 70h is captured by a TFT-based image sensor 20 at each color. Various periods between successive illuminations can be used. In one embodiment, about 5 minutes elapse during the illumination of the sample 110. This enables a time-lapse image 70h of the growth plate 14 containing the microorganisms and / or colonies thereof to be taken using a holographic imager device 12. The time-lapse image 70h is then processed, and true microorganisms and / or colonies are detected (and optionally, counted) using a first trained deep neural network (DNN) 90 as shown in FIG. 3.

[0021] Figure 3 illustrates an exemplary workflow of a deep learning-based CFU detection and classification system. Here, eight full FOV RGB images are processed at 20-minute time intervals for difference analysis 200 to select initial colony “candidates” for candidate generation 202. Of course, more or fewer images can be processed at different intervals. A digitally cropped 8-frame RGB image sequence 204 (e.g., a video) (three such candidates are illustrated in FIG. 3) for each individual colony candidate is first fed into the CFU detection neural network 90. This neural network 90 eliminates various non-colony objects (such as dust and bubbles, here candidate 3) among the initial colony candidates and achieves the detection of true colonies (candidates 1 and 2). Next, the colored image sequence 206 of the truly detected colonies is passed through the CFU classification neural network 92 to identify their species (e.g., E. coli or other total coliforms, in other words, binary classification). Finally, the detected microorganisms and / or colonies are then classified (and optionally, counted) using a second trained deep neural network (DNN) 92.

[0022] Figure 6 illustrates further details regarding how the differential analysis 200 is used to generate the colony candidates 202 shown in FIG. 3. In operation (a), a raw time-lapse image is captured by the TFT sensor 20 having RGB channels. Background subtraction is performed to create a background-subtracted image as shown in operation (b). Next, the image is averaged in the time domain to smooth / denoise the image as shown in operation (c). The difference stack and RGB channels of the resulting smoothed / denoised image in operation (d) are merged (averaged) as shown in operation (e). Next, in operation (f), a minimum projection image is generated and undergoes thresholding and morphological processing to generate a coarse detection mask as shown in operation (g). The localization colony positions are identified and colony candidates are selected as shown in operation (h). After the colony candidates are selected, the video of the colony candidates in the RGB color channel is then cropped as the RGB image sequence 204 as shown in operation (i).

[0023] Figure 7 illustrates the network architectures for the CFU detection neural network 90 and the CFU classification neural network 92. A high-density network design is adopted here, where the 2D convolutional layers are replaced with pseudo-3D convolutional blocks. The CFU detection and classification neural networks 90, 92 share the same architecture, but the hyperparameters [m, n, p, q] are selected to be different as shown in FIG. 7.

[0024] Experiment Results The success of system 10 was demonstrated by detecting and classifying Escherichia coli colonies as grayish-green on eosin methylene blue agar plates and other coliform bacteria as pinkish-colored, and suppressing the growth of different bacterial colonies when other types of bacteria were present in the sample, i.e., detecting and differentiating Escherichia coli colonies from two other types of total coliform bacteria, namely Citrobacter and Klebsiella pneumoniae. Each sample 110 was prepared using a Petri dish 14 according to the method of EPA-1103.1 (see the method). After the sample 110 was prepared, the sample 110 was placed directly on top of the TFT-based image sensor 20 as part of the lens-free imaging system 12, and the entire imaging modality (except for the laptop 82 in FIG. 2A) was placed inside the incubator 16 to record the growth of the colonies with an imaging interval of 5 minutes. For each time interval, three images 70h were continuously collected using the TFT image sensor 20 under red (620nm), green (520nm), and blue (460nm) illumination light. This multi-wavelength design enabled the monochromatic TFT image sensor 20 to reconstruct a color image of the bacterial colonies and was used to identify their species mainly by leveraging the color information provided by the selective eosin methylene blue agar medium 112. The recorded time-lapse images 70h were processed using the workflow shown in FIGS. 3 and 6, in which differential analysis 200 was used to generate initial colony candidates 202, and two deep neural networks (DNNs) 90, 92 were trained to further screen the colony candidates to detect the true colonies and infer their species classification / species (see the method section). All these image processing steps took <25 seconds and consumed <1GB of memory using a computer equipped with an Intel Core i7-7700CPU (no GPU required).

[0025] The presented TFT imaging system 10 periodically captures an image 70h of the agar plate 14 under test based on lens-free in-line holography. Due to its large pixel size (375 μm) and relatively small sample sensor distance (equal to the thickness of the agar, approximately 5 mm), a free-space backpropagation step is not required. By directly using the raw intensity image 70h as part of the RGB color channels and calibrating the background, a color image of the agar plate can be generated in <0.25 seconds after the TFT image is recorded. FIGS. 4A - 4B show examples of (color) images of Escherichia coli, Citrobacter, and Klebsiella pneumoniae colonies at different stages of their growth, captured by the system 10. Consistent with the EPA-approved method (EPA-1103.1), when using chromogenic agar, Escherichia coli colonies exhibit a grayish-green color, and Citrobacter and Klebsiella pneumoniae colonies exhibit a pinkish color.

[0026] Based on the imaging performance of the TFT-based CFU detection system 10 summarized in FIGS. 4A-4B, its early detection and classification performance was quantified as shown in FIGS. 5A-5F. For this purpose, the detection and classification neural network model was trained on a dataset of 442 colonies (128 Escherichia coli colonies, 126 Citrobacter, and 188 Klebsiella pneumoniae colonies) captured from 17 independent experiments (see the method for details of the training). The test dataset was populated using 265 colonies from 13 independent experiments with a total of 85 Escherichia coli colonies, 66 Citrobacter colonies, and 114 Klebsiella pneumoniae colonies. The detection rate was defined as the ratio of the number of true colonies confirmed by the CFU detection neural network 90 to the total number of colonies counted by an expert after 24 hours of incubation. FIGS. 5A, 5C, 5E show the detection rates achieved in the blind test phase as a function of the incubation time. As shown in FIGS. 5A, 5C, 5E, a detection rate of >90% was achieved with 8 hours of incubation for Escherichia coli, 9 hours of incubation for Citrobacter, and 7 hours 40 minutes of incubation for Klebsiella pneumoniae. Moreover, a detection rate of 100% was obtained within 10 hours of incubation for Escherichia coli, 11 hours of incubation for Citrobacter, and 9 hours 20 minutes of incubation for Klebsiella pneumoniae. Compared to the EPA-approved standard readout time (24 hours), the TFT-based CFU detection system 10 achieved a time savings of >12 hours. Moreover, from the detection rate curves reported in FIGS. 5A-5F, it can be qualitatively inferred that the colony growth rate of Klebsiella pneumoniae is greater than that of Escherichia coli, which is greater than that of Citrobacter, as the earliest detection times for Escherichia coli, Citrobacter, and Klebsiella pneumoniae colonies were 6 hours, about 6.5 hours, and about 5.5 hours of incubation, respectively.

[0027] To quantify the performance of the bacterial colony classification neural network 92, the recovery rate was defined as the ratio of the number of correctly classified colonies to the total number of colonies counted by an expert after 24 hours of incubation. Figures 5B, 5D, and 5F show the recovery rate curves over all blind test experiments as a function of incubation time. It can be seen that a recovery rate >85% was achieved at 11 hours 20 minutes for E. coli, 13 hours for Citrobacter, and 10 hours 20 minutes for Klebsiella pneumoniae. It is difficult to achieve a 100% recovery rate for all colonies because some of the slow-growing "starter" colonies are unable to grow to a sufficiently large size with the correct color information even after 24 hours of incubation. Figures 5A - 5F also reveal that there is an approximate 3-hour time delay between colony detection time and species discrimination time, which is expected because the detected colonies need more time to grow larger and provide distinguishable color information for the correct classification of their species.

[0028] Discussion It should be noted that the results presented in Figures 5A - 5F represent the storage performance of the TFT-based CFU detection method since the ground truth colony information was obtained after 24 hours of incubation. At the initial stage of the incubation period, some bacterial colonies were not even physically present. Therefore, even higher detection and recovery rates could be reported in Figures 5A - 5F if the number of colonies present at each time point were used as the ground truth.

[0029] Overall, the performance of the TFT-based CFU detection system 10 is similar to that of the CMOS-based time-lapse hologram imaging method with respect to colony detection speed. However, due to its large pixel size (375 μm) and limited spatial resolution, the TFT-based method has a slightly delayed colony classification time. Its extremely large imaging FOV (approx. 10 cm 2) By this, the TFT-based CFU detection method eliminates (1) the time-consuming mechanical scanning of Petri dishes and the associated optomechanical hardware, as well as (2) the image processing steps for image alignment and stitching required by the limited FOV of the CMOS-based imager. In addition to saving image processing time, this also means that System 10 does not include image alignment and stitching artifacts, and thus System 10 can accurately capture the minute spatio-temporal changes in the agar caused by bacterial colony growth at an early stage, helping the system increase CFU detection sensitivity. Due to the massive scalability of the TFT-based image sensor 20 array, the imaging FOV of the platform can be further increased to tens to hundreds of cm in a cost-effective manner 2 and can provide an unprecedented level of imaging throughput for automated CFU detection, for example, using roll-to-roll manufacturing of TFTs, as employed in the flexible display industry.

[0030] Another significant advantage of the TFT imager-based detection system 10 is that it can be adapted to image a wide range of biological samples 110 using a cost-effective, field-portable interface. If the user has concerns about any contamination, the TFT image sensors 20 shown in FIGS. 2B and 2C can be replaced and even used in a disposable manner (e.g., integrated as part of a growth plate 14 (e.g., a Petri dish)). Moreover, the heat generated by the TFT image sensors 20 during the data acquisition process is negligible, ensuring that the biological samples 110 can grow at their desired temperature without being perturbed. Finally, the TFT-based CFU detection system 10 is user-friendly and easy to use because it does not require complex optical alignment, a high-precision mechanical scanning stage, or image alignment / registration steps.

[0031] The presented CFU detection system 10 using a TFT image sensor 20 array provides a high-throughput, cost-effective, and user-friendly solution for the early detection and classification of bacterial colonies, opening up a unique opportunity for microbiological instrumentation in laboratory and field settings.

[0032] Materials and Methods Sample Preparation All bacterial sample preparations were performed in a Biosafety Level 2 laboratory in accordance with the environmental, health, and safety rules of the University of California, Los Angeles. Escherichia coli (Migula) Castellani and Chalmers (ATCC® 25922™), Citrobacter (ATCC® 43864™), and Klebsiella pneumoniae subsp. pneumoniae (Schroeter) Trevisan (ATCC® 13883™) were used as cultured microorganisms. A CHROMagar™ ECC (product number EF322, DRG International, Inc., Springfield, NJ, USA) chromogenic substrate mixture was used as a solid growth medium for detecting E. coli and other total coliform colonies.

[0033] For each time-lapse imaging experiment, a bacterial suspension in phosphate-buffered saline (PBS) (product number 20-012-027, Fisher Scientific, Hampton, NH, USA) was prepared from solid agar plates incubated for 24 hours. The concentration of the suspension was measured using a spectrophotometer (model number ND-ONE-W, Thermo Fisher). Serial dilutions were then made to approximately 10 3It was carried out with PBS until the concentration of CFU / mL was finally reached. An approximately 100 μL diluted suspension with approximately 100 CFU was spread onto CHROMagar™ ECC plates using an L-shaped spreader (product number 14-665-230, Fisher Scientific, Hampton, NH, USA). Next, the growth plate 14 was covered with its lid, inverted, and placed on the TFT image sensor 20 together with the entire imaging system 12 inside an incubator 16 (product number 151030513, ThermoFisher Scientific, Waltham, MA, USA) maintained at 37 ± 0.2 °C.

[0034] Additionally, CHROMagar™ ECC plates were prepared in advance using the following method. CHROMagar™ ECC (6.56 g) was mixed with 200 mL of reagent-grade water (product number 23-249-581, Fisher Scientific, Hampton, NH, USA). The mixture was then heated to 100 °C on a hot plate while being stirred periodically using a magnetic stir bar. After cooling the mixture to approximately 50 °C, 10 mL of the mixture was dispensed into each Petri dish (60 mm × 15 mm) (product number FB0875713A, Fisher Scientific, Hampton, NH, USA). When the agar plates solidified, the agar plates were sealed using parafilm (product number 13-374-16, Fisher Scientific, Hampton, NH, USA) and covered with aluminum foil to keep them in the dark before use. These plates were stored at 4 °C and used within two weeks after preparation.

[0035] Imaging Setup The on-site portable CFU imager 12 includes an illumination module containing a light source 18 and a TFT-based image sensor 20. The light from the three-color LED light source 18 directly illuminates the sample 110 and forms an in-line hologram on the TFT image sensor 20 (JDI, Japan Display Inc., Japan). The TFT module includes a control printed circuit board (PCB) that provides illumination and image capture control signals and an image sensor 20 (80×84 pixels, pixel size = 375 μm). For the illumination module, the three-color LED (EDGELEC) is controlled by a microcontroller 26 (Arduino Micro, Arduino LLC) through a constant current LED driver (TLC5916, Texas Instrument, TX, USA) to continuously provide red (620 nm), green (520 nm), and blue (420 nm) illumination beams. The microcontroller 26, the LED driver, and the three-color LED are all integrated on a single PCB, and the single PCB is powered by a 5V-1A voltage adapter and communicates with the TFT PCB through the LED power signal.

[0036] The illumination light passes through the transparent solid agar and forms a lens-free image of the growing bacterial colonies on the TFT image sensor 20. The distance between the LED and the sample (in other words, the z 1 distance shown in Fig. 2C) is approximately 15.5 cm, which is large enough to uniformly cover the entire sample surface with the illumination light. The distance (z 2 ) between the sample 110 and the TFT sensor 20 is approximately 5 mm, which is approximately equal to the thickness of the solid agar. The mechanical support material for the PCB, the sample, and the sensor was custom-made using a 3D printer (Objet30 Pro, Stratasys, Minnesota, USA).

[0037] Image data acquisition Time-lapse imaging experiments were conducted to collect data for both the training phase and the test phase. The CFU imaging modality captured 70 h of time-lapse images of the agar plates under test every 5 minutes under red, green, and blue illumination. A control program 28 with a graphical user interface (GUI) 94 was developed to automatically perform illumination switching and image capture. The raw TFT hologram images 70 h were saved in a 12-bit format. After the experiment was completed, the samples were disposed of as solid biomedical waste. In total, 70 h of time-lapse TFT hologram images of 889 E. coli colonies from 17 independent experiments were collected to initially train the CFU detection neural network model. In addition to this, 442 bacterial colonies (128 E. coli, 126 Citrobacter, and 188 Klebsiella pneumoniae) were populated from 17 new agar plates and used to train (1) the final CFU detection neural network 90 (through transfer learning from the initial detection model) and (2) the CFU classification neural network 92. A third independent dataset of 265 colonies from 13 new experiments was used to test the trained neural network models blindly.

[0038] Bacterial colony candidate selection The overall candidate selection workflow consists of image preprocessing, difference analysis, colony mask segmentation, and candidate position localization, following the operations (operations a - i) illustrated in FIG. 6. For each time point, three raw TFT images 70 h (red, green, and blue channels) were obtained over a FOV of approximately 10 cm 2 . The TFT image I N_raw,C , where N refers to the Nth image obtained at T N , and C represents the color channel, i.e., R (red), G (green), and B (blue), after which a series of preprocessing operations were performed to enhance the image contrast. First, as shown in operations a - b in FIG. 6, the image was interpolated by a factor of 5 and T 0It was normalized by directly subtracting the first frame. After this normalization step, the background region had a nearly zero signal, and the region representing the growing colony had negative values because the colony partially blocked and scattered the illumination light. Then, by adding 127 and saving the image as an unsigned 8-bit integer array, T N the current frame in N was scaled to 0 - 127 and denoted as I N_norm,C As per operations b - c in Figure 6, I N_norm,C was averaged as shown in Equation (1) to perform smoothing in the time domain, resulting in I N_denoised,C .

[0039] TIFF2025517633000002.tif12170

[0040] To further improve the sensitivity of the system, the difference image I N_diff averaged over three color channels was calculated as follows.

[0041] TIFF2025517633000003.tif9170

[0042] By this operation, the signal of static artifacts was suppressed, and the spatio-temporal signal of the growing colony was enhanced as a ring-shaped pattern. Next, the minimum intensity projection for each pixel was performed from the difference image of I (N-7)_diff to project the minimum intensity onto I N_diff to obtain the image I N_projection . Following this step, using an empirically set intensity threshold, I N_projection was segmented into a binary mask. After morphological operations to fill the ring-shaped pattern and watershed region-based division of the clustered regions, M N was obtained as presented in operation g of Figure 6. This binary mask, i.e., M NBased on this, the connected components were extracted and their centroids were localized as shown in operation h of FIG. 6. These centroid coordinates were updated dynamically for each time point to ensure maintaining localization at the center of the growing colony.

[0043] Despite this preprocessing of the acquired TFT image 70h, there still exist some time-varying non-colony objects that can be selected as false colony candidates (such as bubbles, dust, or other features created by uncontrolled movement on the agar surface). Therefore, the deep neural network 90 was trained to further screen each colony candidate to eliminate false positives, the details of which will be considered in the next subsection.

[0044] DNN-based Detection of Bacterial Colony Growth I N_denoised,CThe time-lapse videos 204 of each colony candidate region over eight frames were cropped as shown in operation i of FIG. 6. These videos 204 were then upsampled in the spatial domain and compiled as a 4D array (3×8×160×160, or in other words, color channel×number of frames×x×y) fed to the CFU detection neural network 90, which adopted the architecture of a dense net but with 2D convolutional layers replaced by pseudo-3D convolutional layers (see FIG. 7). The weights of this CFU detection DNN 90 were initialized using a pre-trained model obtained using the E. coli CFU dataset with a single illumination wavelength of 515 nm. This pre-trained model was obtained using a total of 889 colonies (positive) and 159 non-colony objects (negative) from 17 independent agar plates. This initial neural network model was then transferred to multiple wavelength illumination image datasets using 442 new colonies and 135 non-colony objects from another 17 independent agar plates. Both the positive and negative image datasets were augmented over a time domain with different start and end points, resulting in over 10,000 videos used for training. A 5-fold cross-validation strategy was adopted to select the best combination of hyperparameters. Once the hyperparameters were determined, all the collected data was used to train to complete the CFU detection neural network 90. Data augmentation such as flipping and rotation was also applied when loading the training dataset.

[0045] The network model 90 was optimized using the Adam optimizer with a momentum coefficient of (0.9, 0.999). The learning rate was 1×10 -4It started as such, and the scheduler was used to decrease the learning rate by a factor of 0.8 every 10 epochs. The batch size was set to 8. The loss function was selected as follows.

[0046] TIFF2025517633000004.tif12170

[0047] Here, p is the network output which is the probability of each classification before the SoftMax layer, g is the ground truth label (equal to 0 or 1 for binary classification), K is the total number of training samples in one batch, and w is the weight assigned to each classification, defined as w = 1 - d (where d is the percentage of samples in one classification). The training process was implemented using a GPU (GTX1080Ti) and took about 5 hours to converge. Using a decision threshold of 0.5, the CFU detection neural network 90 converged with a sensitivity of 92.6% and a specificity of 95.8%. In the test phase, the decision threshold was set to be 0.99, which achieved 100% specificity.

[0048] DNN-based classification of Escherichia coli and other total coliform colonies To classify the species of the detected bacterial colonies, a second DNN-based classifier 92 was constructed. The CFU classification neural network 92 was trained on the same multi-wavelength dataset populated with 442 colonies (128 Escherichia coli colonies, 126 Citrobacter colonies, and 188 Klebsiella pneumoniae colonies). The input to the classification DNN 92 was organized into a 4D array (3×8×160×160, in other words, color channel × number of frames × x × y), but different normalization methods were used. Different from the background subtraction normalization adopted for the CFU detection neural network 92, for the classification DNN 92, the network input was at the first time point T 0It was renormalized by dividing by the background intensity obtained in []. This division-based normalization was performed on the three color channels such that the background was normalized to approximately 1 in the three channels, revealing the white in the background. Through this operation, color variations across different experiments were minimized, improving the generalization ability of the classification DNN92.

[0049] The network structure of the classification DNN92 was the same as that of the CFU detection network 90, but there were some differences in hyperparameter selection (see Figure 7). The classification neural network model was randomly initialized and optimized using the Adam optimizer with a momentum coefficient of (0.9, 0.999). The learning rate started at 1×10 -3 and the scheduler was used to decrease the learning rate by a factor of 0.7 every 30 epochs. Also, the batch size was set to 8. The classification neural network also used the weighted cross-entropy loss function shown in Equation (3). The training process was carried out using a GPU (GTX1080Ti) and took approximately 5 hours to converge. A decision threshold of 0.5 was used to classify Escherichia coli colonies and all other total coliform colonies in the training process, achieving accuracies of 91% and 97% respectively. In the test phase, the decision threshold was set to 0.8, which achieved 100% classification accuracy. Additionally, a colony size threshold of 4.5mm 2 was used in the test phase to ensure that only colonies large enough to distinguish their species passed through the classification network 92.

[0050] Embodiments of the invention have been shown and described, but various modifications can be made without departing from the scope of the invention. For example, multiple TFT-based image sensors can be used to perform detection and sorting over a larger area or different growth plates. The invention, therefore, should not be limited except by the following claims and their equivalents.

Claims

1. A system for detecting and classifying living microorganisms and / or their colonies in a sample using time-lapse imaging, comprising: a light source; a thin-film transistor (TFT)-based image sensor disposed along an optical path generated from the light source; a growth plate inserted along the optical path and disposed adjacent to the TFT-based image sensor, the growth plate containing a growth medium thereon and containing the sample; a microcontroller or other circuit configured to periodically illuminate the growth plate with light from the light source and capture time-lapse images of microorganisms and / or their colonies on the growth plate using the TFT-based image sensor; a computing device configured to execute image processing software for processing and analyzing the time-lapse images of the microorganisms and / or their colonies on the growth plate and detecting candidate microorganisms and / or their colonies in the time-lapse images A system comprising.

2. The system according to claim 1, further comprising an incubator integrated with the light source, the TFT-based image sensor, and the growth plate.

3. The system according to claim 1, wherein the light source comprises one or more selectively actuated spectral bands.

4. The image processing software is configured to receive the captured time-lapse images of the microorganisms and / or their colonies on the growth plate, and the image processing software: (1) uses a first trained deep neural network trained to detect true microorganisms and / or their colonies from non-microbial objects to detect candidate microorganisms and / or their colonies in the time-lapse images; and (2) uses a second trained deep neural network that receives as input at least one time-lapse image or at least one digitally processed time-lapse image of the true microorganisms and / or their colonies to output a species classification associated with the detected true microorganisms and / or their colonies. The system according to claim 1, configured to perform.

5. The system according to any one of claims 1 to 4, wherein the microorganism comprises a prokaryotic cell, a eukaryotic cell, a bacterium, a fungus, a virus, a multicellular organism, or a cluster, membrane, or colony thereof.

6. The system according to claim 1, wherein the computing device comprises a local and / or remote computing device.

7. The system according to claim 1, wherein a lens or a set of lenses is used to magnify or reduce a hologram of the microorganism and / or its colony on the TFT-based image sensor.

8. The TFT-based image sensor captures a field of view of at least 10 cm ２ The system according to any one of claims 1 to 7, wherein the field of view is captured.

9. The system according to claim 1, wherein the TFT-based sensor is integrated on or within the growth plate.

10. The system according to claim 1, wherein the TFT-based sensor is disposable.

11. The system according to claim 1, wherein the growth medium comprises a chromogenic agar plate.

12. A method of using the system according to claim 1, comprising: placing the growth plate comprising the sample in the optical path; periodically illuminating the growth plate using the light source, wherein the periodic illumination comprises continuously illuminating the growth plate in one or more spectral bands of illumination; obtaining a plurality of time-lapse images of the microorganism and / or its colony on the growth plate; and.

13. Processing the time-lapse images of the microorganism and / or its colonies on the growth plate using image processing software, wherein the image processing software is further configured to detect candidate microorganisms and / or their colonies in the time-lapse images based on differential image analysis in time-lapse holographic images, and a first trained deep neural network trained to detect true microorganisms and / or their colonies from non-microbial objects, and a second trained deep neural network that receives as input at least one time-lapse image or at least one digitally processed time-lapse image of the true microorganism and / or its colonies and outputs a species classification associated with the detected true microorganism and / or its colonies, further comprising processing the time-lapse images, the method according to claim 12.

14. The method according to claim 13, wherein the microorganism comprises a prokaryotic cell, a eukaryotic cell, a bacterium, a fungus, a virus, a multicellular organism, or a cluster, membrane, or colony thereof.

15. The method according to claim 12, wherein the sample comprises one or more of a water sample, a food sample, a biological or other fluid sample.

16. A method for detecting and classifying living microorganisms and / or their colonies using time-lapse imaging, comprising: Providing a growth plate containing a growth medium thereon and containing a sample; Periodically illuminating the growth plate with illumination light of at least one spectral band from a light source; Capturing a time-lapse image of the microorganism and / or its colonies on the growth plate using a TFT-based image sensor. An image processing software including a first trained deep neural network trained to detect true microorganisms and / or colonies thereof from non-microbial objects, and a second trained deep neural network that receives at least one time-lapse image or digitally processed time-lapse image as an input and outputs a species classification associated with the detected true microorganisms and / or colonies thereof, is used to detect candidate microorganisms and / or colonies thereof in the time-lapse image A method comprising the above. **Claim 17** The method according to claim 16, wherein the microorganism comprises a prokaryotic cell, a eukaryotic cell, a bacterium, a fungus, a virus, a multicellular organism, or a cluster, membrane, or colony thereof. **Claim 18** The method according to claim 16, wherein the time-lapse image is obtained several times per hour over several hours. **Claim 19** The method according to claim 16, wherein the TFT-based image sensor uses a lens or a set of lenses to capture an enlarged or reduced hologram of a microbial object and / or a microbial colony thereof.

Images