Systems and methods for detection and classification of living microorganisms using thin film transistor (TFT) image sensors and deep learning

Abstract

A bacterial colony forming unit (CFU) detection system is disclosed that utilizes a thin film transistor (TFT)-based image sensor array that saves about 12 hours compared to methods approved by Environmental Protection Agency (EPA). Lensless imaging modes are constructed using TFT image sensors having a sample field of view of about 10 cm2. And automatically collecting delay images of bacterial colonies cultured on the chromogenic agar plate every 5 minutes. Two deep neural networks are used to detect and count colonies in growth and identify their species. When 265 escherichia coli and other coliforms (namely citrobacter and klebsiella pneumoniae) colonies are used for blind detection, the average CFU detection rate of the system reaches 97.3% when the system is cultured for 9 hours, and the average recovery rate of the system reaches 91.6% when the system is cultured for about 12 hours. The TFT-based sensor can be applied to various microbiological detection methods. The imaging field of view of the platform can be economically and efficiently increased to more than 100cm < 2 >.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 338,972, filed on May 6, 2022, which is incorporated herein by reference. Priority is claimed under 35 U.S.C. §119 and any other applicable statutes.

[0003] The technical field generally relates to early screening and detection methods for detecting and / or identifying living microorganisms, such as cells (prokaryotic or eukaryotic), viruses, fungi, bacteria, yeasts, and multicellular organisms. More specifically, the technical field relates to systems and methods for periodically capturing holographic microscopic images of bacterial growth on a growth plate and automatically analyzing these time-lapse spatiotemporal patterns or holograms using multiple deep neural networks to rapidly detect and / or classify the corresponding microbial species. Background Art

[0004] In developed and developing countries, bacterial infection has become the main factor causing millions of deaths every year. Only in the U.S., the annual treatment costs of bacterial infection are more than 4 billion U.S. dollars. Therefore, in terms of preventing this infection caused by the pollution of food and drinking water, it is very important to detect pathogenic bacteria quickly and accurately for human health. Among these pathogenic bacteria, Escherichia coli and other coliforms are the most common, and they show that there is fecal contamination in food and water samples. The most basic and most commonly used method for detecting Escherichia coli and total coliforms includes culturing samples on solid agar plates or liquid culture media according to the scheme (for example, EPA 1103.1 and EPA 1604 methods) approved by the U.S. Environmental Protection Agency (EPA). However, these traditional methods based on culture usually need 24 hours or more to finally read the results, and need microbiologists to visually identify and count colony forming units (CFU). Although various nucleic acid-based molecular assays have been developed for rapid bacterial detection that provide results in less than a few hours, these methods generally exhibit low sensitivity and present challenges in distinguishing live from dead bacteria; in fact, there are no EPA-approved nucleic acid-based coliform detection methods for screening water samples.

[0005] Various other methods have been developed to provide high sensitivity and specificity for the detection of bacteria based on different approaches, e.g., fluorimetry, solid-phase cytometry, fluorescence microscopy, Raman spectroscopy, etc.; however, these systems are generally not applicable to large sample volumes (e.g., >0.1 L). As another alternative, Wang et al. demonstrated a time-lapse imaging platform based on complementary metal oxide semiconductor (CMOS) image sensors for early detection and classification of coliforms. See alsoWang, H. et al., A. Early Detection and Classification of Live Bacteria Using Time-Lapse Coherent Imaging and Deep Learning, Light Sci. Appl. 2020, 9(1), 118, https: / / doi.org / 10.1038 / s41377-020-00358-9. This approach saves more than 12 hours of detection time and provides more than 80% species classification accuracy within a 12-hour incubation time. In this design, the field of view (FOV) of the CMOS image sensor is less than 0.3 cm 2 , so the culture dish area needs to be mechanically scanned to obtain an image of the entire FOV of the culture sample. This is not only time-consuming and requires additional sample scanning hardware, but also brings some additional digital processing burden for image registration and stitching.

[0006] In recent years, with the rapid development of thin film transistors (TFTs), TFT technology has been widely used in the flexible display industry, radio frequency identification tags, ultrathin electronic devices and large-scale sensor fields due to its high scalability, low-cost large-scale production (including, for example, roll-to-roll manufacturing), low power consumption and low heat generation characteristics. TFT technology is also applied in the field of biosensing to detect pathogens by converting, for example, antibody-antigen binding, enzyme-substrate catalytic activity or DNA hybridization into electrical signals. For example, Hu et al. developed a low-cost TFT nanobelt sensor to detect gene copies of Escherichia coli and Klebsiella pneumoniae (Klebsiella) within a few minutes using pH changes caused by DNA amplification. See also Hu, C. et al., Ultra-Fast Electronic Detection of Antimicrobial Resistance Genes Using Isothermal Amplification and Thin Film Transistor Sensors, Biosens. Bioelectron. 2017, 96, 281-287, https: / / doi.org / 10.1016 / j.bios.2017.05.016. As another example, Salinas et al. implemented a ZnO TFT biosensor for real-time Escherichia coli detection using a recyclable plastic substrate. See alsoSatinas, RA et al., Biosensors Based on Zinc Oxide Thin-Film Transistors Using Recyclable Plastic Substrates as an Alternative for Real-Time Pathogen Detection, Taianta 2022, 237, 122970. However, these TFT-based biosensing methods cannot distinguish between live and dead bacteria and cannot provide quantification of the CFU concentration of the tested samples. Summary of the invention

[0007] Here, TFT-based image sensors are used to build a real-time CFU detection system to automatically count bacterial colonies and quickly identify their species using deep learning. 2 or larger), so no mechanical scanning of the agar plate is required, which allowed us to create a field-portable and cost-effective lens-free CFU detector, such as FIG. 2A to FIG. 2C This compact system consists of sequentially switched red, green, and blue light-emitting diodes (LEDs) that periodically illuminate the cultured samples (Escherichia coli, Citrobacter rodentium, and Klebsiella pneumoniae), as shown in Figure 1. Figure 2C As shown, the spatiotemporal pattern of the sample is collected by a TFT image sensor with an imaging cycle of 5 minutes. Two deep learning-based classifiers were trained to detect bacterial colonies and then classified them into Escherichia coli and total coliforms. A blind test was conducted on a dataset containing 265 colonies (85 CFU Escherichia coli, 66 CFU Citrobacter rodentium, and 114 CFU Klebsiella pneumoniae). The TFT-based system was able to detect the presence of colonies as early as about 6 hours of the culture period and achieved an average CFU detection rate of 97.3% at 9 hours of culture, saving more than 12 hours compared to the EPA-approved culture-based CFU detection method. For the classification of detected bacterial colonies, an average recovery rate of 91.6% was achieved at approximately 12 hours of culture.

[0008] This TFT-based field-portable CFU detection system greatly benefits from the cost-effectiveness and ultra-large FOV of TFT image sensors, which can be further scaled up to achieve lower costs with larger FOVs, for example, based on roll-to-roll manufacturing methods commonly used in the flexible display industry. In some embodiments, the TFT image sensor can be integrated with each agar plate to be tested and can be discarded after the CFU count is determined, which provides a variety of new opportunities for microbiological instrumentation in laboratory and field environments.

[0009] In one embodiment, a system for detecting and classifying live microorganisms and / or their colonies in a sample using time-lapse imaging is disclosed. The system includes a light source and a thin film transistor (TFT)-based image sensor, the TFT-based image sensor being positioned along a light path originating from the light source. A growth plate containing a growth medium and a sample thereon is inserted along the light path and disposed adjacent to the TFT-based image sensor. A microcontroller or other circuit in the system is configured to periodically illuminate the growth plate with light from the light source, and to capture time-lapse images of microorganisms and / or their colonies on the growth plate with a TFT-based image sensor. The system includes a computing device configured to execute image processing software to process and analyze time-lapse images of microorganisms and / or their colonies on the growth plate, and to detect candidate microorganisms and / or their colonies in the time-lapse images.

[0010] In another embodiment, a method for detecting and classifying live microorganisms and / or their colonies using time-lapse imaging is disclosed. The method includes providing a growth plate having an agar medium thereon and containing a sample; periodically illuminating the growth plate with at least one spectral band of illumination light from a light source; capturing time-lapse images of microorganisms and / or their colonies on the growth plate with a TFT-based image sensor; and detecting candidate microorganisms and / or their colonies in the time-lapse images with image processing software, the image processing software including a first trained deep neural network and a second trained deep neural network, the first trained deep neural network being trained to detect real microorganisms and / or their colonies from non-microorganism objects, the second trained deep neural network receiving at least one time-lapse image or a digitally processed time-lapse image as input, and outputting a species classification associated with the detected real microorganisms and / or their colonies. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic representation of a system for early detection and classification of live microorganisms and / or their colonies in a sample using time-lapse imaging and deep learning.

[0012] FIG. 2A to FIG. 2C A real-time CFU detection and classification system using a TFT image sensor is shown. Figure 2APhoto images of the lensless imaging system, the sample to be tested, and the laptop computer used to control the hardware are shown. The chromogenic agar medium causes E. coli colonies to appear gray-green, while other coliform bacteria appear pink; in addition, when other types of bacteria are present in the sample, the growth of different bacterial colonies is inhibited or colorless colonies are displayed. Figure 2B A magnified photograph of a TFT image sensor with a FOV of 32 mm x 30 mm is shown. Figure 2C A detailed illustration of the lens-free imaging mode is shown. The red (620nm), green (520nm), and blue (460nm) LEDs are sequentially turned on at 5-minute intervals to directly illuminate the cultured samples, which are imaged by the TFT image sensor in a single shot. The distance between the three-color LEDs and the agar plate sample (z1) is 15.5cm, while the sample-to-sensor distance (z2) is about 5mm.

[0013] Figure 3 A schematic diagram of the workflow of a deep learning-based CFU detection and classification system is shown. Eight (8) full FOV RGB images are processed at 20 minute intervals for differential analysis to select initial colony "candidates". A digitally cropped 8-frame RGB image sequence of each individual colony candidate is first fed into the CFU detection neural network. The neural network rejects various non-colony objects (in the initial colony candidates), such as dust and bubbles, to achieve true colony detection. Next, the detected colonies are passed through the CFU classification neural network to identify their species (E. coli or other total coliforms, i.e., binary classification).

[0014] FIG. 4A to FIG. 4B Shown is the visual assessment of early detection and classification of coliform bacteria colonies using TFT image sensors. Figure 4A Full FOV color images of E. coli cultured for 11 hours, Citrobacter rodentium cultured for 13 hours, and K. pneumoniae cultured for 11 hours are shown. Figure 4B An example of an image sequence of each isolated colony growth is shown. Three independent colony growth sequences were selected for each bacterium. The dashed box marks the first colony detection time confirmed by the CFU detection neural network, and the dashed box corresponds to the first classification time correctly predicted by the CFU classification neural network.

[0015] FIG. 5A to FIG. 5F Quantitative performance evaluation of early detection and classification of coliform bacteria using TFT image sensors is shown. Figure 5A , Figure 5C , Figure 5EThe change of colony detection rate of Escherichia coli, Citrobacter rodentium and Klebsiella pneumoniae with culture time is shown. The average value and standard deviation of the detection rate were calculated for 85 Escherichia coli colonies, 66 Citrobacter rodentium colonies and 114 Klebsiella pneumoniae colonies at each time point. Figure 5B , Figure 5D , Fig. 5F The colony recovery rates of E. coli, Citrobacter rodentium and K. pneumoniae as a function of culture time are shown. The mean and standard deviation of the recovery rates were calculated for 85 E. coli colonies, 66 Citrobacter rodentium colonies and 114 K. pneumoniae colonies at each time point.

[0016] Figure 6 The colony candidate generation workflow (step ai) is shown. The acquired TFT images are subjected to image preprocessing step ai to select colony candidates; then, the cropped video of the colony candidates is passed through the trained CFU detection neural network to determine true positives and eliminate false positives.

[0017] Figure 7 The network architectures of the CFU detection neural network and the CFU classification neural network are shown. A dense network design is adopted here, replacing the 2D convolutional layers with pseudo 3D convolutional blocks. The CFU detection and classification neural networks share the same architecture, but the hyperparameters [m, n, p, q] are chosen differently, as Figure 7 shown. DETAILED DESCRIPTION

[0018] Figure 1 A system 10 for early detection and classification of live microorganisms and / or colonies of live microorganisms in a sample 110 using time-lapse imaging and deep learning according to one embodiment is shown. Microorganisms include prokaryotic cells, eukaryotic cells (e.g., stem cells), fungi, bacteria, viruses, multicellular organisms (e.g., parasites), or clusters or membranes or colonies thereof. The system 10 includes a holographic imaging device 12 (see also FIG. 2A to FIG. 2C ), the holographic imaging device is used to obtain a time-lapse image 70h of microbial growth occurring on one or more growth plates 14 (e.g., a petri dish containing chromogenic agar as a solid growth medium plus nutrients for culturing the microorganisms or other growth medium appropriate for the type of microorganism). The image 70h contains a spatiotemporal pattern (e.g., a hologram) of the sample 110.

[0019] The holographic imaging device 12 includes a light source 18 for directing light onto the sample 110. As described herein, the light source 18 may include a three-color LED module that sequentially switches red, green, and blue light emitting diodes (LEDs). In alternative embodiments, other selectively activated spectral bands may be used in the light source 18. The holographic imaging device 12 also includes a TFT-based image sensor 20 that is disposed along the optical path of the light emitted from the light source 18. Figure 2A and Figure 2C As shown, the holographic imaging device 12 includes a frame or housing 13, wherein a light source 18 is located at one end (e.g., the top) and a TFT-based image sensor 20 is located at an opposite end (e.g., the bottom). The growth plate 14 containing the sample 110 on the growth plate 14 is then inserted into the optical path between the light source 18 and the TFT-based image sensor 20. In some embodiments, the growth plate 14 can be placed directly on the TFT-based image sensor 20. In other embodiments, the growth plate 14 can contain a TFT-based image sensor 20 directly on the growth plate 14 or within the growth plate. The TFT-based image sensor 20 can be reusable or, in some embodiments, disposable. An optional lens or set of lenses (not shown) can be positioned along the optical path and used to magnify or reduce the hologram captured by the TFT-based image sensor 20. The distance between the light source 18 and the sample 110 (i.e., Figure 2C The z1 distance shown is significantly greater (>>) than the distance (z2) between the sample 110 and the TFT sensor 20. For example, in one embodiment, the z1 distance is about 15.5 cm and the z2 distance is about 5 mm.

[0020] In some embodiments, the holographic imaging device 12 may include an incubator 16 to heat one or more growth plates 14 and / or maintain the temperature at an optimal set point temperature or temperature range for microbial growth. A separate incubator 16 may also be used with the holographic imaging device 12. In one embodiment, the incubator 16 may include an optically transparent plate or substrate containing a heating element for adjusting the temperature of one or more growth plates 14. The incubator 16 may also include a fully or partially enclosed housing that houses the holographic imaging device 12 and one or more growth plates 14. The holographic imaging device 12 may also include one or more optional humidity control units 17 that are used to maintain one or more growth plates 14 at a set point humidity level or range. The humidity control unit 17 may be integrated with the incubator 16, the holographic imaging device 12, or a separate component.

[0021] A series of time-lapse images 70h of microorganisms and / or their colonies on the growth plate 14 are used to identify microorganism colony candidates based on difference images (i.e., time-lapse images) obtained over time. The difference images (images 70h obtained at different times) include images of growing microorganisms and / or colonies, but also include non-microorganism objects, such as dust, water bubbles, or surface movement of agar itself and other artifacts. The image processing software 80 executed on a computing device 82 with one or more processors 84 is used to perform image preprocessing, difference analysis, colony mask segmentation and candidate position positioning, and cropping of colony candidate videos. However, some of the videos of these colony candidates are not real microorganism colonies, but may represent inanimate objects or artifacts that need to be covered or excluded, such as bubbles, dust, etc. As explained herein, the image processing software 80 uses a first trained deep neural network (DNN) 90 to detect actual microorganisms and / or colonies and ignore non-microorganism objects. Once "real" microorganisms and / or colonies are selected, one or more time-lapse images and / or at least one digitally processed time-lapse image (e.g., a renormalized image generated by partition-based normalization as explained herein) are sent to a second trained deep neural network (DNN) 92, which is used to classify the species category or specific species of the microorganisms and / or colonies.

[0022] The system 10 is implemented with a holographic imaging device 12, which includes a holographic imaging system that captures holographic images of growing microorganisms and / or colonies. A light source 18 (e.g., an illumination module including a three-color light emitting diode (LED)) illuminates the microorganisms and / or colonies thereof on one or more growth plates 14 (cultured using an incubator 16), and captures a holographic image 70h of the microorganisms and / or colonies thereof using at least one TFT-based image sensor 20. The light source 18 preferably emits one or more illumination spectral bands, which can be activated (e.g., turned on / off) as needed. This can be achieved by different spectral bands emitted by the light source 18 or by using filters that allow different spectral bands to pass. The holographic imaging device 12 can be placed in a separate incubator 16, or the holographic imaging device 12 can be integrated with the incubator 16. It is noteworthy that there is no need to scan one or more growth plates 14. The TFT-based image sensor 20 captures a large field of view (FOV). In some embodiments, a lens or a group of lenses is used to capture an even larger field of view of one or more growth plates 14. Alternatively, a larger TFT-based image sensor 20 may be used. In a preferred embodiment, the captured FOV is at least 10 cm 2 or larger. Even larger FOVs are envisioned, including 100cm 2 or larger FOV.

[0023] A microcontroller or control circuit 26 is provided for controlling the illumination of the light source 18, the incubator 16, the humidity control unit 17, and capturing images with the TFT-based image sensor 20. The microcontroller or control circuit 26 may also communicate with the computing device 82, for example, receiving instructions and / or sending data using a program 28 executed by the computing device 82. The microcontroller or control circuit 26 may include one or more microprocessors, drivers, etc. located on a printed circuit board (PCB) for operating various subsystems and transmitting data. This includes the timing and sequence of illumination with the light source 18, acquiring images from the TFT-based image sensor 20, etc. The microcontroller or control circuit 26 may also be used to control the set temperature or temperature range of the incubator 16. The control circuit 26 may also be used to control the set point humidity level or humidity range of the incubator 16 using the humidity control unit 17. As Figure 2A As shown, the microcontroller or control circuit 26 may be located external to the frame 13, or alternatively, may be contained therein.

[0024] The system 10 includes at least one computing device 82 (e.g., a personal computer, laptop, tablet PC, server, etc.) having one or more processors 84 for executing image processing software 80 to process the image 70h obtained from the TFT-based image sensor 20. The computing device 82 can be located with the holographic imaging device 12 (e.g., a local implementation), or can be remote from the holographic imaging device (e.g., a remote computing device, such as a server). In other embodiments, multiple such computing devices 82 can be used (e.g., one controls the holographic imaging device 12 and another processes the image 70h). In addition, in some embodiments, the computing device 82 is capable of controlling various aspects of the operation of the holographic imaging device 12 using a microcontroller or control circuit 26. For example, using a graphical user interface (GUI) 94 that can be viewed on the display 83, a user can control various aspects of the system 10 (e.g., the period or timing of image scans, the operation of the TFT-based image sensor 20, the temperature control of the incubator 16, the transmission of the image file 70h from the TFT-based image sensor 20 to the computing device 82, etc.). The GUI 94 may also be used to display video, classified colonies 102, colony counts, and display colony growth graphs for observation / interaction.

[0025] The computing device 82 executes image processing software 80, which includes a microorganism and / or colony detection deep neural network 90, which is used to identify real microorganisms and / or colonies from other non-microorganism artifacts (e.g., dust, bubbles, spots, etc.). The computing device 82 also executes a separate classification deep neural network 92 in the image processing software 80 to classify specific species categories or actual species of microorganisms and / or colonies. In an alternative embodiment, the functions of the first trained deep neural network 90 and the second trained deep neural network 92 are combined into a single trained deep neural network (e.g., deep neural network 90). Microorganisms and / or colonies of multiple different species can be identified in a single sample. In a specific embodiment, the system 10 is capable of rapidly detecting Escherichia coli and total coliforms (i.e., Klebsiella and Klebsiella pneumoniae subspecies pneumoniae) in water samples. This automated and cost-effective live microorganism detection system 10 is revolutionary for a wide range of applications in microbiology, significantly reducing detection time and automatically identifying microorganisms and / or colonies without the need for marking or experts.

[0026] To use system 10, a sample 110 is obtained and the sample is optionally subjected to a signal amplification operation in which the sample is mixed with a growth medium 112 ( Figure 1 ) is pre-cultured at an elevated temperature for a period of time and then filtered using, for example, a filter membrane. The sample 110 is typically a fluid and may include, for example, a water sample (although it may be a food sample, biological or other fluid sample). The filter membrane is then physically contacted with one or more growth plates 14 (e.g., the agar surface of the growth plate 14) under light pressure for a period of time to transfer the microorganisms (e.g., bacteria) to the agar growth medium in the growth plate 14 and then removed. However, in other embodiments, the sample 110 may also be placed directly on the growth plate 14 and applied using, for example, an L-shaped applicator disclosed herein. Then, one or more growth plates 14 are covered in a holographic imaging device 12 placed in / on an incubator 16 (e.g., the agar surface faces the TFT-based image sensor 20, upside down). Then, the growth plate 14 with the sample 110 is allowed to culture for several hours and is periodically imaged by the TFT-based image sensor 20. In some embodiments, a single growth plate 14 is imaged by a TFT-based image sensor 20. In other embodiments, multiple growth plates 14 are imaged by a TFT-based image sensor 20. In the latter embodiments, multiple TFT-based image sensors 20 may be used. In some embodiments, the TFT-based image sensor 20 is separate from the growth plate 14. In other embodiments, the growth plate 14 may be integrated with the TFT-based image sensor 20. This may be located on the growth plate 14 or within the growth plate.

[0027] In one particular embodiment, a method for detecting and classifying live microorganisms and / or colonies thereof using time-lapse imaging includes loading a growth plate 14 containing a sample 110 into or onto an incubator 16. One or more growth plates 14 are then illuminated with light of different spectral bands (e.g., colors) from a light source 18. Specifically, the growth plates 14 are illuminated periodically in sequence by illumination light of different spectral bands (e.g., colored LEDs). The TFT-based image sensor 20 captures an image 70h of each color. Different cycles between consecutive illuminations may be used. In one embodiment, approximately five (5) minutes pass between illuminations of the sample 110. This enables time-lapse images 70h of the growth plate 14 containing microorganisms and / or colonies thereof to be taken using a holographic imaging device 12. The time-lapse images 70h are then processed, and a first trained deep neural network (DNN) 90 is used to detect (and optionally count) real microorganisms and / or colonies, such as Figure 3 shown.

[0028] Figure 3 An exemplary workflow of a deep learning based CFU detection and classification system is shown. Here, eight (8) full FOV RGB images are processed at 20 minute intervals for differential analysis 200 to select initial colony "candidates" for candidate generation 202. Of course, more or fewer images can be processed at different time intervals. First, each individual colony candidate ( Figure 3 A digitally cropped 8-frame RGB image sequence 204 (e.g., video) of the initial colony candidates (three such candidates are shown in FIG) is fed into the CFU detection neural network 90. ​​The neural network 90 rejects various non-colony objects (in the initial colony candidates), such as dust and bubbles (here, candidate 3), enabling true colony detection (candidates 1 and 2). Next, a colored image sequence 206 of the real detected colonies is passed through the CFU classification neural network 92 to identify their species (e.g., E. coli or other total coliforms, i.e., a binary classification). Finally, a second trained deep neural network (DNN) 92 is then used to classify (and optionally count) the detected microorganisms and / or colonies.

[0029] Figure 6 Further details are shown on how to use the differential analysis 200 to generate colony candidates 202, such as Figure 3As shown. In operation (a), the TFT sensor 20 with RGB channels captures the original time-lapse image. As shown in operation (b), background subtraction is performed to create a background subtracted image. Next, as shown in operation (c), the image is averaged in the time domain to smooth / denoise the image. In operation (d), as shown in operation (e), the obtained smoothed / denoised image and the differential stack of the RGB channels are merged (averaged). Next, in operation (f), a minimum projection image is generated, and threshold processing and morphological processing are performed on it to generate a rough detection mask, as shown in operation (g). As shown in operation (h), the local colony location is identified and the colony candidate is selected. After the colony candidate is selected, the video of the colony candidate in the RGB color channel is cropped into an RGB image sequence 204, as shown in operation (i).

[0030] Figure 7 The network architecture of the CFU detection neural network 90 and the CFU classification neural network 92 is shown. A dense network design is adopted here, and the 2D convolutional layers are replaced by pseudo 3D convolutional blocks. The CFU detection and classification neural networks 90, 92 share the same architecture, but the hyperparameters [m, n, p, q] are selected to be different, such as Figure 7 shown.

[0031] experiment

[0032] result

[0033] The success of the system 10 was demonstrated by detecting and classifying E. coli colonies and two other types of total coliforms (i.e., Citrobacter rodentium and Klebsiella pneumoniae) on chromogenic agar plates, which resulted in a gray-green color for E. coli colonies and a pink color for other E. coli colonies, and also inhibited the growth of different bacterial colonies when other types of bacteria were present in the sample. Each sample 110 was prepared using a petri dish 14 in accordance with the EPA-1103.1 method (see Methods). After the sample 110 was prepared, it was placed directly on top of the TFT-based image sensor 20 as part of the lensless imaging system 12, and the entire imaging mode (except for Figure 2A The laptop computer 82 (outside of the one in the incubator) was placed in the incubator 16 and the growth of the colonies was recorded at 5-minute imaging intervals. For each time interval, three images were collected sequentially for 70 h using the TFT image sensor 20 under red (620 nm), green (520 nm) and blue (460 nm) illumination light. This multi-wavelength design allows the monochrome TFT image sensor 20 to reconstruct a color image of the colonies and is primarily used to identify their species by utilizing the color information provided by the selectively chromogenic agar medium 112. Use Figure 3 and Figure 6The workflow shown processes the recorded time-lapse images 70h, where differential analysis 200 is used to generate initial colony candidates 202, and two deep neural networks (DNNs) 90, 92 are trained to further screen the colony candidates to specifically detect real colonies and infer their species category / species (see Methods section). All of these image processing steps take less than 25 seconds and consume less than 1GB of memory using a computer powered by an Intel Core i7-7700 CPU (no GPU required).

[0034] The presented TFT imaging system 10 periodically captures an image 70h of the agar plate 14 under test based on lensless coaxial holography; however, due to its large pixel size (375 μm) and relatively small sample-to-sensor distance (about 5 mm, equal to the thickness of agar), no free-space back-propagation step is required. By directly using the raw intensity image 70h as part of the RGB color channels and calibrating the background, a color image of the agar plate can be generated in less than 0.25 seconds after recording the TFT image. FIG. 4A to FIG. 4B Shown are examples of images (in color) of E. coli, Citrobacter rodentium, and K. pneumoniae colonies at different stages of growth captured by system 10. Consistent with EPA-approved methods (EPA-1103.1), using chromogenic agar, E. coli colonies appear gray-green, while Citrobacter rodentium and K. pneumoniae colonies appear pink.

[0035] based on FIG. 4A to FIG. 4B The imaging performance of the TFT-based CFU detection system 10 outlined in the paper is quantified for its early detection and classification performance, as FIG. 5A to FIG. 5F As shown. For this reason, the detection and classification neural network model is trained on the data set of 442 bacterium colonies (128 Escherichia coli colonies, 126 Citrobacter colonies and 188 Klebsiella pneumoniae colonies) captured from 17 independent experiments (see the method for training details). The test data set is filled with 265 bacterium colonies from 13 independent experiments, with a total of 85 Escherichia coli colonies, 66 Citrobacter colonies and 114 Klebsiella pneumoniae colonies. The detection rate is defined as the ratio of the true colony count confirmed by CFU detection neural network 90 in the total colony count counted by experts after 24 hours of cultivation. Figure 5A , Figure 5C , Figure 5E The detection rate achieved in the blind test phase varies with incubation time. Figure 5A , Figure 5C , Figure 5EAs shown, the detection rate was >90% after culturing E. coli for 8 hours, Citrobacter rodentium for 9 hours, and Klebsiella pneumoniae for 7 hours and 40 minutes. In addition, 100% detection rate was obtained after culturing E. coli for 10 hours, Citrobacter rodentium for 11 hours, and Klebsiella pneumoniae for 9 hours and 20 minutes. Compared with the standard readout time (24 hours) approved by the EPA, the TFT-based CFU detection system 10 saves more than 12 hours. In addition, from FIG. 5A to FIG. 5F From the detection rate curve reported in , it can also be qualitatively inferred that the colony growth rate of Klebsiella pneumoniae is greater than that of Escherichia coli, and that of Escherichia coli is greater than that of Citrobacter rodentium, because the earliest detection time of Escherichia coli, Citrobacter rodentium, and Klebsiella pneumoniae colonies is 6 hours, about 6.5 hours, and about 5.5 hours of culture time, respectively.

[0036] To quantify the performance of the bacterial colony classification neural network 92, the recovery rate was defined as the ratio of the number of correctly classified colonies to the total number of colonies counted by the expert after 24 h of incubation. Figure 5B , Figure 5D , Fig. 5F The recovery curves for all blind test experiments are shown as a function of incubation time. It can be seen that >85% recovery was achieved with 11 hours and 20 minutes of incubation for E. coli, 13 hours for Citrobacter rodentium, and 10 hours and 20 minutes for K. pneumoniae. It was difficult to achieve 100% recovery for all colonies because some late-growing "awake" colonies could not grow to a large enough size with correct color information even after 24 hours of incubation. FIG. 5A to FIG. 5F It also revealed a time delay of approximately 3 hours between the time of colony detection and the time of species identification; this time delay was expected because the detected colonies require more time to grow larger and provide discernible color information for the correct classification of their species.

[0037] discuss

[0038] Notice, FIG. 5A to FIG. 5F The results presented in represent the conservative performance of the TFT-based CFU detection method, because the ground truth colony information is obtained after 24 h of incubation. In the early stages of the incubation period, some bacterial colonies are not even physically present. Therefore, if the number of existing colonies at each time point is used as the ground truth, FIG. 5A to FIG. 5F Higher detection rates and recoveries can be reported.

[0039] In general, the performance of the TFT-based CFU detection system 10 is similar to that of the CMOS-based time-lapse holographic imaging method in terms of colony detection speed. However, due to its large pixel size (375 μm) and limited spatial resolution, the TFT-based method has a slightly delayed colony classification time. 2 ), the TFT-based CFU detection method eliminates (1) the time-consuming mechanical scanning of culture dishes and the associated optomechanical hardware, and (2) the image processing steps of image registration and stitching, both of which are required due to the limited FOV of CMOS-based imagers. In addition to saving image processing time, this also helps the system increase CFU detection sensitivity because the system 10 is free of any image registration and stitching artifacts and, therefore, can accurately capture subtle spatiotemporal changes in the agar caused by early bacterial colony growth. Due to the tremendous scalability of the TFT-based image sensor 20 array, the imaging FOV of the platform can be further increased to tens to hundreds of cm in a cost-effective manner. 2 , which can provide unprecedented imaging throughput levels for automated CFU detection using roll-to-roll manufacturing of, for example, TFTs, as employed in the flexible display industry.

[0040] Another significant advantage of the TFT-imager based detection system 10 is that it can be adapted to image a wide range of biological samples 110 using a cost-effective and field-portable interface. If the user has any contamination concerns, the Figure 2B , Figure 2C The TFT image sensor 20 shown in FIG. 1 can be used even in a disposable manner (e.g., integrated as part of a growth plate 14 (e.g., a culture dish)). In addition, the heat generated by the TFT image sensor 20 during data acquisition is negligible, thereby ensuring that the biological samples 110 can be grown at their desired temperature without being disturbed. Finally, the TFT-based CFU detection system 10 is user-friendly and easy to use because no complex optical alignment, high-precision mechanical scanning stages, or image registration / alignment steps are required.

[0041] The presented CFU detection system 10 using an array of TFT image sensors 20 provides a high throughput, cost effective, and easy to use solution to perform early detection and classification of bacterial colonies, offering unique opportunities for microbiological instrumentation in both laboratory and field environments.

[0042] Materials and methods

[0043] Sample preparation

[0044] All bacterial sample preparations were performed in a biosafety level 2 laboratory in accordance with UCLA environmental, health, and safety regulations. Escherichia coli (Migra) Castellani and Chalmers ( 25922 TM )、Citrobacter rodentium ( 43864 TM ) and Klebsiella pneumoniae subsp. pneumoniae (Schroeter) Trevisan ( 13883 TM ) is used to culture microorganisms. TM ECC (Product No. EF322, DRG International, Inc., Springfield, NJ, USA) chromogenic substrate mixture was used as a solid growth medium to detect E. coli and other total coliforms.

[0045] For each time-lapse imaging experiment, a bacterial suspension in phosphate buffered saline (PBS) was prepared from a 24-hour solid agar plate (Product No. 20-012-027, Fisher Scientific, Hampton, NH, USA). The concentration of the suspension was measured using a spectrophotometer (Model ND-ONE-W, Thermo Fisher). Serial dilutions were then performed in PBS to a final concentration of approximately 10 3 Approximately 100 μL of the diluted suspension with approximately 100 CFU / mL was spread on CHROMagar using an L-shaped spreader (Product No. 14-665-230, Fisher Scientific, Hampton, NH, USA). TM Next, the growth plate 14 is covered with a lid, turned upside down, and placed on the TFT image sensor 20, which together with the entire imaging system 12 is placed in an incubator 16 maintained at 37±0.2°C (Product No. 151030513, Thermo Fisher Scientific, Waltham, MA, USA).

[0046] In addition, prepare CHROMagar in advance using the following method TM ECC board. TMECC (6.56 g) was mixed with 200 mL of reagent grade water (Product No. 23-249-581, Fisher Scientific, Hampton, NH, USA). The mixture was then heated to 100 ° C on a hot plate while stirring regularly with a magnetic stirring bar. After the mixture was cooled to about 50 ° C, 10 mL of the mixture was dispensed into each Petri dish (60 mm × 15 mm) (Product No. FB0875713A, Fisher Scientific, Hampton, NH, USA). When the agar plates solidified, they were sealed with paraffin film (Product No. 13-374-16, Fisher Scientific, Hampton, NH, USA) and covered with aluminum foil to keep them in the dark before use. The plates were stored at 4 ° C and used within two weeks after preparation.

[0047] Imaging setup

[0048] The field portable CFU imager 12 includes an illumination module that includes a light source 18 and a TFT-based image sensor 20. Light from the tricolor LED light source 18 directly illuminates the sample 110 and forms a coaxial hologram on the TFT image sensor 20 (JDI, Japan Display Co., Ltd., Japan). The TFT module includes a control printed circuit board (PCB) and an image sensor 20 (with 80×84 pixels, pixel size=375μm) that provide illumination and image capture control signals. For the illumination module, the tricolor LED (EDGELEC) is controlled by a microcontroller 26 (Arduino Micro, Arduino LLC) through a constant current LED driver (TLC5916, Texas Instrument, Texas, USA) to sequentially provide red (620nm), green (520nm) and blue (420nm) illumination beams. The microcontroller 26, LED driver and tricolor LED are all integrated on a single PCB, which is powered by a 5V-1A voltage adapter and communicates with the TFT PCB through an LED power signal.

[0049] The illumination light passes through the transparent solid agar and forms a lens-free image of the growing bacterial colony on the TFT image sensor 20. The distance between the LED and the sample (i.e. Figure 2C The z1 distance shown in Figure 1 is about 15.5 cm, which is enough to evenly cover the entire sample surface with illumination light. The distance (z2) between the sample 110 and the TFT sensor 20 is roughly equal to the thickness of solid agar, which is about 5 mm. The mechanical support materials for the PCB, sample, and sensor were custom-made using a 3D printer (Objet30 Pro, Stratasys, Minnesota, USA).

[0050] Image data acquisition

[0051] Time-lapse imaging experiments were performed to collect data for the training and testing phases. The CFU imaging mode captured time-lapse images of the tested agar plates every 5 minutes under red, green, and blue illumination for 70 h. A control program 28 with a graphical user interface (GUI) 94 was developed to automatically perform illumination switching and image capture. The original TFT holographic images 70 h were saved in 12-bit format. After the experiment was completed, the samples were disposed of as solid biohazardous waste. A total of 889 time-lapse TFT holographic images of Escherichia coli colonies from 17 independent experiments were collected for 70 h to first train the CFU detection neural network model. In addition to this, 442 bacterial colonies (128 Escherichia coli, 126 Citrobacter rodentium, and 188 Klebsiella pneumoniae) were propagated from 17 new agar plates and used to train (1) the final CFU detection neural network 90 (through transfer learning from the initial detection model) and (2) the CFU classification neural network 92. A third independent dataset of 265 colonies from 13 new experiments was used to blindly test the trained neural network model.

[0052] Bacterial colony candidate selection

[0053] The entire candidate selection workflow consists of image preprocessing, difference analysis, colony mask segmentation, and candidate position localization, following Figure 6 For each time point, at about 10 cm 2 Three raw TFT images 70h (red, green and blue channels) are obtained on the FOV. N_raw,C After that, where N refers to the N The Nth image obtained at , C represents the color channel, R (red), G (green) and B (blue), performs a series of preprocessing operations to enhance the image contrast. First, as Figure 6 As shown in operation ab, the image is interpolated and normalized 5 times by directly subtracting the first frame at T0. After this normalization step, the background area has approximately 0 signal, while the area representing the growing colonies has negative values ​​because the colonies partially block and scatter the illumination light. T is then normalized by adding 127 and saving the image as an unsigned 8-bit integer array. N The current frame at is scaled to 0-127, denoted as I N_norm,C .exist Figure 6 After operation bc in , as shown in equation (1), N_norm,C Averaging is performed to perform smoothing in the time domain, which yields I N_denoised,C :

[0054]

[0055] To further improve the sensitivity of the system, the difference image I averaged over the three color channels is N_diff The calculation is as follows:

[0056]

[0057] By this operation, the signal of static artifacts is suppressed, and the spatiotemporal signal of growing colonies is enhanced as a ring pattern. Next, a pixel-wise minimum intensity projection is performed, as Figure 6 The operation ef is shown in order to change the minimum intensity of the difference image from I (N-7)_diff Projection to I N_diff , generating image I N_projection After this step, using the empirically set intensity threshold, I N_projection is segmented into a binary mask. After morphological operations to fill the ring pattern and watershed-based segmentation of the clustered regions, M is obtained N ,like Figure 6 The operation is shown in g. Based on this binary mask M N , the connected components are extracted and their centroids are located, as Figure 6 These centroid coordinates are dynamically updated at each time point to ensure that the center of the growing colony remains positioned.

[0058] Despite this preprocessing of the acquired TFT image 70h, there are still some time-varying non-colony objects that can be selected as false colony candidates (e.g., bubbles, dust, or other features generated by uncontrolled motion of the agar surface). Therefore, the deep neural network 90 is trained to further screen each colony candidate to eliminate false positives, the details of which will be discussed in the next subsection.

[0059] Bacterial colony growth detection based on DNN

[0060] like Figure 6 As shown in operation i, 8 frames are cropped. N_denoised,C The time-lapse video 204 of each colony candidate region in the image is obtained. These videos 204 are then upsampled in the spatial domain and organized into a four-dimensional array (3×8×160×160, i.e., color channel×number of frames×x×y) to be fed to the CFU detection neural network 90, which adopts the architecture of a dense network, but the 2D convolutional layer is replaced by a pseudo 3D convolutional layer (see Figure 7). The weights of the CFU detection DNN 90 were initialized with a pre-trained model obtained on an E. coli CFU dataset with a single illumination wavelength of 515 nm. The pre-trained model was obtained using a total of 889 colonies (positive) and 159 non-colony objects (negative) from 17 independent agar plates. Then, the initial neural network model was transferred to a multi-species image dataset with multi-wavelength illumination using 442 new colonies and 135 non-colony objects from another 17 independent agar plates. Both the positive image dataset and the negative image dataset were augmented in the time domain with different start and end time points, resulting in more than 10,000 videos for training. A 5-fold cross-validation strategy was used to select the best hyperparameter combination. Once the hyperparameters were determined, all collected data were used for training to finally determine the CFU detection neural network 90. ​​When loading the training dataset, data augmentation, such as flipping and rotation, was also applied.

[0061] The network model was optimized using the Adam optimizer with a momentum coefficient of (0.9, 0.999) for 90 seconds. The learning rate was increased from 1×10 -4 Start with , and use a scheduler to reduce the learning rate by a factor of 0.8 every 10 polls. The batch size is set to 8. The loss function is chosen as:

[0062]

[0063] Where p is the network output, i.e., the probability of each class before the SoftMax layer, g is the ground truth label (equal to 0 or 1 for binary classification), K is the total number of training samples in a batch, and w is the weight assigned to each class, defined as w=1-d, where d is the percentage of samples in a class. The training process was performed using a GPU (GTX1080Ti), which took about 5 hours to converge. With a decision threshold of 0.5, the CFU detection neural network 90 converged with a sensitivity of 92.6% and a specificity of 95.8%. In the testing phase, the decision threshold was set to 0.99, which achieved 100% specificity.

[0064] Classification of Escherichia coli and other total coliforms based on DNN

[0065] In order to classify the species of the detected bacterial colonies, a second DNN-based classifier 92 was established. The CFU classification neural network 92 was trained on the same multi-wavelength dataset with 442 colonies (128 Escherichia coli colonies, 126 Citrobacter rodentium colonies, and 188 Klebsiella pneumoniae colonies). The input of the classification DNN 92 is organized as a four-dimensional array (3×8×160×160, i.e., color channel×number of frames×x×y), but a different normalization method is used. Different from the background subtraction normalization used for the CFU detection neural network 92, for the classification DNN 92, the network input is renormalized by dividing the background intensity obtained at the first time point T0. This division-based normalization is performed on three color channels, so the background will be normalized to approximately 1 in the three channels, showing white in the background. Through this operation, color variations in different experiments are minimized, improving the generalization ability of the classification DNN 92.

[0066] The network architecture of the classification DNN 92 is the same as that of the CFU detection network 90, but with some differences in the selection of hyperparameters (see Figure 7 ). The classification neural network model was randomly initialized and optimized using the Adam optimizer with a momentum coefficient of (0.9, 0.999). The learning rate was set from 1×10 -3 The training process was performed using a GPU (GTX1080Ti) and took about 5 hours to converge. During the training process, a decision threshold of 0.5 was used to classify E. coli colonies and other total coliform colonies, achieving 91% and 97% accuracy, respectively. In the testing phase, the decision threshold was set to 0.8, which achieved 100% classification accuracy. In addition, in the testing phase, a 4.5 mm 2 A colony size threshold was set to ensure that only colonies large enough to identify their species passed the classification network92.

[0067] Although embodiments of the present invention have been shown and described, various modifications may be made without departing from the scope of the present invention. For example, multiple TFT-based image sensors may be used to detect and classify over a larger area or on different growth plates. Therefore, the present invention should not be limited except as set forth in the appended claims and their equivalents.

Claims

1. A system for detecting and classifying live microorganisms and / or colonies of live microorganisms in a sample using time-lapse imaging, the system comprising: light source; a thin film transistor (TFT) based image sensor, the TFT based image sensor being positioned along a light path originating from the light source; a growth plate comprising a growth medium on said growth plate and comprising said sample inserted along said optical path and disposed adjacent to said TFT-based image sensor; a microcontroller or other circuit configured to periodically illuminate the growth plate with light from the light source and to capture time-lapse images of microorganisms and / or colonies of microorganisms on the growth plate with the TFT-based image sensor; as well as A computing device is configured to execute image processing software to process and analyze the time-lapse images of the microorganisms and / or colonies of the microorganisms on the growth plate, and detect candidate microorganisms and / or colonies of the candidate microorganisms in the time-lapse images.

2. The system of claim 1, further comprising an incubator integrated with the light source, the TFT-based image sensor, and the growth plate.

3. The system according to claim 1, wherein: The light source includes one or more selectively activated spectral bands.

4. The system according to claim 1, wherein: The image processing software is configured to receive captured time-lapse images of the microorganisms and / or colonies of the microorganisms on the growth plate, and the image processing software is configured to: (1) detect candidate microorganisms and / or colonies of the candidate microorganisms in the time-lapse images using a first trained deep neural network, wherein the first trained deep neural network is trained to detect real microorganisms and / or colonies of the real microorganisms from non-microorganism objects, and (2) output species categories associated with the detected real microorganisms and / or colonies of the detected real microorganisms using a second trained deep neural network, wherein the second trained deep neural network receives at least one time-lapse image of the real microorganisms and / or colonies of the real microorganisms or at least one digitally processed time-lapse image as input.

5. The system according to any one of claims 1 to 4, wherein: The microorganisms include prokaryotic cells, eukaryotic cells, bacteria, fungi, viruses, multicellular organisms or clusters, membranes or colonies thereof.

6. The system according to claim 1, wherein: The computing devices include local and / or remote computing devices.

7. The system according to claim 1, wherein: A lens or a group of lenses is used to magnify or reduce the hologram of the microorganism and / or the colony of microorganisms onto the TFT-based image sensor.

8. The system according to any one of claims 1 to 7, wherein: The TFT-based image sensor captures at least 10cm 2 field of view.

9. The system according to claim 1, wherein: The TFT based sensor is integrated on or within the growth plate.

10. The system according to claim 1, wherein: The TFT based sensor is disposable.

11. The system according to claim 1, wherein: The growth medium includes chromogenic agar plates.

12. A method of using the system according to claim 1, the method comprising: placing the growth plate containing the sample within the optical path; periodically illuminating the growth plate with the light source, wherein the periodic illuminating comprises sequentially illuminating the growth plate with one or more illumination spectral bands; and A plurality of time-lapse images of microorganisms and / or colonies of microorganisms on the growth plate are obtained.

13. The method according to claim 12 further includes processing the time-lapse images of the microorganisms and / or the colonies of the microorganisms on the growth plate with image processing software, the image processing software is also configured to detect candidate microorganisms and / or colonies of the candidate microorganisms in the time-lapse images based on differential image analysis in the time-lapse holographic images, and also includes a first trained deep neural network and a second trained deep neural network, the first trained deep neural network is trained to detect real microorganisms and / or colonies of the real microorganisms from non-microorganism objects, the second trained deep neural network receives at least one time-lapse image of the real microorganisms and / or colonies of the real microorganisms or at least one digitally processed time-lapse image as input, and outputs a species category associated with the detected real microorganisms and / or the detected colonies of real microorganisms.

14. The method according to claim 13, wherein: The microorganisms include prokaryotic cells, eukaryotic cells, bacteria, fungi, viruses, multicellular organisms or clusters, membranes or colonies thereof.

15. The method according to claim 12, wherein: The sample includes one or more of a water sample, a food sample, a biological or other fluid sample.

16. A method for detecting and classifying live microorganisms and / or colonies of said live microorganisms using time-lapse imaging, the method comprising: providing a growth plate comprising a growth medium thereon and containing a sample; periodically illuminating the growth plate with at least one spectral band of illumination light from a light source; capturing time-lapse images of microorganisms and / or colonies of microorganisms on the growth plate using a TFT-based image sensor; as well as The candidate microorganisms and / or colonies of the candidate microorganisms in the time-lapse images are detected using image processing software, the image processing software comprising a first trained deep neural network and a second trained deep neural network, wherein the first trained deep neural network is trained to detect real microorganisms and / or colonies of the real microorganisms from non-microorganism objects, and the second trained deep neural network receives at least one time-lapse image or a digitally processed time-lapse image as input and outputs a species classification associated with the detected real microorganisms and / or colonies of the detected real microorganisms.

17. The method according to claim 16, wherein: The microorganisms include prokaryotic cells, eukaryotic cells, bacteria, fungi, viruses, multicellular organisms or clusters, membranes or colonies thereof.

18. The method according to claim 16, wherein: The time-lapse images are acquired multiple times per hour over multiple hours.

19. The method according to claim 16, wherein: The TFT-based image sensor uses a lens or a group of lenses to capture a magnified or reduced hologram of a microbial object and / or a microbial colony of the microbial object.

Images