Artificial intelligence techniques for generating predicted future images of microbial growth

Abstract

An exemplary system includes an image capture device configured to capture a sequence of images representing a sample of microbial growth, and a processing unit having one or more processors, the one or more processors configured to pass image data for the sequence of images to a machine learning model, the machine learning model trained to generate one or more predicted future images of the microbial colony at a future time, the machine learning model trained using past image data, the past image data including a plurality of past image data sets, each past image data set among the plurality of past image data sets including image data for a sequence of past images of a corresponding past microbial colony sample, and a predicted time interval between the future time and the capture time of the last image in the sequence of images is greater than each of the sampling time intervals.

Description

[Technical field]

[0001] background Many real-world processes, especially those of a chemical and biological nature, evolve over time. For example, microbial growth may not be detectable initially, but may become detectable after an incubation period that provides sufficient time for the microorganisms to replicate and form colonies that are visible, for example, to the human eye or to a camera or other recording device. To test for the presence or number of microorganisms on or in a material, a sample of the material can be taken and placed under predefined conditions, i.e., usually conditions of elevated temperature in the presence of nutrients, for a period of time sufficient to allow any microorganisms that may be present (e.g., bacteria, yeast, fungi, molds, etc.) to replicate and form detectable and / or countable colonies. For example, to test for the presence of microorganisms in food or on food processing equipment, a sample can be taken from the surface of the food or food processing equipment and placed on a culture plate (e.g., a Petri dish or other substrate such as a specific film that can be used to culture cells). After an incubation period, which may be as long as 24 hours, 72 hours, or even longer in some cases, it is possible to count the number of colony forming units (CFU) of the microorganisms on the sample plate, which provides information regarding the presence and / or number of microorganisms in the sample.

[0002] overview Generally, the present disclosure describes techniques for generating predicted future images showing the growth of a current sample of microorganisms (bacteria, yeast, mold, etc.). More specifically, the present disclosure describes exemplary techniques for generating predicted future images based on applying machine learning models to a time series of images of a sample plate. The predicted future images can be provided hours or days in advance of actual images of microbial growth, allowing assessors to make early determinations of the presence and abundance of microorganisms based on reasonably accurate predictions of probable future images of expected microbial growth.

[0003] As described herein, the image capture device or a component of the predictive device can capture a time series of images of microbial growth at a relatively early state of microbial growth, for example, during the first 8, 16, or 24 hours of growth. The predictive device can generate a predicted future image of the microbial growth as it will likely appear at a future time point, for example, 24, 48, or 72 hours into the future. A processing unit of the predictive device receives the image data and provides the image data to a machine learning model trained to generate a predicted future image of the microbial growth. For example, microbial growth may typically require a 24-48 hour growth period before the microbial colonies become large enough that they can be relied upon for purposes of making a decision. In various examples described herein, the predictive device can receive a time series of images from an early period of growth, for example, from the first 8 hours, and process the time series of images to generate a predicted future image of the microbial growth as it will likely appear after 24, 48, or 72 hours of growth.

[0004] The disclosed technology can provide at least one technical advantage over existing methods. For example, a practical application of the disclosed technology is a predictive device that can generate predicted future images of microbial growth that can be used to make a decision regarding the presence and amount of microbial counts much earlier than it is possible to visualize and count the microbial growth. Using the above example, a predictive device using the disclosed technology can correspondingly reduce the waiting time for making a decision based on a sample of microbial growth from 24 to 48 hours to 8 or 24 hours. The ability to make an early decision can provide an end user with a result based on the presence and amount of microbial growth counted in a shorter period of time than techniques that require relatively long incubation times.

[0005] In one example, the disclosure provides a system including an image capture device configured to capture image data for a sequence of images representing growth of a microbial colony at a plurality of times, each image prior to a final image in the sequence of images being separated by a sampling time interval between the image and a next image, the system including a processing unit having one or more processors, the one or more processors passing the image data for the sequence of images to a machine learning model, the machine learning model being trained to generate image data representing one or more predicted future images of the growth of the microbial colony, and the one or more predicted future images being processed by a machine learning model. The system is configured to execute instructions that cause a processing unit to: each of the images represents a microbial colony at a corresponding future time, the machine learning model has been trained using past image data, the past image data including one or more past image data sets, each past image data set of the one or more past image data sets including image data for a sequence of past images of a corresponding past microbial colony sample, a predicted time interval between the future time and a capture time of a last image in the sequence of images being greater than each of the sampling time intervals; and output image data representing the one or more predicted future images of the microbial colony.

[0006] In another example, the disclosure describes a method including: receiving, by a processing unit including one or more processors, image data for a sequence of images captured at multiple times representing growth of a sample of a microbial colony, where each image in the sequence of images prior to a final image is separated by a sampling time interval between the image and a next image; passing the image data for the sequence of images to a machine learning model, where the machine learning model has been trained to generate image data representing one or more predicted future images of the microbial colony, where each of the one or more predicted future images represents the microbial colony at a corresponding future time, where the machine learning model has been trained using past image data, where the past image data includes a plurality of past image data sets, where each past image data set in the plurality of past image data sets includes image data for a sequence of past images of a corresponding past microbial colony sample, where a prediction time interval between the future time and a capture time of a final image in the sequence of images is greater than each of the sampling time intervals; and outputting the image data representing the predicted future images of the microbial colony.

[0007] In a further example, the disclosure describes a method including: receiving past image data, the past image data including a plurality of past image data sets, each past image data set of the plurality of past image data sets including image data for a sequence of past images of a corresponding microbial colony sample, each image in the sequence of past images prior to a final image in the sequence of past images being separated by a sampling time interval between the image and a next image; training a machine learning model to generate, for each past image data set of the plurality of past image data sets, one or more predicted future images of the microbial colony, each image corresponding to a future time from the sequence of past images, each prediction time interval between the future time and a capture time of a final image in the sequence of past images being greater than each of the sampling time intervals; and adjusting weights in a layer of the machine learning model based on a difference between the one or more predicted future images and one or more target images associated with the microbial colony sample.

[0008] In another example, the disclosure describes a system that includes: means for receiving image data for a sequence of images captured at a plurality of times representing growth of a sample of a microbial colony, where each image in the sequence of images prior to a final image is separated by a sampling time interval between the image and a next image; means for passing the image data for the sequence of images to a machine learning model, where the machine learning model has been trained to generate image data representing one or more predicted future images of the microbial colony, where each of the one or more predicted future images represents the microbial colony at a corresponding future time, where the machine learning model has been trained using past image data, where the past image data includes a plurality of past image data sets, where each past image data set in the plurality of past image data sets includes image data for a sequence of past images of a corresponding past microbial colony sample, where a prediction time interval between the future time and a capture time of a final image in the sequence of images is greater than each of the sampling time intervals; and means for outputting the image data representing the predicted future images of the microbial colony.

[0009] In a further example, the disclosure describes a method including means for receiving past image data, the past image data including a plurality of past image data sets, each past image data set of the plurality of past image data sets including image data for a sequence of past images of a corresponding microbial colony sample, each image in the sequence of past images prior to a final image in the sequence of past images being separated by a sampling time interval between the image and a next image; means for training a machine learning model to generate one or more predicted future images of the microbial colony for each past image data set of the plurality of past image data sets, each image corresponding to a future time from the sequence of past images, each prediction time interval between the future time and a capture time of a final image in the sequence of past images being greater than each of the sampling time intervals; and means for adjusting weights in a layer of the machine learning model based on a difference between the one or more predicted future images and one or more target images associated with the microbial colony sample.

[0010] The details of at least one example of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 is a block diagram illustrating a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. [Figure 1B] FIG. 1 is a block diagram illustrating a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. [Figure 2A] FIG. 1 is a block diagram illustrating a training system in accordance with at least one example technique described in this disclosure. [Figure 2B] FIG. 1 is a block diagram illustrating a training system in accordance with at least one example technique described in this disclosure. [Diagram 3] FIG. 2 is a block diagram illustrating another training system in accordance with at least one example technique described in this disclosure. [Figure 4A] FIG. 4 is a block diagram illustrating an example training framework for the training system shown in FIGS. 2 and 3 in accordance with at least one example technique described in this disclosure. [Figure 4B] FIG. 4 is a block diagram illustrating an example training framework for the training system shown in FIGS. 2 and 3 in accordance with at least one example technique described in this disclosure. [Figure 4C] FIG. 4 is a block diagram illustrating an example training framework for the training system shown in FIGS. 2 and 3 in accordance with at least one example technique described in this disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating a system for generating predicted future images of microbial growth using tiled image data, according to at least one exemplary technique described in this disclosure. [Figure 6] FIG. 2 is a block diagram of an exemplary processing unit of a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating an example operation of a forecasting system in accordance with one or more techniques of the present disclosure. [Figure 8] FIG. 1 is a flow diagram illustrating an example operation of a training system in accordance with one or more techniques of the present disclosure. [Figure 9] FIG. 1 illustrates an exemplary sequence of images showing the growth of a microbial colony over a period of time. [Figure 10] FIG. 2 illustrates an exemplary input image sequence and predicted future images generated in accordance with at least one exemplary technique described in this disclosure.

[0012] Detailed Description Systems and techniques are described for generating predicted future images of microbial growth in samples taken, for example, from food products or food processing environments. An image capture device or a component of the prediction system can capture a time series of samples over an initial sample period and generate predicted future images of the samples as they will appear at a future time point. The difference between the future time point and the last sample in the time series can be referred to as the prediction interval. The sample period can be relatively much shorter than the full growth period of microbial colonies that may be present in the sample, i.e., the sampling period can be much shorter than the prediction period.

[0013] 1A is a block diagram illustrating a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. In some embodiments, the system 100 includes a prediction system 102, an image capture device 110, and a user interface 111.

[0014] In some embodiments, the microbial colony sample 109 may originate from a food product, e.g., a sample obtained from a food product from a food manufacturer. A food manufacturer or intermediary may wish to determine the microbial quality of a food product by taking a sample of the food product and determining if microorganisms are present in the sample. The sample may be placed on a plate (e.g., a thin film culture plate) having a growth medium, and the image capture device 110 may capture a sequence of images of the plate. If microbial colonies are present in the sample, they typically grow and become more visible as the sequence of images progresses. FIG. 9 shows an example image sequence of microbial colonies with an initial image followed by image captures at about 14 hours, about 35 hours, about 50 hours, about 66 hours, and about 70 hours during the sampling period. In some exemplary implementations, the microbial colony sample 109 may be disposed on a thin film culture plate, such as that available under the name PETRIFILM™ from 3M Company, St. Paul, Minn., USA, and the image capture device 110 may be a plate reader, such as the PETRIFILM™ Plate Reader Advanced from 3M Company, St. Paul, Minn., USA, however, the techniques disclosed herein are equally applicable to image data acquired from other thin film culture plate imaging systems.

[0015] The image capture device 110 acquires an image of the microbial colony sample 109. The image capture device 110 may be a camera or other component configured to capture image data representative of the microbial colony sample 109. The image capture device 110 may include any component capable of capturing image data, such as a video recorder, an infrared camera, a charge coupled device (CCD) array, or a laser scanner. Although one image capture device 110 is shown in FIG. 1A, multiple image capture devices 110 may be present.

[0016] In some embodiments, the image capture device 110 captures a sequence (e.g., a time series) of images of the microbial colony sample 109. The sequence of images is referred to herein as a microbial colony image sequence 112. Each of the images in the image sequence 112 is captured (or sampled) at a different time. Image data for the images in the microbial colony image sequence 112 may be captured at different and possibly unequal time intervals between image captures. Using the image sequence shown in FIG. 9 as an example, the time intervals between image captures in the sequence are about 14 hours, about 21 hours, about 15 hours, about 16 hours, and about 4 hours.

[0017] In some embodiments, the images in the sequence of images may be represented as two-dimensional images. In some embodiments, the images in the sequence of images may be a three-dimensional (3D) volume of images. For example, the images may be represented as a 3D volume of image data recorded over a relatively short period of time. By way of example, the 3D volume may be a video recording. The three dimensions of the volume may be the x-dimension, the y-dimension, and the time dimension. Thus, capturing image data may refer to capturing a 2D image or may refer to recording multiple frames of image data as a 3D volume over a period of time. In some embodiments, the time period may correspond to the rate of colony growth. For example, the time period for the 3D volume for a first microbial colony may be 30 seconds, while the time period for the 3D volume for a second microbial colony that grows more slowly may be 5 minutes.

[0018] After the image capture device 110 captures an image of the microbial colony sample 109, the image capture device 110 may store the image data for the captured image in the storage unit 105 as part of a microbial colony image sequence 112. The image capture device 110 may associate a timestamp indicating when the image capture device 110 captured the image. The timestamp may be stored with the image or may be stored as metadata 114. The metadata 114 may also include data such as the environmental conditions (e.g., temperature) when the sample was taken, the environmental conditions of the sample plate, the growth medium of the sample plate, etc.

[0019] The processing unit 104 of the prediction system 102 can read the microbial colony image sequence 112. For example, the prediction system 102 can read the microbial colony image sequence 112 in response to receiving a command from a user via the user interface 111. The processing unit 104 can utilize an artificial intelligence (AI) engine 106 and a machine learning model 108 to process the image data of the microbial colony image sequence 112 and, optionally, the metadata 114 to generate predicted microbial colony image data 116. In some embodiments, the AI ​​engine 106 and the machine learning model 108 can implement a neural network. For example, the machine learning model 108 can define layers of a neural network that has been trained using techniques described herein to receive the microbial colony image sequence 112 as an input and to generate predicted microbial colony image data 116 as an output. In some embodiments, the predicted microbial colony image data 116 is in the same form as the image data for the images in the microbial colony image sequence 112. For example, if an image in the microbial colony image sequence 112 is a 2D image, the predicted microbial colony image data 116 represents a 2D image. Similarly, if an image in the microbial colony image sequence 112 is a 3D volume, the predicted microbial colony image data 116 represents a 3D volume. In some embodiments, the predicted microbial colony image data 116 can have a different form than the image data for an image in the microbial colony image sequence 112. For example, the image in the microbial colony image sequence 112 can be a 3D volume. The prediction system 102 can generate the predicted microbial colony image data 116 as a 2D image.

[0020] In some aspects, the prediction system can extract image processing features (e.g., differences from an initial image over time, gradient-based images, etc.) and use such features as additional inputs to the prediction system 102 and machine learning model 108.

[0021] 1A, the predicted microbial colony image data 116 may be a sequence of images (e.g., 2D images or 3D volumes). In some embodiments, the sequence of images may be a time sequence having a temporal order of predicted images. For example, the first image in the predicted microbial colony image data 116 may be the earliest predicted image and the last image may be the predicted image at the latest time in the sequence.

[0022] 1A and the discussion above, it is assumed that the microbial colony image sequence 112 is a sequence of images. In some embodiments, the microbial colony image sequence 112 may be a single image (e.g., a single 2D image or a single 3D volume), and the machine learning model 108 may be trained to generate predicted microbial colony image data 116 from a single input image.

[0023] In some embodiments, the prediction system 102 can provide the predicted microbial colony image data 116 to the decision support system 103. The decision support system 103 can analyze the predicted microbial colony image data 116 to determine whether the food product from which the microbial colony sample 109 was sourced is of sufficient quality to be shipped to a customer.

[0024] In some examples, the user interface 111 allows a user to control the system 100. The user interface 111 may include any combination of a display screen, a touch screen, buttons, speaker input, or speaker output. In some examples, the user interface 111 is configured to power on or off any combination of elements of the system 100, provide configuration information for the prediction system 102, the processing unit 104, and / or the decision support system 103, and display output from the prediction system 102.

[0025] FIG. 1B is a block diagram illustrating input image data for a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. In the example shown in FIG. 1B, image capture device 110 captures image data for images 112A-112C at different times over an input time interval 120. In this example, the time interval between image capture of image 112A and image capture of image 112B is m hours. The time interval between image capture of image 112B and image capture of image 112C is m±k hours. The predicted time interval 122 between image 112C and predicted microbial colony image data 116 is m+h hours. The predicted time interval 122 does not represent an amount of time that actually passes. Instead, the predicted time interval m+h represents a simulated time interval between image 112C and predicted microbial colony image data 116. The actual time interval between the image 112C and the generation of the predicted microbial colony image data 116 may simply be the amount of time it takes the prediction system 102 to generate the predicted microbial colony image data 116. The prediction time interval 122 may be much longer than the input time interval 120. For example, the prediction time interval 122 may be more than twice the input time interval 120, and in some instances even longer.

[0026] As an example, the image capture device 110 can capture image data of the microbial colony sample 109 every 15-30 minutes over a six hour period to generate a microbial colony image sequence 112. The prediction system 102 can process the microbial colony image sequence 112 to generate predicted microbial colony image data 116 representing predicted future images of the microbial colony sample at a future time point, e.g., 24 hours into the future. The predicted future images can be used to determine whether the food product from which the sample was sourced is safe for shipment to a customer. Using the techniques described herein, a user (or user system) can use the predicted future images to reach a conclusion regarding food safety much sooner than would be possible using current existing methods. In the example described above, the user can reach a conclusion regarding food safety 24 hours earlier than current methods.

[0027] The example shown in FIG. 1B illustrates several aspects of the microbial colony image sequence 112 and the predicted microbial colony image data 116. The first aspect is that there may be long sampling intervals. The respective input images may be separated from each other by hours and days. Within these intervals, there may be many variations governed by potentially nonlinear processes. Therefore, it is difficult to generate accurate future images or 3D volumes.

[0028] A second aspect is that the time intervals between image captures may be inconsistent and unequal. As shown in FIG. 1B, the first two samples may be m times apart, while the time between the next two samples may be greater than m times or less than m times (i.e., m±k times). Also, the expected intervals may be relatively large due to missing or damaged samples. This may cause k to be even larger than if no samples were missing.

[0029] A third aspect is that the predicted time interval (e.g., m+h) associated with the predicted microbial colony image data 116 can be very long compared to the interval between samples. For example, the predicted time interval 122 between the last sampled image (e.g., image 112C) of the microbial colony image sequence 112 and the predicted microbial colony image data 116 can be several hours to several days.

[0030] FIG. 2A is a block diagram illustrating training data for a training system, such as the training system discussed below with reference to FIG. 2B. In some embodiments, the training data includes a plurality of past microbial colony image sequences. A past microbial colony image sequence 212 includes images 232A-232N, which include image data for a corresponding sequence of past microbial colony images captured at different times during a past microbial colony growth period 208. In some embodiments, as described further below, the training data can include images captured during a sampling period 210 (e.g., images 232A-232M). The final image of the sequence, image 232N, can be the target image of the sequence. That is, the final image 232N can be used as a ground truth for the past microbial colony growth. In some aspects, as described further below, the training data may include images captured during sampling period 210 (e.g., images 232A-232M) and images captured after sampling period 210 (e.g., images 232M+1-232N-1).

[0031] FIG. 2B is a block diagram illustrating a training system according to at least one exemplary technique described in this disclosure. The training system 202 can include a machine learning framework 204 including a machine learning engine 206. The machine learning framework 204 can receive training data 203 and process the training data to generate a machine learning model 224. In some aspects, the machine learning framework 204 includes the machine learning engine 206, which can use supervised or unsupervised machine learning techniques to train the machine learning model 224. In some aspects, the machine learning engine 206 can be a deep learning engine implementing a convolutional neural network (CNN). In some aspects, the machine learning engine 206 can be, for example, a generative adversarial network (GAN). As an example, the machine learning engine can be a T-adversarial GAN. In some aspects, the machine learning engine 206 can be a U-Net-based machine learning engine, including the U-Net2D architecture and the U-Net3D architecture. The U-Net architecture can be used to preserve content such as spatial information in the training data. The U-Net architecture typically has contraction and expansion paths, which, together with skip connections within layers, may be used to link corresponding feature maps on the encoder and the decoder. Linking feature maps can promote feature reuse in the encoder, thereby reducing information loss. Additionally, the U-Net architecture may be computationally efficient and may be trained using a relatively small data set.

[0032] In some aspects, the machine learning framework 204 can implement multiple machine learning techniques and can apply these multiple machine learning techniques together in training the machine learning model 224. For example, the machine learning engine 206 can be a U-Net engine, and the machine learning framework can apply a cyclic learning technique using the machine learning engine 206. Further details regarding the machine learning framework and cyclic learning are provided below in connection with FIGS. 4A-4C.

[0033] The training data 203 may include past microbial colony image sequences 212A-212N (collectively past microbial colony image sequences 212). Each past microbial colony image sequence 212 in the training data is a sequence of images of a particular microbial colony growth captured or recorded over a period of time prior to training the machine learning model 224. For example, past microbial colony image sequence 212A may be image data for a sequence of images showing growth of a first microbial colony over time, past microbial colony image sequence 212B may be image data for a sequence of images showing growth of a second microbial colony over time, past microbial colony image sequence 212C may be image data for a sequence of images showing growth of a third microbial colony over time, and so on.

[0034] Each historical microbial colony image sequence 212A-212N in the training data 203 can have a corresponding target image 220A-220N. The target image for an image sequence is a final image that is the "ground truth," e.g., an actual image of a microbial colony associated with the image sequence captured at the end of a growth period.

[0035] The training data 203 may also include metadata 214 that may be used to train the machine learning model 224. The metadata 214 may include timestamps indicating when the images in the sequence of historical microbial colony images 212 were captured, environmental conditions associated with the samples and sample plates, growth media, etc. In some embodiments, the metadata 214 may be added to the training system 202 (or the prediction system 102 of FIG. 1) in at least two parts by padding the metadata information to the boundaries of the images. In some embodiments, the metadata may be added as a vector to the latent feature vectors generated in an intermediate layer of the machine learning model 224.

[0036] The training system 202 provides the training data 203 to the machine learning framework 204 for processing by the machine learning engine 206. The machine learning engine 206 processes the past microbial colony image sequence 212 to generate predicted image data 218. The predicted image data 218 can include a sequence of predicted future images, each of which has an associated future time. The machine learning framework 204 can compare the predicted future images to a target image 220 associated with the past microbial colony image sequence 212 to determine a difference between the predicted future image and the target image 220. The difference between the predicted future image and the target image 220 is used to update the training weights in the machine learning model 224 to attempt to improve the model's ability to generate accurate predicted future images. In some embodiments, the weights in the machine learning model 224 can be adjusted using a loss function such as a reconstruction loss or a GAN loss.

[0037] In some embodiments, the training system 202 weights images in the sequence 212 of past microbial colony images to influence their effect on training the machine learning model 224. For example, images in the sequence 212 of past microbial colony images captured later in the input time interval may be weighted more heavily than images captured earlier in the sequence 212. This reflects the fact that images captured later in the input time interval may be closer to the target image 220 associated with the sequence and therefore have greater training value.

[0038] After the training system 202 trains the machine learning model 224, the model can be deployed to a prediction system 216, which may be an implementation of the prediction system 102 of Figure 1. The prediction system 216 can receive the microbial colony image sequence 112 and process the image sequence using an AI engine 222 and the deployed machine learning model 224 to generate predicted microbial colony image data 116.

[0039] As shown in FIG. 2B, the machine learning framework 204 can generate predicted image data 218, which can include a sequence of images (e.g., a 2D image or a 3D volume). In some embodiments, the machine learning framework 204 can generate predicted image data 218, which can be a single 2D image or a 3D volume. Additionally, the past microbial colony image sequence 212 can be a sequence of images, as shown in FIG. 2B. In some embodiments, the microbial colony image sequence 212 can be a single image (e.g., a single 2D image or a single 3D volume), and the machine learning framework 204 can train the machine learning model 224 to generate predicted image data 218 from a single input image.

[0040] In some aspects, the machine learning engine 206 can implement a weighted loss function that assigns different weights to images in the predicted image data 218. For example, the weighted loss function can assign a greater weight to a later image in a sequence of images than an earlier image in the sequence. In other words, a first predicted future image having an associated predicted future time that is earlier than a predicted future time associated with a second predicted future image will have a smaller weighting than the second predicted future image. This may be desirable because an accurate and temporally later predicted future image in a sequence of predicted images may be more valuable to an end user than another predicted image predicted for an earlier future time in the sequence.

[0041] 3 is a block diagram illustrating further aspects of a training system according to at least one example technique described in this disclosure. In the example illustrated in FIG. 3, the training system 300 includes a loading and formatting unit 302, a data splitting unit 304, a spatial expansion unit 306, a temporal expansion unit 308, a sampling unit 310, a batch processing unit 312, a machine learning framework 314, a testing unit 320, and a result visualization unit 322. The loading and formatting unit 302, the data splitting unit 304, the spatial expansion unit 306, the temporal expansion unit 308, the sampling unit 310, the batch processing unit 312, the machine learning framework 314, the testing unit 320, and the result visualization unit 322 may be implemented as a configurable pipeline for processing the candidate image data set 301 into a batch of image data sets to be used by the machine learning framework 314 to train the machine learning model 319.

[0042] The loading and formatting unit 302 can process the candidate image data set 301 to format the image sequences in the candidate image data set 301 into a format that the training system can process. For example, the images can be scaled, resized, cropped, etc., to be in a format compatible with the machine learning framework 314.

[0043] The data splitting unit 304 can split the candidate image data set 301 into training data, testing data and / or validation data. For example, the percentage of the data set to be used as training data, testing data and / or validation data can be specified by input parameters.

[0044] The spatial expansion unit 306 can increase the amount of training data by transforming an existing image into one or more additional training images. For example, an image can be transformed by taking a section of the image and moving the section left or right along a diagonal axis, rotating the image, mirroring the image, etc. to create a new image, which can be included in the training data.

[0045] The time expansion unit 308 can control the selection of images from the candidate image data set 301 based on the temporal aspects of the candidate training data. The time expansion unit 308 can select an image sequence based on where the images are located on the time axis. As an example, the time expansion unit 308 can select images based on a start time and an end time.

[0046] The sampling unit 310 may select images from the training data according to a skip factor 311. For example, rather than including all images in the candidate image data set 301, the sampling unit 310 may select a subset of images in the candidate data set. The skip factor 311 may be used to control the manner in which images are selected. For example, a skip factor of four may cause the sampling unit 310 to skip four images in the candidate data set before selecting the next image for inclusion in the training data.

[0047] The configuration data 324 may include data that determines the data sources, hyperparameters, machine learning parameters, type of machine learning, etc. for use by the machine learning framework 314.

[0048] The batch processing unit 312 creates and controls batches of training data to be processed as a unit. For example, a first batch of training data may be used to train a first machine learning model 319, and a second batch of training data may be used to train a second machine learning model 319. The batch processing unit 312 can use the configuration data 324 to determine which data source to use for a batch of training data. The batch processing unit 312 can also use the configuration data 324 to specify configuration parameters that the machine learning framework 314 should use when training the machine learning model 319 with the corresponding batch of training data.

[0049] The batch processing unit 312 can provide batches of training data to the machine learning framework 314 for use in training the machine learning model 319. The machine learning framework 314 can include a machine learning engine 316. In some aspects, the machine learning framework 314 and / or the machine learning engine 316 can be an implementation of the machine learning framework 204 and / or the machine learning engine 206 of FIG. 2B. As discussed above, the machine learning engine 316 can be a deep learning engine implementing a CNN, GAN, U-Net-based machine learning engine, including the U-Net2D and U-Net3D architectures.

[0050] The testing unit 320 can test the machine learning model 319 to determine the accuracy of the predicted microbial colony images generated using the machine learning model 319. The testing unit 320 can receive test data as input and can use the machine learning model 319 to generate predicted microbial colony images as output. The testing unit 320 can compare the predicted microbial colony images with target microbial colony images that represent "ground truth." For example, as described above, the candidate data image set 301 can be divided into training data and testing data.

[0051] The machine learning framework 314 can train the machine learning model 319 using techniques described herein to generate predicted microbial colony images. Once trained, the testing unit 320 can apply the generated machine learning model 319 to a sequence of past microbial colony images in the test data to generate predicted microbial colony images. To determine the accuracy of the machine learning model 319, the predicted microbial colony images can be compared to a target microbial colony image for the test data. As an example, the test data can include a sequence of past images of microbial colony growth, where the last image in the sequence can be the target microbial colony image. The testing unit 320 can apply the machine learning model 319 to a first portion of the sequence of past images to generate predicted microbial colony images. The inspection unit 320 can compare the predicted microbial colony images to the target microbial colony images and determine the accuracy of the predicted microbial colony images based on the comparison. The testing unit 320 can determine different measures of the performance of the machine learning model 319 and compare these measures to other machine learning models that may have been generated using different training parameters and / or training data. The results of this comparison can be used to determine a machine learning model 319 that produces better (e.g., more accurate) predicted microbial colony images.

[0052] The result visualization unit 322 can provide feedback to the user regarding the training of the machine learning model 319. For example, the result visualization unit 322 can output statistics regarding the accuracy of the predicted microbial colony images generated by the machine learning model 319. In some embodiments, the result visualization unit 322 can output examples of the input microbial colony image sequence and the predicted microbial colony images generated by the machine learning model 319. The user can use the output of the result visualization unit 322 to determine whether any adjustments need to be made regarding the training of the machine learning model 319. For example, the user can adjust the hyperparameters, prediction time interval, or other configuration data 324 and send a signal to the batch processing unit 312 to start providing another batch of training data to train the new machine learning model 319. The result visualization unit 322 can provide an output that can be used to compare the performance of the machine learning model 319 with other machine learning models. Training system 300 need not include all of the components shown in FIG. 3, and in different implementations, training system 300 can include different combinations of one or more of the components shown in FIG.

[0053] 4A-4C are block diagrams illustrating an example interactive machine learning framework used in the training system illustrated in FIGS. 2 and 3, in accordance with at least one example technique described in this disclosure. For purposes of illustrating the example illustrated in FIGS. 4A-4C, it can be assumed that a microbial growth takes 24 hours to fully grow and develop. A sequence of input images 406 captured during the first 6 hours of growth can be used to train a machine learning model to generate predicted microbial colony images, potentially cutting out the 18 hour wait time between taking a sample and getting a result. In the example illustrated in FIGS. 4A-4C, the image sequence 406 is generated from k videos V1-V2. kwhere V1 is the first video in the sequence and V k This is the last video in the six-hour sequence. out is the predicted future image generated by the machine learning framework using image data selected from the first 6 hours of images,406, and V label is an image representing the "ground truth" image, also called the target image.

[0054] 4A is a block diagram illustrating a machine learning framework 402 that performs two passes through an image sequence to train a machine learning model to predict future microbial colony images. The machine learning framework 402 may be an implementation of the machine learning framework 204 of FIG. 2B and / or the machine learning framework 314 of FIG. 3. The machine learning framework 402 includes two deep learning architectures 404A and 404B. The deep learning architecture 404A is used to train the machine learning model 405 to generate predicted future images from a sequence of previous images, and the deep learning architecture 404B is used to train the machine learning model 405 to reconstruct previous images from future images and predicted future images. The deep learning architectures 404A and 404B may be CNNs (including U-Net2D and U-Net3D), GANs, T-Adversarial GANs, Time Cyclic GANs, or GANs using privileged information, respectively. In some aspects, deep learning architectures 404A and 404B share layers in machine learning model 405. The shared layers provide constraints for learning, where prior information is linked to future information such that a predicted future image can be used to reconstruct the prior image. In the example shown in FIG. 4A, the goal of the first pass is to use deep learning architecture 404A to learn a ground truth image V label A predicted future image V that is the same as or similar to out The purpose of this study is to generate V out V labelCompared with V out and V label The difference between the generated predicted future image V out The training weights in the deep learning architecture 404A are used to update the training weights in the deep learning architecture 404A in an attempt to improve . In some aspects, the weights in the machine learning model 405 can be adjusted using a loss function such as a reconstruction loss or a GAN loss. The process described above is generally the same as the process used for unidirectional training in some implementations.

[0055] In the second pass, the secondary goal is to use deep learning architecture 404B to learn the next images V2 to V3. k Predicted future image V out and are used as inputs to the deep learning architecture 404B to generate a reconstructed first image V1' in the sequence that is the same as or similar to the actual first image V1 in the sequence. V1' is compared to V1 and the difference is used to adjust the weights in the layers of the machine learning model 405. This second pass can make the layer weights more robust and can avoid the machine learning model overfitting to the training data.

[0056] 4B is a block diagram illustrating a machine learning framework 410 that implements two bidirectional passes to train a machine learning model to predict future microbial colony images. The machine learning framework 410 includes deep learning architectures 411A, 411B, and 411C (collectively, "deep learning architectures 411"). Each of the deep learning architectures 411 may be a CNN (including U-Net2D and U-Net3D), a GAN, a T-adversarial GAN, a time-cyclic GAN, or a GAN that uses privileged information.

[0057] In some aspects, the deep learning architecture 411A is implemented similarly to the deep learning architectures 404A and 404B described above with reference to FIG. 4A. That is, the deep learning architecture 411A may be bidirectional and perform two passes through the image sequence, i.e., predicting a future image V from an initial image in the image sequence 406. out The first pass generates the predicted future image V out and a second pass to generate a first reconstructed image V1 based on the first image in the image sequence and images subsequent to the first image in the image sequence. In some aspects, deep learning architecture 411A differs from deep learning architectures 404A and 404B of FIG. 4A in that it includes a longer range of image data in training the machine learning model. For example, deep learning architecture 411A may use input images 406 (e.g., V1-V2) collected during the first 6 hour period of microbial growth to generate a second pass to generate a reconstructed first image V1 based on the first image in the image sequence and images subsequent to the first image in the image sequence. k ), images 408 collected during the later portion of the 24 hour growth period may also be included. For example, images from 19 hours to 23 hours (e.g., images V k+n ,V k+n+1 ,V k+n+2 ) can be provided as training data to the deep learning architecture 411A.

[0058] In the example shown in FIG. 4B, the first stage of generating a machine learning model 415 is to use a machine learning architecture 411A to generate a ground truth image V label A predicted future image V that is the same as or similar to out training a machine learning model 415 to generate out V label Compared with V out and V label The difference between the generated predicted future image V outThe deep learning architecture 411A is used to update the training weights in the machine learning model 415 in an attempt to improve V. In some aspects, the weights in the machine learning model 415 can be adjusted using a loss function such as a reconstruction loss or a GAN loss. Additionally, the deep learning architecture 411A is out 4B, the input to the deep learning architecture 411A during the first stage of machine learning is the images captured in the first 6 hours of a 24-hour period (e.g., V1 through V2). k ) and images captured later in the 24-h period (e.g., V captured between 19 and 23 h). k+n ,V k+n+1 ,V k+n+2 Thus, in the first stage of training, the machine learning framework 410 utilizes longer frames of data to improve the accuracy of the machine learning model 415.

[0059] While it may be advantageous to use longer frames to improve the accuracy of predicted future images, an aspect of the technology disclosed herein is a machine learning model that can use images captured during earlier stages of growth to generate predicted future images without relying on later images. Thus, in the example shown in FIG. 4B, during the second stage, the machine learning framework 410 continues to train the machine learning model 415′ using images 406 captured during the first 6 hours of growth. As with the first stage, the second stage of training may be bidirectional, with deep learning architecture 411B sharing layers of the machine learning model 415′ with deep learning architecture 411C. For example, deep learning architecture 411B may use input images 406 (e.g., V1-V k ) to predict the future image V out A machine learning model 415' is trained to generate V out V labelto determine adjustments to the weights of the machine learning model 415'. Additionally, the deep learning architecture 411C compares V out and V k A machine learning model 415' is trained to generate a reconstructed image V1' using .about.V2 as input, and V1' is compared to V1 to determine adjustments to the weights of the machine learning model 415'.

[0060] The machine learning framework 410 can impose constraints 412 on the training of the machine learning model 415'. For example, the machine learning framework 410 can enforce a constraint that a particular layer of the machine learning model 415' matches the weights of a corresponding layer of the machine learning model 415. In some aspects, the constraint may be that the weights of a final layer of the machine learning model 415' match the weights of a final layer of the machine learning model 415. In some aspects, the constraint may be that the weights of an intermediate layer of the machine learning model 415' match the weights of a corresponding intermediate layer of the machine learning model 415.

[0061] In addition to the aspects discussed above, a further aspect of the present disclosure illustrated in FIG. 4B is that additional data acquired in the future can be used to train the machine learning model 415′. For example, the machine learning model 415 may be trained using an initial set of training data. As more data becomes available in the future, the machine learning model 415′ can be trained as discussed above to potentially improve the accuracy of the predicted image.

[0062] 4B, machine learning framework 410 implements bidirectional training (i.e., cyclic training) to train both machine learning model 415 and machine learning model 414'. However, bidirectional training is not required, and in some aspects machine learning framework 410 may train one or both of machine learning models 415 and 415' using a unidirectional method.

[0063] 4C is a block diagram illustrating another machine learning framework 420 that implements two bidirectional passes to train a machine learning model to predict future microbial colony images. The machine learning framework 420 includes deep learning architectures 422A, 422B, and 422C (collectively, "deep learning architectures 422"). Each of the deep learning architectures 422 may be a CNN (including U-Net2D and U-Net3D), a GAN, a T-adversarial GAN, a time-cyclic GAN, or a GAN that uses privileged information.

[0064] In some aspects, deep learning architecture 425A is implemented similarly to deep learning architectures 404A and 404B described above in FIG. 4A and deep learning architecture 411A described above in FIG. 4B. That is, deep learning architecture 422A may be bidirectional and may perform two passes through the image sequence, e.g., predicting a future image V from an initial image in image sequence 406. out The first pass generates the predicted future image V out and a second pass to generate a reconstructed first image V1 based on the first image in the image sequence and the images subsequent to the first image in the image sequence. Similar to deep learning architecture 411A, deep learning architecture 422A includes more training data than the example shown in FIG. 4A. However, in the example of FIG. 4C, the additional training data acquired may include a larger number of sampled images from images 406 acquired during the first 6 hour period of the 24 hour period. In the example shown in FIG. 4C, deep learning architecture 422A initially trains machine learning model 425 using images V1-V6. Thus, in the first stage of training, deep learning architecture 422A utilizes more image samples to improve the accuracy of machine learning model 425.

[0065] During the second stage, deep learning architecture 422B trains machine learning model 425′ using fewer images from image 406. In the example shown in FIG. 4C, deep learning architectures 422B and 422C use half the number of images (e.g., V1, V3, and V5). As with the first stage, the second stage of training may be bidirectional with deep learning architecture 422B sharing layers of machine learning model 425′ with deep learning architecture 422C. For example, deep learning architecture 422B trains machine learning model 425′ using fewer images from image 406 (e.g., V1, V3, and V5). out A machine learning model 425' is trained to generate V out V label to determine adjustments to the weights of the machine learning model 425′. Additionally, the deep learning architecture 422C compares V out , train the machine learning model 425' to generate a reconstructed image V1' using V5 and V3 as inputs, and compare V1' to V1 to determine adjustments to the weights of the machine learning model 425'.

[0066] The machine learning framework 420 can impose constraints 424 on the training of the machine learning model 425'. For example, the machine learning framework 420 can enforce a constraint that a particular layer of the machine learning model 425' matches the weights of a corresponding layer of the machine learning model 425. In some aspects, the constraint may be that the weights of a final layer of the machine learning model 425' match the weights of a final layer of the machine learning model 425. In some aspects, the constraint may be that the weights of an intermediate layer of the machine learning model 425' match the weights of a corresponding intermediate layer of the machine learning model 425.

[0067] In addition to the aspects discussed above, a further aspect of the present disclosure illustrated in FIG. 4C is that the machine learning framework 420 can generate a machine learning model 425′ that, despite using fewer samples during an initial phase of training (e.g., a testing phase), can still have the same or similar accuracy as a machine learning model trained using a larger number of samples.

[0068] 4B, in the example shown in FIG. 4C, machine learning framework 420 implements bidirectional training (i.e., cyclic training) to train both machine learning model 425 and machine learning model 425'. However, bidirectional training is not required, and in some aspects machine learning framework 410 may train one or both of machine learning models 415 and 415' using a unidirectional method.

[0069] FIG. 5 is a block diagram illustrating a system for generating predicted future images of microbial (e.g., bacteria, mold, fungus) colony growth using tiled image data, according to at least one exemplary technique described in this disclosure. In some aspects, images of an image sequence of microbial growth may be tiled prior to training a machine learning model. Tiling can be used in addition to other image transformations to generate more image data for use in training the machine learning model. For example, image tiles can be randomly selected from images in the candidate image data set 301 (FIG. 3) and provided to the training system. Additionally, tiling can be used to reduce image size. For example, in some implementations, the candidate image data set 301 included images having a size of 1844×1956 pixels. Image tiles were created having an image size of 512×512 pixels. In some aspects, if the size of the input images is small and / or if the amount of available training data is already large, the prediction system 502 may not tile the input images.

[0070] In the example shown in FIG. 5, the prediction system 502 is configured to tile images as part of generating predicted microbial colony image data 516. The prediction system 502 may be an implementation of the prediction system 102 of FIG. 2B and / or the prediction system 216 of FIG. 2B. The prediction system 502 includes an image tiler 512, a processing unit 504, and a tile merger 510. The prediction system 502 receives the microbial colony image sequence 112. The image tiler 512 tiles the images with the sequence 112 to generate tile sequences 514A-514D. In some embodiments, the image tiler 512 generates spatially sequential tile sequences. In the example shown in FIG. 5, the tile sequences 514A-514D are generated from left to right and top to bottom. The prediction system 502 provides the tile sequences 514A-514D to the processing unit 504. The processing unit 504 includes an AI engine 506 and a machine learning model 508. The AI ​​engine processes each of the tile sequences 514A-514D using the machine learning model 508 to generate a corresponding predicted tile 515A-515D. The predicted tiles 515A-515D are in the same sequence as the input tile sequences 514A-514D. That is, predicted tile 515A corresponds to tile sequence 514A, predicted tile 515B corresponds to tile sequence 514B, and so on. The tile merger 510 reassembles the predicted tiles 515A-515D in the same sequential order to create a predicted microbial colony image 516.

[0071] 5 shows that the image tiler 512 generates four tile sequences for one input image sequence, the number of tile sequences may be different and may be more or less than four.

[0072] FIG. 6 is a block diagram of an exemplary processing unit of a system for generating predicted future images of microbial growth according to at least one exemplary technique described in this disclosure. FIG. 6 is a block diagram illustrating an exemplary processing unit 600 according to at least one exemplary technique described in this disclosure. The processing unit 600 may be an exemplary or alternative implementation of the processing unit 104 of FIG. 1 and / or the processing unit 504 of FIG. 5. The architecture of the processing unit 600 shown in FIG. 6 is shown for illustrative purposes only. The processing unit 600 should not be limited to the exemplary architecture shown. In other examples, the processing unit 600 may be configured in a variety of different ways. In the example shown in FIG. 6, the processing unit 600 includes a prediction unit 610 configured to generate a predicted microbial colony image based on a sequence of microbial colony images, and the prediction unit 610 may include an AI engine 612 configured to process the microbial colony image sequence using a machine learning model 614 to generate a predicted microbial colony image as an output.

[0073] In some aspects, the machine learning models 614 may include data defining a CNN. In some aspects, the machine learning models 614 may include data defining a generative adversarial network (GAN), a T-adversarial GAN, a U-Net, including U-Net2D and U-Net3D.

[0074] Processing unit 600 may be implemented as any suitable computing system (e.g., at least one server computer, workstation, mainframe, appliance, cloud computing system, and / or other computing system) that may be capable of performing the operations and / or functions described in accordance with at least one aspect of the present disclosure. In some examples, processing unit 600 represents (or is a part of) a cloud computing system, a server farm, and / or a server cluster configured to be connected to system 100 via a wired or wireless connection. In other examples, processing unit 600 may represent or be implemented via at least one virtualized compute instance (e.g., a virtual machine or container) of a data center, a cloud computing system, a server farm, and / or a server cluster. In some examples, processing unit 600 includes at least one computing device, each computing device having a memory and at least one processor.

[0075] As shown in the example of FIG. 6, the processing unit 600 includes a processing circuit 602, at least one interface 604, and at least one storage unit 606. A prediction unit 610 including an AI engine 612 may be implemented as program instructions and / or data executable by the processing circuit 602 stored in the storage unit 606. The storage unit 606 may store a machine learning model 614. The storage unit 606 of the processing unit 600 may also store an operating system (not shown) executable by the processing circuit 602 to control the operation of the components of the processing unit 600. The components, units, or modules of the processing unit 600 may be coupled (physically, communicatively, and / or operatively) using a communication channel for inter-component communication. In some examples, the communication channel includes a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0076] Processing circuitry 602, in one example, may include at least one processor configured to implement functions and / or process instructions for execution within processing unit 600. For example, processing circuitry 602 may be capable of processing instructions stored by storage unit 606. Processing circuitry 602 may include, for example, a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry, or a combination of any of the foregoing devices or circuits.

[0077] To facilitate processing multiple inspection operations in parallel, there may be multiple instances of processing circuitry 602 within processing unit 600. The multiple instances may be of the same type, such as a multi-processor system or a multi-core processor, or the multiple instances may be of different types, such as a multi-core processor with multiple associated graphics processor units (GPUs).

[0078] Processing unit 600 can utilize interface 604 to communicate with external systems over at least one network. In some examples, interface 604 includes an electrical interface configured to electrically couple processing unit 600 to prediction system 102. In other examples, interface 604 can be a network interface (e.g., an Ethernet interface), an optical transceiver, a radio frequency (RF) transceiver, Wi-Fi, or through the use of wireless technology known under the trademark "BLUETOOTH," a telephone interface, or any other type of device capable of transmitting and receiving information. In some examples, processing unit 600 utilizes interface 604 to wirelessly communicate with external systems.

[0079] The storage unit 606 may be configured to store information within the processing unit 600 during operation. The storage unit 606 may include a computer-readable storage medium or a computer-readable storage device. In some examples, the storage unit 606 includes at least a short-term memory or a long-term memory. The storage unit 606 may include, for example, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetic disk, an optical disk, a flash memory, a magnetic disk, an optical disk, a flash memory, or a form of electrically programmable memory (EPROM) or electrically erasable and programmable memory (EEPROM). In some examples, the storage unit 606 is used to store program instructions for execution by the processing circuit 602. The storage unit 606 may be used by software or applications executing on the processing unit 600 to temporarily store information during program execution.

[0080] 7 is a flow diagram illustrating an example operation of a prediction system according to one or more techniques of the present disclosure. The prediction system can receive, by a processing unit including one or more processors, image data for a sequence of images captured at multiple times representing a sample of a microbial colony, where each image of the sequence of images prior to the final image is separated by a sampling time interval between the image and the next image (705). The prediction system can then pass the image data for the sequence of images to a machine learning model, where the machine learning model has been trained to generate image data representing a predicted future image of the microbial colony at a future time, where the machine learning model has been trained using past image data, where the past image data includes a plurality of past image data sets, where each past image data set of the plurality of past image data sets includes image data for a sequence of past images of a corresponding past microbial colony sample, where the prediction time interval between the future time and the capture time of the final image of the sequence of images is greater than each of the sampling time intervals (710). The prediction system can then output image data representing a predicted future image of the microbial colony (715).

[0081] 8 is a flow diagram illustrating an example operation of a training system according to one or more techniques of the present disclosure. The training system can receive past image data, the past image data including a plurality of past image data sets, each past image data set of the plurality of past image data sets including image data for a sequence of past images of a corresponding microbial colony sample, each image of the sequence of past images prior to a final image of the sequence of past images being separated by a sampling time interval between the image and the next image (805). The training system can then train a machine learning model to generate predicted future images at a future time from the sequence of past images, the prediction time interval between the future time and the capture time of the final image of the sequence of past images being greater than each of the sampling time intervals (810). The training system can then adjust weights in a layer of the machine learning model based on a difference between the predicted future image and a target image associated with the microbial colony sample (815).

[0082] FIG. 10 illustrates a predicted future image generated according to at least one exemplary technique described in this disclosure. FIG. 10 illustrates the final images of three different input sequences 1004A, 1004B, and 1004C, which include image samples of the same microbial image colony. Predicted future images 1006A, 1006B, and 1006C correspond to input sequences 1004A, 1004B, and 1004C, respectively, and are the resulting predicted future images generated by a prediction system using the same machine learning model. Each of the image sequences 1004A, 1004B, and 1004C includes 16 input images, but spaced apart to a different total time. For example, sequence 1004A includes 16 images sampled during the first 2.6 hours of microbial colony growth, resulting in a sampling time interval of about 10 minutes between images (e.g., 16 samples x 10 minutes per sample = 160 = 2.6 hours) and a prediction interval of 61.4 hours. Sequence 1004B includes 16 images sampled during the first 8 hours of microbial colony growth, resulting in a sampling time interval of 30 minutes between images (e.g., 16 samples x 30 minutes per sample = 480 minutes = 8 hours) and a prediction interval of 56 hours (e.g., 64 hours - 8 hours = 56 hours). Image sequence 1004C includes 16 images sampled during the first 16 hours of microbial colony growth, resulting in a sampling time interval of 1 hour between images (e.g., 16 samples x 1 hour per sample = 16 hours) and a prediction interval of 48 hours (e.g., 64 hours - 16 hours = 48 hours). Image 1002 is a ground truth image of microbial colony growth after 64 hours have passed.

[0083] As can be seen from Fig. 10, predicted future image 1006A corresponding to image sequence 1004A does not appear to be close to target image 1002. This may indicate that the 2.6 hour image sampling period may not be long enough to generate accurate predicted future images for a microbial colony with a 64 hour growth period. However, predicted future image 1006B corresponding to image sequence 1004B is relatively close to target image 1002, and even after 8 hours, the growth shown in the final image of image sequence 1004B is still not visible to the human eye. Similarly, predicted future image 1006C corresponding to image sequence 1004C is also relatively close to target image 1002. Thus, the results shown in FIG. 10 demonstrate that by using images captured during the first 8 or 16 hours of growth, it is possible to obtain a relatively accurate predicted future image of the microbial growth, resulting in a time saving of 48 hours (i.e., 64 hours - 16 hours) to 56 hours (i.e., 64 hours - 8 hours) compared to waiting the full 64 hours of growth.

[0084] The example shown in Figure 10 utilizes an image sequence containing 16 images. In other implementations, the image sequence may have fewer images or a greater number of images. In some implementations, an input sequence with as few as two or three images may be used to generate satisfactory predicted future images.

[0085] The above discussion has been presented in the context of predicting future images of microbial colonies based on images taken earlier in the growth cycle. The techniques discussed herein are readily applicable to other fields. For example, the techniques can be applied to wound analysis to generate predicted future images of a wound that show how the wound may appear at a future time point based on a sequence of images of the wound.

[0086] The techniques disclosed herein are also applicable to agriculture. Plant growth behaviors such as microbial colony growth and wound healing can have slow and long-term progressions. Using the techniques described herein, image sequences of a field can be used to predict new cultivars and field regions that may be most resistant to disease.

[0087] The techniques described in this disclosure may be implemented at least in part as hardware, software, firmware, or any combination thereof. For example, different aspects of the techniques described may be implemented in at least one processor, including at least one microprocessor, DSP, ASIC, FPGA, and / or any other equivalent integrated circuit or discrete logic circuit, and any combination of such components. The term "processor" or "processing circuit" may generally refer to any of the aforementioned logic circuits alone or in combination with other logic circuits or any other equivalent circuit. A control unit including hardware may also implement at least one of the techniques of this disclosure.

[0088] Such hardware, software, and firmware may be implemented in the same device or in separate devices to support different operations and functions described in this disclosure. Additionally, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logical devices. The description of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware or software components. Rather, the functionality associated with at least one module and / or unit may be performed by separate hardware or software components or may be integrated within a common or separate hardware or software component.

[0089] The techniques described in this disclosure may be embodied or encoded in a computer-readable medium, such as a non-transitory computer-readable medium or computer-readable storage medium that includes instructions. The instructions embedded or encoded in the computer-readable medium can (e.g., when executed) cause a programmable processor or other processor to perform the method. The computer-readable storage medium can include RAM, read-only memory (ROM), programmable read-only memory (PROM), EPROM, EEPROM, flash memory, hard disk, CD-ROM, floppy disk, cassette, magnetic medium, optical medium, or other computer-readable storage medium. The term "computer-readable storage medium" refers to a physical storage medium and not a signal or carrier wave, although the term "computer-readable medium" can include a transitory medium such as a signal in addition to a physical storage medium.

Claims

1. 1. A system comprising: The system includes an image capture device configured to capture image data for a sequence of images representative of growth of a microbial colony at a plurality of times, each image in the sequence of images prior to a final image being separated by a sampling time interval between the images; The system includes a processing unit having one or more processors, the one or more processors being: passing the image data for the sequence of images to a machine learning model, the machine learning model being trained to generate image data representative of one or more predicted future images of the growth of the microbial colony, each of the one or more predicted future images representing a microbial colony at a corresponding future time, the machine learning model being trained using past image data, the past image data including one or more past image data sets, each past image data set of the one or more past image data sets including image data for a sequence of past images of a corresponding past microbial colony sample, a prediction time interval between the future time and a capture time of a last image in the sequence of images being greater than each of the sampling time intervals; outputting the image data representative of the one or more predicted future images of the microbial colony; and configured to execute instructions that cause the processing unit to perform the system.

2. 2. The system of claim 1 , wherein the machine learning model is trained using a weighted loss, the weighted loss assigning a first weight to a first output image that is less than a second weight assigned to a second output image having a corresponding predicted future time that is later than the predicted future time corresponding to the first output image.

3. The system of claim 1 , wherein the machine learning model is trained using a weighted loss that assigns equal weighting to each output image.

4. 2. The system of claim 1 , wherein the machine learning model is trained bidirectionally, a first training direction training the machine learning model to generate the one or more predicted future images from the sequence of past images, and a second training direction training the machine learning model to generate a reconstructed initial image from the one or more predicted future images and an image subsequent to an initial image in the sequence of past images.

5. The system of claim 4 , wherein a layer in the machine learning model is shared by the first training direction and the second training direction.

6. the machine learning model includes a second machine learning model; a first machine learning model trained prior to the second machine learning model using a first training image data set, the first training image data set including a first subset of images of the sequence of past images captured during a sampling period associated with the past microbial colony sample and a second subset of images captured between an end of the sampling period and an end of a growth period of the past microbial colony sample; the second machine learning model is constrained to include one or more layers of the first machine learning model; The system of claim 4.

7. The system of claim 6 , wherein the first machine learning model is trained bidirectionally.

8. The system of claim 6 , wherein the one or more layers include a final layer, a penultimate layer, or one or more middle layers.

9. the machine learning model includes a second machine learning model; a first machine learning model trained prior to the second machine learning model using a first training image data set, the first training image data set including a first subset of images of the sequence of past images captured during a sampling period associated with the past microbial colony samples and a second subset of images captured during the sampling period, the number of images in the first subset of images being greater than the number of images in the second subset of images; the second machine learning model is constrained to use one or more layers of the first machine learning model; The system of claim 1 .

10. The instructions further cause the processing unit to generate, from each image in the sequence of images, a corresponding plurality of image tiles associated with the image, each of the plurality of image tiles corresponding to a different position within the image; The instructions for causing the processing unit to pass image data of the sequence of images to a machine learning model include, for each of the different locations, instructions for causing the processing unit to pass a plurality of image tiles corresponding to the same location to the machine learning model to generate predicted future image tiles corresponding to the same location; the instructions for causing the processing unit to output image data representative of a predicted future image of the microbial colony include instructions for causing the processing unit to assemble the predicted future image tiles for each of the respective different locations to generate image data representative of a predicted future image of the microbial colony. The system of claim 1 .

11. The system of claim 1 , wherein each image in the sequence of images comprises multiple frames of a video recording.

12. The system of claim 1 , wherein the prediction time interval is greater than an input time interval associated with the sequence of images.

13. The system of claim 1 , wherein the microbial colonies comprise bacterial colonies, yeast colonies, fungal colonies, or mold colonies.

14. 1. A method, comprising: receiving, by a processing unit including one or more processors, image data for a sequence of images captured at multiple times representing growth of a sample of microbial colonies, each image prior to a final image in the sequence of images being separated by a sampling time interval between the images; passing the image data for the sequence of images to a machine learning model, the machine learning model being trained to generate image data representative of one or more predicted future images of the microbial colony, each of the one or more predicted future images representing the microbial colony at a corresponding future time, the machine learning model being trained using past image data, the past image data including a plurality of past image data sets, each past image data set of the plurality of past image data sets including image data for a sequence of past images of a corresponding past microbial colony sample, a prediction time interval between the future time and a capture time of a last image in the sequence of images being greater than each of the sampling time intervals; outputting the image data representative of the predicted future image of the microbial colony; A method comprising:

15. 1. A method, comprising: receiving past image data, the past image data including a plurality of past image data sets, each past image data set of the plurality of past image data sets including image data for a sequence of past images of a corresponding microbial colony sample, each image in the sequence of past images prior to a final image in the sequence of past images being separated by a sampling time interval between the images; training the machine learning model to generate, for each past image data set of the plurality of past image data sets, one or more predicted future images of the microbial colony, each image corresponding to a future time from the sequence of past images, a prediction time interval between the future time and a capture time of a last image in the sequence of past images being greater than each of the sampling time intervals; adjusting weights in layers of the machine learning model based on differences between the one or more predicted future images and one or more target images associated with the microbial colony sample; A method comprising:

Images