High content imaging system for generating enhanced images and method of operating same - Patents.com

Abstract

The high-content imaging system includes a stage, a controller, a machine learning system, and an image generator. The controller receives a request including an output imaging configuration, and in response, the controller (1) selects a training model associated with the output imaging configuration, (2) determines an input imaging configuration associated with the training model, and (3) configures the high-content imaging system according to the input imaging configuration. The machine learning system is configured using the training model such that when the machine learning system is presented with an image obtained using the input imaging configuration, the machine learning system generates an output image according to the output imaging configuration. The image generator generates images of a sample on the stage and provides the generated images to the machine learning system, and in response, the machine learning system generates an output image according to the output imaging configuration.

Description

[Technical field]

[0001] (Related Applications) This application claims the benefit of priority from U.S. Provisional Application No. 62 / 925,554, filed October 24, 2019, the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE The present subject matter relates to microscopy systems, and more particularly to high-content imaging systems for generating enhanced images and methods of operating such systems. [Background technology]

[0003] A high content imaging system (HCIS) may be used to acquire microscopy images of one or more biological samples, such as DNA, proteins, cells, etc. The biological samples may be disposed within wells of a microplate having a two-dimensional pattern of wells. Such microplates typically have 96 or 384 wells, but may have more or fewer wells. To acquire images of the biological samples disposed within such microplates, an autofocus system of the HCIS focuses on each well of the microplate or a portion thereof and acquires an image. To develop a high resolution image of the biological sample within a particular well, an imaging sensor may be positioned relative to different portions of the well focused on such portions and acquire images. The images of such different portions may be combined (e.g., concatenated) to generate an image of the entire well. Additionally, the HCIS may develop multiple images of the well, each one of the multiple images being captured using a different imaging configuration. Such imaging configurations define the illumination source used when the image is captured, one or more filters positioned in the optical path between the well (or a portion thereof) and the imaging sensor, etc.

[0004] It should be apparent that the repeated focusing and positioning operations described above combined with scanning the plate and generating a high resolution image can require a significant amount of time. Moreover, such time increases substantially if multiple microplates need to be imaged or if the user of the HCIS needs to check and / or adjust the sample positioning and / or the focus of the HCIS during the imaging process. In addition, image processing operations such as deconvolution, noise reduction, etc., can typically be applied using a computer that receives the images generated by the HCIS, further adding to the amount of time required to generate images of the biological sample from the HCIS suitable for further analysis. Summary of the Invention [Means for solving the problem]

[0005] According to one aspect, a high-content imaging system includes a stage adapted to have a sample disposed thereon, a controller, a machine learning system, and an image generator. The controller receives a request including a specification of an output imaging configuration, and in response, the controller (1) selects a training model associated with the output imaging configuration, (2) determines an input imaging configuration associated with the training model, and (3) configures the high-content imaging system according to the input imaging configuration. The machine learning system is configured using the training model such that when the machine learning system is presented with an image obtained according to the input imaging configuration, the machine learning system generates an output image according to the output imaging configuration. The image generator generates an image of the sample disposed on the stage and automatically provides the generated image to the machine learning system, and in response, the machine learning system generates an output image according to the output imaging configuration.

[0006] According to another aspect, a method of operating a high-content imaging system including a stage on which a sample may be placed, a machine learning system, and an image generator includes receiving, by one or more processors, a request including a specification of an output imaging configuration. The method also includes selecting, by the one or more processors, a training model associated with the output imaging configuration, determining, by the one or more processors, an input imaging configuration associated with the training model in response to receiving the request, and configuring, by the one or more processors, the high-content imaging system according to the input imaging configuration. The method further includes deploying a trained machine learning system operating on the one or more processors using the training model such that when the trained machine learning system is presented with an image obtained according to the input imaging configuration, the trained machine learning system generates an output image according to the output imaging configuration. The method includes the additional steps of automatically generating an image of the sample placed on the stage using the one or more processors and an image capture device, automatically providing the image of the sample to the trained machine learning system, and operating the trained machine learning system to automatically generate an output image according to the output imaging configuration in response to receiving the image of the sample.

[0007] Other aspects and advantages will become apparent upon consideration of the following detailed description and the accompanying drawings, in which like numerals designate like structure throughout the specification. The present invention provides, for example, the following: (Item 1) 1. A high content imaging system, comprising: a stage adapted to receive the sample; a controller receiving a request including a specification of an output imaging configuration, in response to which the controller (1) selects a training model associated with the output imaging configuration, (2) determines an input imaging configuration associated with the training model, and (3) configures the high-content imaging system according to the input imaging configuration; an image generator for generating an image of the sample disposed on the stage in accordance with the input imaging configuration; a machine learning system that automatically receives the image of the sample generated by the image generator and, in response, automatically generates an output image; Equipped with The machine learning system is configured using the training model such that when the machine learning system is presented with an input image obtained according to the input imaging configuration, the machine learning system generates an output image according to the output imaging configuration. (Item 2) 2. The high-content imaging system of claim 1, wherein the machine learning system includes a machine learning system controller and a plurality of graphical processing units, the machine learning system controller receiving the generated image from an image generator and, in response, automatically selecting one of the plurality of graphical processing units to generate the output image. (Item 3) 2. The high-content imaging system of claim 1, further comprising a database of a plurality of training models, wherein the controller receives a further request including a further output imaging configuration, the controller determines that none of the plurality of training models is associated with the further output imaging configuration, and in response, the controller configures the high-content imaging system in accordance with the further output imaging configuration. (Item 4) 2. The high-content imaging system of claim 1, further comprising a machine learning system trainer that receives a further input imaging configuration and a further output imaging configuration, wherein the machine learning system trainer, when presented with a further input image obtained using the further input imaging configuration, trains the machine learning system to develop a further output image according to the further output imaging configuration. (Item 5) 5. The high-content imaging system of claim 4, wherein the machine learning system trainer instructs the image generator to generate a plurality of training images of a training sample positioned on the stage using the further input imaging configuration, and to generate a plurality of ground truth images of the training sample using the further output imaging configuration. (Item 6) The high-content imaging system of item 1, wherein the controller automatically instructs the image generator to automatically generate a plurality of input images of a plurality of samples disposed in a microplate arranged on the stage according to the input imaging configuration, and automatically instructs the machine learning system to automatically generate an output image from each one of the plurality of input images according to the output imaging configuration. (Item 7) 7. The high-content imaging system of claim 6, further comprising a robotic microplate loader, wherein the controller (1) instructs the robotic microplate loader to load an additional microplate onto the stage after the multiple input images are generated, (2) instructs the image generator to automatically generate multiple additional input images of the additional microplate, and (3) instructs the machine learning system to automatically generate multiple additional output images from the input images. (Item 8) 2. The high-content imaging system of claim 1, wherein the controller instructs the image generator to (1) develop a series of input images of the sample taken at varying foci according to the input imaging configuration, and (2) simultaneously provide the series of images to the machine learning system to generate a series of output images according to the output imaging configuration. (Item 9) 2. The high-content imaging system of claim 1, further comprising an image processor that performs at least one of: (1) image processing of the generated images before such images are provided as input to the machine learning system; and (2) image processing of the output images developed by the machine learning system. (Item 10) 2. The high-content imaging system of claim 1, wherein the input imaging configuration defines a first set of physical components present in the high-content imaging system and the output imaging configuration defines a second set of physical components, at least one of the second set of physical components not present in the high-content imaging system. (Item 11) 1. A method of operating a high content imaging system, the high content imaging system including a stage on which a sample can be placed and an image generator, the method comprising: receiving, by one or more processors, a request including a specification of an output imaging configuration and selecting a training model associated with the output imaging configuration; in response to receiving the request, determining, by the one or more processors, an input imaging configuration associated with the training model; configuring, by the one or more processors, the high-content imaging system according to the input imaging configuration; deploying a trained machine learning system operating on the one or more processors using the training model such that when the trained machine learning system is presented with images obtained according to the input imaging configuration, the trained machine learning system generates an output image according to the output imaging configuration; automatically generating an image of a sample disposed on the stage using the one or more processors and an image capture device, and automatically providing the image of the sample to the trained machine learning system; in response to receiving the image of the sample, operating the trained machine learning system to automatically generate an output image according to the output imaging configuration; A method comprising: (Item 12) 12. The method of operating a high-content imaging system of claim 11, wherein the trained machine learning system includes a machine learning system controller and a plurality of graphical processing units, the method further including the steps of operating the machine learning system controller to receive the generated image and, in response, automatically selecting one of the plurality of graphical processing units to generate the output image. (Item 13) 13. A method of operating a high-content imaging system as described in item 12, further comprising the step of operating at least one graphical processing unit of the plurality of graphical processing units remotely from the machine learning system controller. (Item 14) 12. The method of operating a high-content imaging system of claim 11, further comprising: receiving a further output imaging configuration; querying a database of a plurality of training models to identify a training model associated with the output imaging configuration; determining that none of the plurality of training models is associated with the further output imaging configuration; and in response, configuring the high-content imaging system in accordance with the further output imaging configuration. (Item 15) 13. A method of operating a high-content imaging system as described in item 12, further comprising the steps of receiving a further input imaging configuration and a further output imaging configuration, and training the machine learning system to develop a further output image according to the further output imaging configuration when presented with a further input image obtained using the further input imaging configuration, wherein training the machine learning system comprises the steps of instructing the image generator to generate a plurality of training images of a training sample positioned on the stage using the input imaging configuration and to generate a plurality of ground truth images of the training sample using the output imaging configuration. (Item 16) 12. The method of operating a high-content imaging system of claim 11, further comprising the steps of instructing the image generator to automatically generate a plurality of input images of a plurality of samples disposed in a microplate disposed on the stage according to the input imaging configuration, and instructing the machine learning system to automatically generate an output image from each one of the plurality of input images according to the output imaging configuration. (Item 17) 20. The method of operating a high-content imaging system of claim 16, further comprising the steps of: operating a robotic microplate loader to load an additional microplate onto the stage after the multiple input images have been generated; instructing the image generator to automatically generate multiple additional input images of the additional microplate; and instructing the machine learning system to automatically generate multiple additional output images from the additional input images. (Item 18) 12. A method of operating a high-content imaging system as described in Item 11, further comprising the steps of instructing the image generator to develop a series of input images of the sample taken at varying focus according to the input imaging configuration and simultaneously providing the series of images to the machine learning system, and in response, automatically generating, by the machine learning system, a series of output images according to the output imaging configuration. (Item 19) 12. The method of operating a high-content imaging system of claim 11, further comprising the step of transmitting the image generated by the image generator and the output image to a computer external to the high-content imaging system. (Item 20) 12. A method of operating a high-content imaging system as described in item 11, wherein the input imaging configuration defines a first set of physical components present in the high-content imaging system and the output imaging configuration defines a second set of physical components, at least one of the second set of physical components not present in the high-content imaging system. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of a high content imaging system (HCIS) in accordance with the present disclosure.

[0009] [Diagram 2] FIG. 2 is a flow chart of the steps performed by the controller of the HCIS of FIG. 1 to generate an output image.

[0010] [Diagram 3] FIG. 3 is a flow chart of the steps performed by the machine learning system trainer of the HCIS of FIG.

[0011] [Figure 4] FIG. 4 is a flow chart of the steps performed by the controller of the HCIS of FIG. 1 to automatically generate images of multiple microplates loaded therein.

[0012] [Diagram 5] FIG. 5 is a schematic diagram of the machine learning system of the HCIS of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] As described in detail below, a high content imaging system (HCIS) according to the present disclosure includes a stage on which a sample or a microplate with samples can be placed, one or more illumination sources, one or more objective lenses, one or more filters, a focusing device, an imaging sensor, a machine learning system, a controller, and one or more machine learning system training models that can be used in conjunction with the machine learning system. Each training model is associated with an input imaging configuration and an output imaging configuration and includes data necessary to configure and train an untrained machine learning system (e.g., a neural network or another deep learning system). For example, if the untrained machine learning system is a neural network, the training model includes parameters for the interconnection of one or more convolution layers, one or more neuron layers, and one or more pooling layers therebetween. The training model also includes scaling coefficients, kernel weights, and the like associated with such layers.

[0014] An untrained machine learning system may be configured to deploy a trained machine learning system with a particular training model, and then when the trained machine learning system is presented with input (or source) images captured when the HCIS is configured with an input imaging configuration associated with the particular training model, the machine learning system generates output (or target) images that represent images that would have been generated if the HCIS had been configured with an output imaging configuration associated with the particular training model.

[0015] In operation, a user uses a computer in communication with the HCIS to transmit requests to a controller of the HCIS to generate output images of biological samples disposed within wells of a microplate disposed therein. Such requests may include specifications of the output imaging configuration to use in generating and identifying the image, such as the objective lens, resolution, illumination source, one or more optical filters, focusing accuracy, and / or types of imaging aberrations or artifacts to be corrected in the output image. Such aberration / artifact correction may include correcting for objective lens field curvature, deconvolution of the image to improve contrast and / or resolution, and the like.

[0016] The controller analyzes the output image specification and selects a training model associated with such output image specification. The controller then determines an input image specification associated with the selected training model, configures an untrained machine learning system with the training model, generates a trained machine learning system, configures the HCIS according to the input imaging configuration to capture images. The controller provides the captured images as inputs to the trained machine learning system, which generates output images representative of images that would have been captured if the HCIS had been configured with the output imaging configuration, and the controller transmits the output images generated by the HCIS to a user's computer.

[0017] In some embodiments, a training model may be associated with an input configuration that requires a first set of physical components (e.g., objectives, illumination sources, etc.) to be installed in the HCIS, and an output configuration that requires a second set of physical components, some of which may not be present in the HCIS. When an untrained machine learning system of the HCIS is configured with such a training model to deploy a trained machine learning system, the HCIS (with the trained machine learning system) will generate output images that represent images that would have been obtained with the second set of physical components from input images obtained using the first set of physical components. Thus, for example, in such an embodiment, the HCIS is capable of generating output images similar to those that would be obtained if the HCIS included the additional components, without the costs associated with such additional components.

[0018] In some embodiments, the HCIS can be operated using an input imaging configuration that quickly captures images (e.g., captures low-resolution images using coarse focusing). Training models associated with such input and output configurations that require more time to capture images (e.g., captures high-resolution images using very precise focusing) can be used to train a machine learning system. An HCIS with such a trained machine learning system generates output images as if they were captured from input images captured at a relatively faster rate using the more time-consuming configuration, which improves the throughput of the HCIS. Also, since additional image processing / correction operations can be incorporated into the training model used to train the machine learning system, performing these operations does not require any additional time beyond what it takes for the trained machine learning system to generate output images from input images, thus resulting in additional throughput improvements.

[0019] 1, an embodiment of HCIS 100 includes a controller 102, a stage 104 on which samples (or a microplate carrying such samples) may be disposed, one or more objective lenses 106, one or more illumination sources 108, an image sensor 110, and a focusing device 112. As described below, HCIS 100 may be used to generate transmitted light and / or fluorescent images of samples 114 disposed within wells 116 of a microplate 118 disposed on stage 104. Stage 104 may be an XY stage that is movable along a plane parallel to the imaging plane of the image sensor, or an XYZ stage that is movable along a plane parallel to and perpendicular to the imaging plane of image sensor 110.

[0020] To generate a transmitted light image of the sample 114, the controller 102 operates the illumination source 108a to transmit light through the sample 114 (and the microplate 118 in which the sample 114 is disposed). The light is then directed through a selected objective lens 106, and optionally redirected by a mirror 120, to form an image on the surface of the image sensor 110. The image sensor 110 generates signals in accordance with the light sensed thereby, and such signals are converted into an image by an image generator 122.

[0021] To generate a fluorescence image of the sample 114, the controller 102 operates the illumination source 108b to generate light that is redirected by one or more mirrors 120 through the objective lens 106, through the microplate 118, and onto the sample 114. The light that reaches the sample 114 may excite the sample 114, causing it to fluoresce. Light emitted by the sample 114 passes through the objective lens 106 and is directed by the one or more mirrors 120 to form an image on the surface of the image sensor 110, which generates a signal in response to such image that is converted into an image by the image generator 122.

[0022] One or more optical filters may also be optionally disposed in the optical path between the illumination source 108 and the sample 114 and / or between the sample 114 and the image sensor 110.

[0023] As will be appreciated by those skilled in the art, the image generator 122 receives signals and converts such signals into pixel values ​​for an image. Additionally, the image generator 122 may receive signals associated with sub-images of the biological sample 114 as the image sensor 110 scans such sample 114, and combine such sub-images into a complete image.

[0024] Continuing with reference to FIG. 1, during operation, the controller 102 receives a request from the computer 124 operated by a user to generate one or more images of the sample 114. As noted above, such a request includes a specification of the output imaging configuration. The controller 102 queries the database 126 to select a training model associated with the output imaging configuration received with the request. In some embodiments, for some training models in the database 126, the HCIS 100 will generate images more efficiently when configured with an input imaging configuration associated with the training model than when configured with an output imaging configuration associated with the request. Furthermore, the input imaging configuration associated with a training model in the database 126 may specify equipment (e.g., illumination source, objective lens, etc.) installed within the HCIS 100, while the output imaging configuration received with the request may require equipment not installed within the HCIS 100.

[0025] After selecting a particular training model from database 126, controller 102 configures an untrained machine learning system 128 according to the training model and deploys the trained machine learning system 128. In some embodiments, controller 102 may reset the machine learning system 128 to an untrained state before configuring such machine learning system 128 with the selected training model.

[0026] In some embodiments, the training model is stored on the user computer 124 or on another computer remote from the HCIS 100. In such embodiments, the controller 102 requests the training model from the computer on which the training model is stored. In some embodiments, the user computer 124 may provide the training model to the controller 102 as part of a request sent thereby to generate one or more images of the sample 114. In such embodiments, the user computer 124 may load the training model from a storage device associated therewith or from a computer remote therefrom. Additionally, in some cases, the user computer 124 may ask the user to, for example, identify one or more data files in which the training model is stored and provide such data files to the controller 102.

[0027] After the trained machine learning system 128 is deployed, the controller 102 configures the HCIS 100 according to the input imaging configuration associated with the selected training model. In particular, the controller 102 loads the objective lens 106, configures the image sensor 110, operates the illumination source 108 and the focusing device 112 according to the input imaging configuration, and directs the image sensor 110 and the image generator 122 to acquire an image.

[0028] After an image is acquired, it is presented as an input to a trained machine learning system 128, which generates an output image therefrom. The controller 102 receives the output image and transmits such output image to the user computer 124. In some embodiments, the image generator 122, the trained machine learning system 128, and the controller 102 all have access to a shared memory that allows images generated by one such component to be accessed by another component. In other embodiments, these components communicate over a local area network (or another type of network) and transmit images between them.

[0029] In some embodiments, the HCIS 100 may include an image quality analyzer (not shown) that receives the output image generated by the trained machine learning system 128. The image quality analyzer applies image analysis algorithms to, for example, detect noise or other artifacts in the image and develop a score that represents the quality of the output image. The image quality analyzer then provides the output image and the score to the controller 102 for transmission to the user computer 124.

[0030] In some embodiments, the HCIS 100 may include a first image processor (not shown) that automatically receives the acquired image from the image generator 122. The first image processor applies one or more image processing functions to the acquired image, e.g., scaling the image, reducing noise in the image, etc., and the processed image is automatically provided as an input to the machine learning system 128. In response, the machine learning system 128 generates an output image, which is then provided to the controller 102, as described above.

[0031] In some embodiments, the first image processor is integral with the image generator 122 and the generated image is the output of such image processor.

[0032] In some embodiments, a second image processor (not shown) may process the output image developed by the machine learning system 128, e.g., scale the image, reduce noise in the image, etc., and such processed images are provided to the controller 102 and to the user computer 124, as described above.

[0033] Whether the first and / or second image processor is used may be determined by a request received from the user computer 124 or may be determined according to the input and / or output imaging configuration.

[0034] Figure 2 shows a flow chart 200 of steps performed by controller 102 to generate an image according to a request received from computer 124. With reference to Figures 1 and 2, in step 202, controller 102 receives a request to capture an image. As discussed above, the request specifies an output imaging configuration for generating the output image.

[0035] In some embodiments, the user instructs the computer 124 to send a request to the controller 102 that includes an indication of whether the HCIS 100 should be operated in non-augmented or augmented mode. In the non-augmented mode, the machine learning system 128 is not used. Rather, the controller 102 instructs the image generator 122 to capture an image, and any such captured image is transmitted by the controller 102 to the user computer 124. In the augmented mode, the trained machine learning system 128 is invoked as described herein to augment or generate an output image from an input image generated by the image generator 122. For example, a request received by the controller 102 may specify that the augmented mode is to be used, that the input image should be obtained using 10x magnification, and that the machine learning system 128 should generate an output image representative of an image that would have been captured using 20x magnification (i.e., enhanced resolution). Similarly, the request may specify that an enhanced mode is to be used, that an input image should be obtained using a short exposure time, and that the machine learning system 128 should generate an output image that represents the image that would have been captured if a long exposure time had been used (i.e., an enhanced exposure).

[0036] In step 204, the controller 102 queries the database 126 to determine whether a training model is stored therein in association with the output imaging configuration specified in the request. If such a training model is identified, the controller 102 proceeds to step 206; otherwise, the controller 102 proceeds to step 207. 207 Proceed to.

[0037] In step 206, the controller 102 loads the training models identified in step 204, and in step 208 configures the untrained machine learning system 128 with the loaded training models and deploys the trained machine learning system 128.

[0038] In step 210, the controller 102 configures the HCIS 100 according to the input imaging configuration associated with the training model identified in step 204. The controller 102 then instructs the image sensor 110 and / or the image generator 122 to capture and generate images. In some embodiments, once the controller 102 configures the HCIS 100 in step 210, the controller 102 also configures the image generator 122 to automatically provide images generated thereby as inputs to the machine learning system 128. For example, in some cases, the image generator 122 may store the images generated thereby in a data storage device (e.g., memory, disk, etc.) accessible by the machine learning system 128, or transmit the images using an inter-process communication channel between the image generator 122 and the machine learning system 128. The machine learning system 128 may be configured to poll such a data storage device or channel for the presence of image data and, when available, load the image data as input. In other embodiments, the image generator 122 may invoke the machine learning system 128 automatically, for example as a function call, and pass the image to the machine learning system 128 as part of such call.

[0039] In step 214, the controller 102 receives the output image generated by the machine learning system 128, either via a shared data storage device or a communication channel, as previously described.

[0040] At step 216, the controller 102 transmits the output images received from the machine learning system 128 to the computer 124. In some embodiments, the controller 102 also receives images from the image generator 122, e.g., images provided to the machine learning system 128 at step 212, and at step 216 transmits to the computer 124 both the images received from the image generator 122 and the images generated by the machine learning system 128.

[0041] In some embodiments, the request received from the controller 102 in step 202 may specify that the enhanced mode should be used to generate multiple output images from one input image captured in step 212. For example, the request may specify that the input image should be captured using an input imaging configuration (e.g., using a 10x objective lens 106), and that the first and second output images should be generated according to a first output imaging configuration (e.g., using a 20x objective lens 106) and a second output imaging configuration (e.g., using a 30x objective lens 106). In response to such a request, the controller 102 loads a first training model associated with the input imaging configuration and the first output imaging configuration, and a second training model associated with the input imaging configuration and the second output imaging configuration. The controller 102 then trains the first and second instances of the machine learning system 128 with the first and second training models. In such an embodiment, the controller 102 directs the image generator 122 to provide input images generated thereby as inputs to each of the first and second instances of the trained machine learning system 128, each instance of the trained machine learning system 182 generating first and second output images, respectively, in response to such input images. The controller 102 receives the first and second output images and transmits such output images (and optionally, the input images) to the user computer 124.

[0042] If, in step 204, the controller 102 determines that the database 126 does not contain a training model associated with the output imaging configuration received with the request in step 202, the controller 102 performs the steps 207 In step 216, the controller 102 configures the HCIS 100 according to the output imaging configuration specified in the request. In step 218, the controller 102 instructs the image sensor 110 and / or the image generator 122 to capture an image. In step 220, the controller 102 receives the captured image from the image generator 122, and in step 216, the controller 102 transmits the captured image to the computer 124.

[0043] In some embodiments, the controller 102 automatically performs step 204 if a suitable training model is not identified in step 204. 207 In another embodiment, the controller 102 performs steps -216. 207 In step S206, the controller 102 instructs the computer 124 to inform the user that a suitable training model is not available in the database 126. In some embodiments, the controller 102 further instructs the computer 124 (also in step S206) to 207 HCIS 100 (at step 202) to prompt the user whether HCIS 100 should be configured according to the output imaging configuration received in step 202. If computer 124 transmits to controller 102 that the user responded affirmatively to such a prompt, controller 102 proceeds to step 207 Do -216.

[0044] Referring to FIG. 1, some embodiments of the HCIS 100 include a machine learning system trainer. 130 and deploying additional training models for use with HCIS100 to provide additional enhanced functionality.

[0045] Figure 3 is a flowchart 300 of steps taken by the machine learning system trainer 130 to develop a new training model for use with the machine learning system 128. With reference to Figures 1 and 3, in step 302, the machine learning system trainer 130 receives a request to develop a new training model. The request includes an input imaging configuration and an output imaging configuration associated with the new training model.

[0046] In step 304 , the machine learning system trainer 130 receives an indication that a microplate has been loaded into the HCIS 100 .

[0047] At step 306, the machine learning system trainer 130 instructs the controller 102 to configure the HCIS 100 according to the input imaging configuration specified in the request received at step 302. At step 308, the machine learning system trainer 130 instructs the controller 102 to capture a number of training images and store such images in a data storage device (not shown) accessible by the machine learning system trainer 130 and the machine learning system 128.

[0048] At step 310, the machine learning system trainer 130 instructs the controller 102 to configure the HCIS 100 according to the output imaging configuration specified in the request received at step 302. At step 312, the machine learning system trainer 130 instructs the controller 102 to obtain a number of ground truth images, each of which is associated with a particular training image captured at step 308. Both such ground truth images and the particular training images are images taken using the input imaging configuration and the output imaging configuration, respectively, of the same portion of the biological sample 114 disposed in the microplate 118.

[0049] In some embodiments, the machine learning system trainer 130 may apply additional image processing functions to each ground truth image according to the output imaging configuration. Such additional image processing functions may include lens field curvature correction, deconvolution, contrast enhancement, shadow correction, image flattening, image stretching, noise removal, etc.

[0050] In step 314, the machine learning system trainer 130 trains the machine learning system 128 using a portion of the first plurality of training images and ground truth images corresponding to such training images in the first portion. In particular, the machine learning system trainer 130 operates the machine learning system 128 using a selected one of the portion of the first training images and receives a predicted image generated by the machine learning system 128. For each pixel of the predicted image, the machine learning system trainer 130 calculates an error value between such predicted pixel and a corresponding pixel of a ground truth image associated with the selected training image. Such an error value may be calculated using a loss function, such as, for example, a weighted classification-specific cross-entropy function. The error value calculated for all of the pixels of the predicted image is used to adjust parameters of the machine learning system 128, for example, using backpropagation, as will be understood by those skilled in the art. The machine learning system trainer 130 develops such error values ​​and performs parameter adjustment using all of the images that comprise the portion of the first training images.

[0051] In step 316, the machine learning system trainer 130 evaluates the performance of the trained machine learning system developed in step 314. In particular, the machine learning system trainer 130 selects a portion of the second training images as evaluation images and presents each of the evaluation images as an input to the machine learning system 128. The machine learning system 128 generates a predicted image in response to each evaluation image presented as an input. For each evaluation image, the machine learning system trainer 130 compares the pixels of the predicted image generated therefrom by the machine learning system 128 with the corresponding pixels of the ground truth image associated with such evaluation image and develops an aggregate error value. All such developed aggregate error values ​​are combined to form an aggregate error metric (e.g., a percentage of the pixels of the predicted image are within a predetermined threshold of the corresponding pixels of the ground truth image).

[0052] In step 318, the machine learning system trainer 130 compares the aggregate error metric to a predetermined acceptable error, and if the aggregate error metric is greater than the predetermined error, the machine learning system trainer 130 proceeds to step 314 to further train the machine learning system 128 using a further portion of the training images and the corresponding ground truth images. In some embodiments, the machine learning system trainer 130 may instruct the computer 124 to display the aggregate error metric and query a user of such computer 124 whether to perform further training. In other embodiments, the machine learning system trainer 130 also determines in step 318 whether to perform additional training according to the quantity of training images used for training, the number of training iterations performed (in step 314), the rate of improvement of the aggregate error metric between successive training iterations, the amount of time performed for training, and other such conditions that would be apparent to one of ordinary skill in the art. If additional training is justified ... 314 Proceed to.

[0053] In some embodiments, metrics may be developed that do not rely on pixel-by-pixel comparison of predicted images generated from the evaluation images and ground truth images associated with the evaluation images. For example, the machine learning system trainer 130 may run object classification operations on the predicted and ground truth images and compare the number and types of objects identified in each image. The percentage of objects of each type that match may be used to evaluate the performance of the trained machine learning system 128, develop aggregate metrics, and / or present to a user of the computer 124.

[0054] In some cases, the machine learning system trainer 130 may determine in step 318 that the aggregate error metric exceeds a predetermined acceptable error but further training is not justified (e.g., the aggregate error metric is not improving). In such cases, the machine learning system trainer 130 may instruct the computer to display such aggregate error metric along with a message that the aggregate error metric exceeds a predetermined acceptable error but that further training will not be performed.

[0055] If, in step 318, the machine learning system trainer 130 determines that no further training is to be performed, then, in step 320, the machine learning system trainer 130 retrieves parameters of the trained machine learning system 128 and stores such parameters in the database 126 as a training model associated with the input and output imaging configurations received in step 302. The machine learning system trainer 130 then terminates.

[0056] In some embodiments, in step 320, the machine learning system trainer 130 transmits the training model developed thereby to the user computer 124 or another computer remote from the HICS 100 for storage.

[0057] The HCIS 100 may include a robotic microplate loader (not shown) that will be apparent to one of ordinary skill in the art, and the HCIS 100 may be operated to automatically generate images of wells 116 of a plurality of microplates 118. Figure 4 is a flowchart 400 of the steps performed by the controller 102 to operate the HCIS 100 in this manner.

[0058] A user loads the microplate 118 to be imaged into the holding area of ​​the microplate loader and instructs the computer 124 to send a request to the HCIS 100 to scan the loaded microplate 118 .

[0059] 1 and 4, in step 402, the controller 102 receives a request to image, including an output imaging configuration, from the computer 124. In step 404, the controller 102 instructs the robotic microplate loader to load a first microplate.

[0060] In step 406, the controller 102 selects and loads from the database 126 a training model associated with the output imaging configuration. In particular, the controller 102 selects the training model associated with the output imaging configuration.

[0061] In step 408, the controller 102 configures the HCIS 100 according to the input imaging configuration associated with the training model selected in step 406 to generate one or more test images of the samples 114 disposed in the microplate 118, transmits such test images to the computer 124, and performs, for each such test image, steps that are the same as steps 212-216 shown in Figure 2. In addition, the controller 102 instructs the computer 124 to display such test images and prompt the user to confirm that the images are acceptable.

[0062] In step 410, the controller 102 checks whether the user indicated that the test images are acceptable and proceeds to step 412. If not, in step 414, the controller 102 may request additional adjustments to the HCIS 100 (e.g., changing the illumination, changing the focus, changing the input and / or output imaging configurations, changing the sample, modifying the machine learning system, etc.). 128 Changing the architecture of machine learning systems 128 In step 406, the controller 102 receives a training model associated with the modified output imaging configuration, such as a change in a hyperparameter of the HCIS 100 or an output imaging configuration. The controller 102 adjusts the components of the HCIS 100 according to the adjustments. The controller 102 then returns to step 406. In step 406, if the output imaging configuration was changed, the controller 102 loads the training model associated with the modified output imaging configuration.

[0063] In some embodiments, the user may adjust the configurations in step 414 to build a more robust training module. For example, the user may instruct the controller 102 to develop a training model using a plurality of first, second, and third sets of training images captured using 10, 50, and 200 millisecond exposures, respectively, and a ground truth image captured using a 1,000 millisecond exposure. Alternatively, the user may instruct the controller 102 to develop a model using training fluorescent images captured using different wavelengths of light to generate an output image associated with a particular output imaging configuration. A machine learning system developed using multiple input imaging configurations and configured using a training model associated with one output imaging configuration may be capable of generating an output image associated with an output imaging configuration from input images captured using multiple different input imaging configurations. As would be understood by one skilled in the art, the multiple different imaging configurations should be related to the one used to develop the training model.

[0064] In step 412, the controller 102 automatically generates and transmits images of all of the wells 116 in the microplate 118 loaded in step 404. As should be apparent to one of ordinary skill in the art, for each image generated, the controller 102 adjusts the stage 104 to position the unimaged wells 116 (or portions thereof) of the microplate 118 relative to the lens 106, illumination source 108, and image sensor 110. The controller 102 then performs steps that are the same as steps 212-216 shown in FIG. 2 to acquire images of such wells 116 and transmit the acquired images to an image storage device (not shown) accessible by the computer 124.

[0065] In step 416, the controller 102 determines whether any additional microplates remain to be imaged. If so, the controller 102 proceeds to step 404 and instructs a robotic microplate loader (not shown) to load the unimaged microplates onto the stage 104. Otherwise, the controller 102 exits.

[0066] With reference to FIG. 2, it should be apparent to those skilled in the art that the machine learning system 128 may be implemented using a graphical processing unit (GPU). With reference also to FIGS. 2 and 5, the machine learning system 128 includes a machine learning system controller 500 and one or more GPUs 502. In step 208 (FIG. 2), the controller 102 instructs the machine learning system controller 500 to configure each GPU 502 with the training model selected in step 206 (FIG. 2). The image generator 122 then provides each image generated thereby to the machine learning system controller 500. The machine learning system controller 500 selects a GPU 502 that is not in use and transfers the generated image thereto for processing. As each GPU 502 completes the generation of an output image, the machine learning system controller 500 receives the output image developed by such GPU and transfers such output image to the controller 102, which in turn transmits the output image to the computer 124. In this manner, the automated scanning process illustrated in FIG. 5 may be implemented in a high-throughput manner. Additionally, the performance of HCIS 100 may be scaled up or down according to the number of GPUs 502 included therein. Additionally, it should be apparent that machine learning system controller 500 may use instances of GPUs installed within HCIS 100 and / or available through cloud service providers such as Amazon AWS, Google Cloud, etc.

[0067] Referring again to FIG. 1 , in some embodiments, the machine learning system 128 is a convolutional neural network. In some embodiments, the machine learning system or neural network 128 is configured using AutoML and NASNet technologies developed by Google Inc. of Mountain View, California. It should be apparent that other neural network technologies known to those skilled in the art may also be used, including, for example, fully convolutional DenseNet, neural networks optimized for machine vision applications, and the like. It should be apparent that the machine learning system 128 may be another type of machine learning system, including random forest trees and the like.

[0068] Although the embodiments described above are directed to the use of a machine learning system 128 within HCIS100 to generate output images associated with output imaging configurations from input images captured using different input imaging configurations of HCIS100, those skilled in the art will understand that such embodiments may be adapted for use with other types of microscopy and / or imaging systems.

[0069] Although the HCIS 100 described above is described as being used to generate individual two-dimensional images, such HCIS 100 may be adapted to generate a series of two-dimensional transmitted light images of samples 114 disposed on a microplate 118 taken at different focal points that represent a three-dimensional representation of such samples 114, with such images comprising a series being associated with substantially the same location on the microplate 118 and corresponding images of the series being associated with different focal points (i.e., different Z locations). In some embodiments, the images comprising a series may all be provided simultaneously to a machine learning system 128, which generates a series of corresponding output images.

[0070] In some embodiments, the machine learning system 128 may be trained to perform super-resolution in three dimensions. For example, such machine learning system 128 may generate an output series of images from an input series of captured images. For example, the input series of images may comprise a plurality of captured images, where the focal distance between successive captured images is varied by a first predetermined amount. The generated output series of images comprises a series of images representing a series of images that would have been captured if the HCIS 100 were operated to vary the focal distance between successive images by a second predetermined amount. Typically, the second predetermined amount will be less than the first predetermined amount. For example, the focal distance between the input series of successive images may be varied by 2 microns, and the machine learning system 128 may generate from such input series an output series representing successive images that would have been captured if the focal distance was varied by 0.5 microns. It should be apparent to one skilled in the art that a training model may be deployed to train the machine learning system 128 to perform such super-resolution using the steps described above in connection with FIG. 3.

[0071] It should be apparent to one skilled in the art that the foregoing disclosure may be applied to other types of imaging systems, such as, for example, confocal microscopy systems, microscopy systems using structured illumination, etc. Furthermore, it is apparent that a training model associated with an imaging configuration associated with a first type of HCIS (e.g., widefield microscopy) and an output imaging configuration with a second type of HCIS (e.g., confocal microscopy) may be used to configure an untrained machine learning system to deploy a trained machine learning system. The first type of HCIS with the trained machine learning system may then be used to generate an output image representative of an image captured using the second type of HCIS.

[0072] It should be apparent to one skilled in the art that any combination of hardware and / or software may be used to implement the HCIS 100 described herein. It should be understood and appreciated that one or more of the processes, sub-processes, and process steps described in connection with FIGS. 1-5 may be implemented by hardware, software, or a combination of hardware and software on one or more electronic or digital control devices. The software may reside in a software memory (not shown) within a suitable electronic processing component or system, such as, for example, one or more of the functional systems, controllers, devices, components, modules, or sub-modules depicted diagrammatically in FIGS. 1-5. The software memory may include an ordered listing of executable instructions for implementing logical functions (i.e., "logic" which may be implemented in digital form, such as digital circuitry or source code, or in analog form, such as analog sources, such as analog electrical, audio, or video signals). The instructions may be executed within a processing module or controller (e.g., controller 102, image generator 122, machine learning system 128, and machine learning system trainer 130 of FIG. 1 and machine learning system controller 500 and GPU 502 of FIG. 5), which may include, for example, one or more microprocessors, general purpose processors, combinations of processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), or application specific integrated circuits (ASICs). Furthermore, the schematic diagrams illustrate logical divisions of functions with physical (hardware and / or software) implementations that are not limited by the architecture or physical layout of the functions. The example systems described herein may be implemented in various configurations and operate as hardware / software components in a single hardware / software unit or in separate hardware / software units.

[0073] The executable instructions may be implemented as a computer program product having instructions stored therein that, when executed by a processing module of an electronic system, instruct the electronic system to execute the instructions. The computer program product may optionally be embodied in any non-transitory computer-readable storage medium for use by or in association with an instruction execution system, apparatus, or device, such as an electronic computer-based system, a processor-containing system, or other system that may selectively fetch instructions from and execute the instructions from the instruction execution system, apparatus, or device. In the context of this document, a computer-readable storage medium is any non-transitory means that may store a program for use by or in association with an instruction execution system, apparatus, or device. A non-transitory computer-readable storage medium may optionally be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. A non-exhaustive list of more specific examples of non-transitory computer readable media includes: an electrical connection having one or more wires (electronic), a portable computer diskette (magnetic), random access, i.e., volatile memory (electronic), read-only memory (electronic), erasable programmable read-only memory (electronic), such as flash memory, compact disc memory, e.g., CD-ROM, CD-R, CD-RW, etc. (optical), and digital versatile disc memory, i.e., DVD (optical).

[0074] Also, it should be understood that receiving and transmitting signals or data as used herein means that two or more systems, devices, components, modules, or sub-modules are capable of communicating with each other via signals traveling through some type of signal path. The signals may be communication, power, data, or energy signals, which may communicate information, power, or energy from a first system, device, component, module, or sub-module to a second system, device, component, module, or sub-module along a signal path between the first and second systems, devices, components, modules, or sub-modules. The signal path may include physical, electrical, magnetic, electromagnetic, electrochemical, optical, wired, or wireless connections. The signal path may also include additional systems, devices, components, modules, or sub-modules between the first and second systems, devices, components, modules, or sub-modules.

[0075] All references cited in this specification, including publications, patent applications, and patents, are herein incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0076] The use of the terms "a" and "an" and "the" and similar references in the context of describing the present invention (particularly in the context of the claims below) is intended to be interpreted to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of individually referring to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "etc.") provided herein is intended merely to better elucidate the disclosure and does not impose limitations on the scope of the disclosure, unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0077] Numerous modifications to the present disclosure will be apparent to those skilled in the art in view of the foregoing description. It is understood that the illustrated embodiments are illustrative only and should not be taken as limiting the scope of the present disclosure.

Claims

1. 1. A high-content imaging system, comprising: a stage adapted to receive the sample; a controller receiving a request including a specification of an output imaging configuration, in response to which the controller: (1) selects a training model associated with the output imaging configuration; (2) determines an input imaging configuration associated with the training model, the input imaging configuration defining a first set of physical components for capturing images, images captured by the first set of physical components being used as input training data when the training model was deployed; and the output imaging configuration defining a second set of physical components for capturing images, images captured by the second set of physical components being used as target output training data when the training model was deployed; and (3) configures the high-content imaging system in accordance with the input imaging configuration such that the high-content imaging system captures images using the first set of physical components. an image generator for generating an image of the sample disposed on the stage together with the first set of physical components; a machine learning system that automatically receives the image of the sample generated by the image generator and, in response, automatically generates an output image; Equipped with The machine learning system is configured using the training model such that when the machine learning system is presented with an input image obtained according to the input imaging configuration, the machine learning system generates an output image according to the output imaging configuration.

2. 2. The high-content imaging system of claim 1, wherein the machine learning system includes a machine learning system controller and a plurality of graphical processing units, the machine learning system controller receiving the generated image from an image generator and, in response, automatically selecting one of the plurality of graphical processing units to generate the output image.

3. 2. The high-content imaging system of claim 1, further comprising a database of a plurality of training models, wherein the controller receives a further request including a further output imaging configuration defining a third set of physical components for capturing images, the controller determines that none of the plurality of training models is associated with the further output imaging configuration, and in response, the controller configures the high-content imaging system in accordance with the further output imaging configuration, thereby causing the high-content imaging system to capture images using the third set of physical components.

4. A high-content imaging system as described in claim 1, further comprising a machine learning system trainer that receives a further input imaging configuration defining a fourth set of physical components for capturing an image and a further output imaging configuration defining a fifth set of physical components for capturing an image, wherein the machine learning system trainer, when presented with a further input image obtained using the further input imaging configuration, trains the machine learning system to develop a further output image in accordance with the further output imaging configuration.

5. 5. The high-content imaging system of claim 4, wherein the machine learning system trainer instructs the image generator to generate a plurality of training images of training samples disposed on the stage using the fourth set of physical components and to generate a plurality of ground truth images of the training samples using the fifth set of physical components.

6. 2. The high-content imaging system of claim 1, wherein the controller automatically instructs the image generator to automatically generate a plurality of input images of a plurality of samples disposed in a microplate disposed on the stage according to the input imaging configuration, and automatically instructs the machine learning system to automatically generate an output image from each one of the plurality of input images according to the output imaging configuration.

7. 7. The high-content imaging system of claim 6, further comprising a robotic microplate loader, wherein the controller (1) instructs the robotic microplate loader to load an additional microplate onto the stage after the multiple input images are generated, (2) instructs the image generator to automatically generate multiple additional input images of the additional microplate, and (3) instructs the machine learning system to automatically generate multiple additional output images from the input images.

8. 2. The high-content imaging system of claim 1, wherein the controller instructs the image generator to (1) develop a series of input images of the sample taken at varying foci according to the input imaging configuration, and (2) simultaneously provide the series of images to the machine learning system to generate a series of output images according to the output imaging configuration.

9. 2. The high-content imaging system of claim 1, further comprising an image processor configured to at least one of: (1) image process the generated images before such images are provided as input to the machine learning system; and (2) image process the output images developed by the machine learning system.

10. The high-content imaging system of claim 1, wherein at least one of the second set of physical components is not present within the high-content imaging system.

11. 1. A method of operating a high content imaging system, the high content imaging system including a stage on which a sample can be placed and an image generator, the method comprising: receiving, by one or more processors, a request including a specification of an output imaging configuration and selecting a training model associated with the output imaging configuration; in response to receiving the request, determining, by the one or more processors, an input imaging configuration associated with the training model, the input imaging configuration defining a first set of physical components for capturing images, images captured by the first set of physical components being used as input training data when the training model was deployed, and the output imaging configuration defining a second set of physical components for capturing images, images captured by the second set of physical components being used as target output training data when the training model was deployed; configuring, by the one or more processors, the high-content imaging system according to the input imaging configuration such that the high-content imaging system captures images using the first set of physical components; deploying a trained machine learning system operating on the one or more processors using the training model such that when the trained machine learning system is presented with images obtained according to the input imaging configuration, the trained machine learning system generates output images according to the output imaging configuration; automatically generating an image of a sample disposed on the stage using the one or more processors, the first set of physical components, and the image generator, and automatically providing the image of the sample to the trained machine learning system; in response to receiving the image of the sample, operating the trained machine learning system to automatically generate an output image according to the output imaging configuration; A method comprising:

12. 12. The method of operating a high-content imaging system of claim 11, wherein the trained machine learning system includes a machine learning system controller and a plurality of graphical processing units, the method further comprising the steps of operating the machine learning system controller to receive the generated image and, in response, automatically selecting one of the plurality of graphical processing units to generate the output image.

13. The method of claim 12 further comprising operating at least one graphical processing unit of the plurality of graphical processing units remotely from the machine learning system controller.

14. A method of operating a high-content imaging system as described in claim 11, further comprising the steps of receiving a further output imaging configuration defining a third set of physical components for capturing images; querying a database of a plurality of training models to identify a training model associated with the further output imaging configuration; determining that none of the plurality of training models is associated with the further output imaging configuration; and in response, configuring the high-content imaging system in accordance with the further output imaging configuration, thereby causing the high-content imaging system to capture images using the third set of physical components.

15. A method of operating a high-content imaging system as described in claim 12, further comprising the steps of receiving a further input imaging configuration defining a fourth set of physical components for capturing an image and a further output imaging configuration defining a fifth set of physical components for capturing an image, and training the machine learning system to develop a further output image in accordance with the further output imaging configuration when presented with a further input image obtained using the further input imaging configuration, wherein training the machine learning system includes the steps of instructing the image generator to generate a plurality of training images of a training sample positioned on the stage using the fourth set of physical components and to generate a plurality of ground truth images of the training sample using the fifth set of physical components.

16. 12. The method of operating a high-content imaging system of claim 11, further comprising the steps of instructing the image generator to automatically generate a plurality of input images of a plurality of samples disposed in a microplate disposed on the stage according to the input imaging configuration, and instructing the machine learning system to automatically generate an output image from each one of the plurality of input images according to the output imaging configuration.

17. 20. The method of operating a high-content imaging system of claim 16, further comprising the steps of: operating a robotic microplate loader to load an additional microplate onto the stage after the multiple input images have been generated; instructing the image generator to automatically generate multiple additional input images of the additional microplate; and instructing the machine learning system to automatically generate multiple additional output images from the multiple additional input images.

18. 12. The method of operating a high-content imaging system of claim 11, further comprising the steps of instructing the image generator to develop a series of input images of the sample taken at varying focus in accordance with the input imaging configuration and simultaneously providing the series of images to the machine learning system, and in response, automatically generating, by the machine learning system, a series of output images in accordance with the output imaging configuration.

19. The method of claim 11 further comprising transmitting the image generated by the image generator and the output image to a computer external to the high-content imaging system.

20. A method of operating a high-content imaging system as described in claim 11, wherein at least one of the second set of physical components is not present within the high-content imaging system.

Images