Automated system and method for experiment handling for incubation and image analysis

Abstract

Systems and methods for automated experiment handling are described. An automated experiment handling device may include an incubator, liquid handler, laminar flow device, imager, camera, and transport device for moving plates for scientific discovery. User input is received at user interface to design experiments and decision making processes to automate orchestration of devices based on the received user input. The user interface may enable various conditions to be user-defined based on metrics of plates and wells. Captured images of plates may be processed and analyzed to determine whether the conditions are satisfied. Image analysis results may be displayed on the user interface.

Description

AUTOMATED SYSTEM AND METHOD FOR EXPERIMENT HANDLING FORINCUBATION AND IMAGE ANALYSISTECHNICAL FIELD

[0001] The present disclosure generally relates to scientific discovery systems and, more particularly, to automated operation of experiment scheduling, planning, and media transport for incubation and image analysis.BACKGROUND

[0002] Scientific discovery' systems largely rely on manual input of experiment configurations, manual plate analysis, and manual scheduling of new experiments. Often, data is captured in batches for multiple periods, generating a large amount of image data for numerous experiments and scenarios. For example, an experimental drug being tested on a cancerous tumor may be applied in different dosages, at different stages of the tumor’s grow th, and so forth. The scientist conducting the experiment may be required to manually deploy the experimental drug via a liquid handler onto various plates with a number of wells of different media containing organoids that have similar cancerous cells. The plates or wells may be tracked individually.

[0003] Existing scientific discovery systems rely on direct user input in ensuring the correct amount of growth and cells are on individual plates, as well as managing the image data captured on the plates. For example, some scientific discovery systems may be configured w ith a series of user input screens enable a particular experiment to be designed. However, these user input screens usually apply to one plate at a time, and manual management of plates is cumbersome and time consuming.

[0004] Systems and methods for improving the management and creation of image data associated with scientific discovery may be advantageous. A reliable and efficient way of managing image data as w ell as experiment data for scientific discovery may' be needed.SUMMARY

[0005] Various aspects for automated experiment handling (including scheduling, planning, execution) and liquid transport for incubation and image analysis in scientific discovery' systems are described. Various types of experiments and actions with different components may be handled, including transporting liquids with or without cells, feeding medium (such as mTESRl), gels (such as matrigel), reagents (such as gentle cell dissociation reagent), medium plus cells (cell suspensions, spheroids), and gels plus cell clusters (broken matrigel domes with intestinal organoids).

[0006] An automated machine by which experiment designers and / or scientists are able to plan and run their experiments by interacting with a graphical representation of plates, enabling the automated triggering of actions based on completion of a condition with the graphical representation in intuitive ways is further described herein. A dynamic graphical user interface may generate on a user device a set of data configuration values. User input to configure one or more experiment protocols is received at the user interface such that modified data configuration values may be determined based on the received user input. The user interface may also be used to schedule the tasks and workflows that are triggered based on user-defined conditions applied on a plate-level or well-level basis. Additionally, cell growth may be continuously monitored, notifying users when cell growth goes wrong or exceeds parameters user-defined in the experiment protocol, for example. Consumables used in experiments are also monitored, notifying users when the system runs out of consumables.

[0007] The various embodiments advantageously apply the teachings of computer-based scientific discover}' systems to improve the functionality of such computer systems. The various embodiments include operations to overcome or at least reduce the issues previously encountered in scientific discover}’ systems and, accordingly, are more effective and / or cost- efficient than other scientific discovery systems. That is, the various embodiments disclosed herein include hardware and / or software with functionality to improve automated operation of experiment scheduling, planning, monitoring cell growth and / or consumables used, and / or media transport for incubation and / or image analysis in scientific discovery systems. Accordingly, the embodiments disclosed herein provide various improvements to scientific discover}' systems.

[0008] Some examples relate to a computer-implemented method, comprising receiving, at a server, image data associated with a well in a plate within an automated experiment handling device based on a schedule of tasks associated with an automated w orkflow . Further, the method comprises generating, at the server, input data associated with the automated workflow, the input data generated using an image processing sendee to process the image data, and receiving, at the server, user input to configure a new workflow. Additionally, the method comprises determining, at the server, one or more conditions of the new workflow based on the received user input and updating, at the server, the schedule of tasks in the automated workflow' with the new' workflow based on existing experiment protocol timing requirements. Furthermore, the method comprises causing, at the server, one or more actions to be executed in the automated experiment handling device according to theupdated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

[0009] The image data may be associated with a well in a plate is captured using an imager included in the automated experiment handling device.

[0010] A location of the plate within the automated experiment handling device may be determined based on the schedule of tasks in the automated workflow.

[0011] The method may further comprise generating, at the server, a data visualization that depicts processed image data, the data visualization including a history of recorded actions performed on the well in the plate, a history of cell passaging across one or more plates, a plurality of images associated with the well, and / or a plurality of image analysis results over time. Additionally, the method may comprises causing the data visualization to be displayed in a user interface at a user device.

[0012] The user input may comprise an expression having the one or more conditions associated with the one or more actions to be executed when the one or more conditions are satisfied.

[0013] The schedule of tasks in the automated workflow may include the existing experiment protocol timing requirements. The method may further comprise determining a set of new tasks associated wi th the new workflow, determining one or more new timing requirements associated with the set of new tasks, and generating the updated schedule of tasks based on the one or more new timing requirements associated with the set of new tasks and the existing experiment protocol timing requirements.

[0014] The one or more actions may be caused to be executed in the automated experiment handling device by sending a request through an application programming interface (API) gateway connected to an embedded PC within the automated experiment handling device.

[0015] The automated experiment handling device may comprise a plurality of platehandling devices, including a transport system to handle a plurality of plates associated with one or more experiment protocols, an incubator providing an environment for cell culture growth in the plurality of plates, an imager configured to capture periodic images of the plurality of plates, a liquid handling system providing periodic liquid maintenance on the plurality of plates, a laminar flow system providing sterile air for the plurality of plates outside of the incubator, and one or more component identifiers to uniquely identify the plurality of plates and one or more inventory supply containers. The method may further comprise sending one or more instructions in the request to the embedded PC within theautomated experiment handling device, the one or more instructions including the one or more actions to be executed in the automated experiment handling device, receiving a status update from the embedded PC, the status update including a progress indication of the one or more actions, responsive to receiving an indication of completion of the one or more actions, retrieving one or more new images from the embedded PC associated with the one or more actions, storing the one or more new images in a data store, and determining a next task in the updated schedule of tasks in the automated workflow.

[0016] Some examples relate to a system, comprising a processor, a non-volatile memory. and a device controller, operable by the processor and the non-volatile memory, the device controller configured to receive image data associated with a plate within an automated experiment handling device based on a schedule of tasks associated with an automated workflow. Further, the device controller is configured to generate input data associated with the automated workflow, the input data generated using an image processing sendee to process the image data. Additionally, the device controller is configured to receive user input to configure a new workflow, determine one or more conditions of the new workflow based on the received user input, update the schedule of tasks in the automated workflow with the new workflow based on existing experiment protocol timing requirements, and cause one or more actions to be executed in the automated experiment handling device according to the updated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

[0017] The image data may be associated with a plate is captured using an imager included in the automated experiment handling device.

[0018] A location of the plate within the automated experiment handling device may be determined based on the schedule of tasks in the automated workflow.

[0019] The user input to configure the new workflow may be received through a user interface provided by a user interface system communicatively coupled to the device controller, the user interface provided for display.

[0020] The user input may comprise an expression having the one or more conditions associated with the one or more actions to be executed when the one or more conditions are satisfied.

[0021] The schedule of tasks in the automated workflow may include the existing experiment protocol timing requirements, and wherein the device controller is further configured to determine a set of new tasks associated with the new workflow, determine one or more new timing requirements associated with the set of new tasks, and generate theupdated schedule of tasks based on the one or more new timing requirements associated with the set of new tasks and the existing experiment protocol timing requirements.

[0022] The one or more actions may be caused to be executed in the automated experiment handling device by sending a request through an application programming interface (API) gateway connected to an embedded PC within the automated experiment handling device. The automated experiment handling device may comprise a plurality of plate-handling devices, including a transport system to handle a plurality of plates associated with one or more experiment protocols, an incubator providing an environment for cell culture growth in the plurality of plates, an imager configured to capture periodic images of the plurality of plates, a liquid handling system providing periodic liquid maintenance on the plurality of plates, a laminar flow system providing sterile air for the plurality of plates outside of the incubator, and / or one or more component identifiers to uniquely identify the plurality of plates and one or more inventory supply containers. The device controller may be further configured to send one or more instructions in the request to the embedded PC within the automated experiment handling device, the one or more instructions including the one or more actions to be executed in the automated experiment handling device, receiving a status update from the embedded PC, the status update including a progress indication of the one or more actions, responsive to receiving an indication of completion of the one or more actions, retrieving one or more new images from the embedded PC associated with the one or more actions, storing the one or more new images in a data store, and determining a next task in the updated schedule of tasks in the automated workflow.

[0023] Some examples relate to a computer-implemented method, comprising providing, by a server, a user interface for display on a user device communicatively coupled to the server, the user interface including a set of data values corresponding to physical attributes of components used in an automated experiment handling device. The method further comprises receiving, at the server from the user device, user input to define an experiment protocol including one or more configuration instructions associated with the components used in the automated experiment handling device and an automated workflow including one or more conditions associated with each action in the automated workflow, the user input received through the user interface. Additionally, the method comprises scheduling a first action in the automated workflow through the user interface, the first action having one or more action parameters and evaluating the one or more conditions associated with the first action in the automated workflow. Further, the method comprises based on the one or more conditions associated with the first action evaluated as being satisfied, causing the first action to beperformed at the automated experiment handling device according to the one or more action parameters, and sending a notification to the user device indicating that the first action has been triggered based on the one or more conditions being satisfied.

[0024] The set of data values corresponding to physical attributes of components used in the automated experiment handling device may comprise at least one of a quantity of inventory supplies required by the experiment protocol, a quantity’ of inventory supplies available in the automated experiment handling device, a first identifier associated with a plate having a standard number of deep wells, a second identifier associated with a plate having a standard number of shallow w ells, a third identifier associated with a plate having a small number of shallow w ells, a fourth identifier associated with a plate having a medium number of shallow wells, a fifth identifier associated with a plate having a large number of shallow wells, a plurality of unique well identifiers, each unique well identifier associated with each well in the automated experiment handling device, a plurality of unique plate identifiers, each unique plate identifier associated with each plate in the automated experiment handling device, a media supply identifier associated with a media supply used in cell culture cultivation, a plurality of liquid handling identifiers, each liquid handling identifier associated with a liquid fed in the automated experiment handling device, a plurality’ of environmental parameters, each environmental parameter describing a unique environmental attribute within the automated experiment handling device, and / or a plurality of timing parameters, each timing parameter indicating a period of time expected to complete an associated action in the automated experiment handling device.

[0025] The user input may comprise an expression of the one or more conditions, the expression using one or more Boolean operators to describe the one or more conditions.

[0026] Evaluating the one or more conditions may comprise based on the one or more conditions, retrieving one or more image files, each image file associated with a well in a plate in the automated experiment handling device, generating one or more image analysis results using an image processing service, and evaluating the one or more conditions based on the one or more image analysis results.

[0027] The method may further comprise receiving an indication that a first resource of inventory supplies needed to complete a current task has been depleted, determining a second resource of inventory' supplies that is available in the automated experiment handling device, modifying the current task to use the second resource of inventory' supplies, and updating one or more scheduled tasks based on a calculated data value corresponding to a quantity of inventory supplies based on the first resource being depleted.

[0028] It should be understood that language used in the present disclosure has been principally selected for readability and instructional purposes, and not to limit the scope of the subject matter disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure la schematically illustrates a conceptual diagram of a computer-based scientific discovery system, illustrating conceptualized functional components of the scientific discovery system.

[0030] Figure lb schematically illustrates the computer-based scientific discovery system of Figure lb, illustrating networked system components including a user device, an automated experiment handling device, a core services handler, a data store, and an image data store connected through a network.

[0031] Figure 2 schematically illustrates a computer-based scientific discovery system that may be implemented by the computer-based scientific discovery system of Figures la- 1b.

[0032] Figure 3 schematically illustrates various functional components of the computer- based scientific discovery system of Figure 2.

[0033] Figure 4 schematically illustrates elements of the computer-based scientific discovery' system of Figure 2.

[0034] Figure 5 schematically illustrates a data flow interaction of elements of the computer-based scientific discovery system of Figure 2.

[0035] Figure 6 schematically illustrates a data flow interaction of image capture and storage elements of the computer-based scientific discovery' system of Figure 2.

[0036] Figure 7 schematically illustrates a data flow interaction of experiment data elements of the computer-based scientific discovery system of Figure 2.

[0037] Figure 8 is a flowchart of an example method of automating decision making for experiment-based scientific discovery.

[0038] Figure 9 is a flowchart of an example method of generating image analysis for experiment-based scientific discovery.

[0039] Figure 10 is a flowchart of an example method of performing actions based on conditions for experiment-based scientific discovery.DETAILED DESCRIPTION

[0040] Figure la schematically illustrates a conceptual diagram of a computer-based scientific discovery’ system, illustrating conceptualized functional components of the scientific discovery' system. An automated scientific discovery instrument may rely on manydifferent components to control the environment for the sample, image the samples periodically to monitor their growth and decide whether to trigger other actions, such as liquid handling and user notification. The automated scientific discovery instrument may then be able to feed and add liquid (e.g., reagents and gels) to the samples for treatments and execute protocols for the growing and harvesting of samples. As the samples are grown on plates, the instrument is also able to present the plates to an exit door for other workflows outside of the instrument, such as a centrifuge or other imaging system.

[0041] '‘Media” refers to media that generally comprise an appropriate source of energy and compounds which regulate the cell cycle, such as liquid for cell culturing. Media can grow cells that get too dense, as measured by confluency, the proportion of or percentage area covered by adherent cells in a cell culture dish. Importantly, cell confluence is measured to determine timings for splitting or '‘passaging” as well as harvesting cells and for drug treatments or differentiation experiments. Other measurements, in addition to cell confluence, may be made and defined by users in an experiment protocol. For example, an area may be measured by analyzing images and pixel count through neural networks to determine a user- specified “segment” which may refer to a cell, organoid, portion of an organoid, and the hke. Other dimensions of measurement include, but are not limited to, count (e g., cell count), confluence, diameter, elongation, form factor (e.g., how circular the cell culture appears), and intensity (e.g., the image intensity or brightness of the cell culture). Accurate and reproducible measurements are key to generating high-quality, reliable data. Additionally, automation of these measurements enables more imaging than manual spot checks, which also increases reliability. It becomes especially important for standardizing cell culture protocols in developing and manufacturing cell therapies, in research, and drug screening.

[0042] Standard tissue culture plates include six, twelve, twenty-four, forty-eight, ninety- six, and three hundred eighty -four well plates. Various types of plates may be used, including flat, U-bottom, and V-bottom plates. Spheroids, for example, are typically cultivated in U-bottom and V-bottom plates. Tissue culture plates may also include chips, such as plates that include three hundred eighty-four wells that include ninety-six independent tissue culture chips with two adjacent channels per chip, forty independent tissue culture chips with three adjacent channels per chip, and sixty-four tissue culture chips with three adjacent channels per chip. Other plates may be used with differing numbers of wells. Deep well plates, reservoir plates, or container plates may be used for 3D cell cultures, such as spheroids and organoids. Spheroids are spherical cellular units that are generally cultured as free-floating aggregates and are of low complexity in mirroring tumor organization.Organoids are an artificially grown mass of cells or tissue that resembles a portion of an organ, in both structure and function. Organoids are also cultivated in plates but use deep well plates as interim plates for seeding and passaging. Various measurements, such as confluency, may be determined for two-dimensional (2D) cell cultures. Three-dimensional (3D) cell cultures may be measured using other parameters, including diameter, texture, optical density, and volume, among others. Organoids provide a better representation of what actually happens in the human body. Thus, it is important to deploy plate management systems that ensure samples remain viable. Timing and scheduling are important to ensure that the confluence of a cell culture is not so dense that cells begin to die due to lack of space on the plate to grow, for example. As another example, 3D cell cultures are monitored to notify users when cell growth is discovered to be outside of expected parameters.

[0043] Various devices are used to maintain an environment where cells are in the optimal environment to grow and reproduce. An incubator regulates humidity, temperature, gas concentration levels (e.g., CO2 levels), and / or other environmental conditions. A laminar flow device, such as a hood or a cabinet, produces a stream or laminar flow of air designed to provide a sterile environment. Laminar flow is typically used when putting a plate into or taking a plate out of a liquid handler, where the laminar flow system exists on top of the liquid handler. A liquid handler is used to introduce liquid to a well for various purposes, including transporting of cells (e.g., seeding, harvesting, and passaging), dome placing, dome destroying, and to replenish the media supply in the well to ensure continued growth of cell cultures. One or more software platforms may be used to interface with an automated liquid handler platform to provide interactive instrument control. Alternatively, in other embodiments, one server may connect to multiple computing devices (e.g. embedded PCs) and packages or platforms may interact with components of the system described herein.Autonomous operation, user notification for user interventions, and control of these devices is needed to orchestrate and schedule workflows of multiple experiments w ith overlapping time periods. User interventions may be needed for critical steps and error handling to ensure experiments remain viable.

[0044] For example, as illustrated in Figure IB. a scientific discovery system 100 may include, as conceptually illustrated in Figure 1 A, a user interface 102 that provides an interactive display of data retrieved from image analysis 110 and autonomous operation 104. Various systems may interact with the autonomous operation 104 conceptualized component, such as an imager 106. planning and scheduling 108, a transport system 112, barcode readers 114, an incubator 116, laminar flow 118, liquid handling 120, and image analysis 110. Theimager 106, such as a camera or other image capturing device, may communicate with the image analysis 110 conceptualized component to process the images taken of plates. In an embodiment, the image analysis 1 10 conceptualized component may be performed by multiple graphic cards on a single computing device, or multiple computers for increased processing capacity' for image analysis. In the example scientific discovery system 100, individual plates include a liquid medium in which cells may grow and / or replicate in the incubator 116. A planning, scheduling, and monitoring 108 conceptualized component may be used to create and update a plan for a set of experiments, receive user-inputted conditions that are monitored that trigger one or more actions, and other user-defined configurations that affect the design of the set of experiments. A transport system 112 moves individual plates within an automated machine and / or system of devices, including external devices such as an external centrifuge, confocal imager, and the like. A laminar flow 118 conceptualized component provides a sterile working environment for samples and work processes such as plate preparation. The transport system 112 conceptualized component enables plates and / or content of plates, such as cells on a plate, to be moved to another plate or system connected within the automated system. The liquid handling 120 conceptualized component encompasses the movement of liquids through a range of robotic platforms to hand-held single channel pipettes. In an embodiment, the liquid handler may perform passaging of cells (i.e., breaking up the cells) or feeding cells. Barcode readers 114 may be used to scan and identify plates and inventory of supplies in the automated system. While barcode readers 114 are described here, other types of identification mechanisms, such as radio-frequency identifiers (RFID), may be used instead of barcodes. In some instances, barcodes are not needed. Instead, wells and plates may be tracked by identifiers, such as well identifiers and plate identifiers that do not use barcodes, using labels affixed to the wells and plates. In such instances, the system tracks the location of the plate, where home position is located in the incubator, and where the plate identifier may include compound information, including any key value path. Thus, the status of plate may be determined from plate identifier (e.g., when it moves from incubator to liquid handler). RFID may be used on liquid supplies, or barcodes may be used on liquid supplies. Autonomous operation and user notification enable orchestrating the different components to allow for long-running experiments involving decision making and liquid handling workflows.

[0045] An example data flow of the scientific discovery system 100 as shown in Figure la in the conceptualized components may begin with the planning and scheduling of an experiment through a user interface 102. A display screen on the exterior of the automatedsystem may present the user interface 102. Alternatively, a user device may connect to the automated system such that a user interface 102 is displayed on the user device. Through the user interface 102 and autonomous operation 104, data configuration values are received for planning, scheduling, and monitoring 108 purposes, such as number of plates in the experiment, the amount of liquid used, the volume, or other analysis parameters, of cells detected that would trigger passaging of cells, flow rate, pipetting depth, positioning in the well, liquid following, tip volume management to avoid bubbles, and so forth. The planning, scheduling, and monitoring 108 conceptualized component executes the configured experiment and creates instructions for the other systems to start conducting the experiment according to the received data configuration values. For example, the planning, scheduling, and monitoring 108 conceptualized component may instruct 8 plates of 100 milliliters of media to be injected with 50 milliliters of the same cell culture. The transport system 1 12, barcode readers 114, incubator 116, laminar flow 118, and liquid handling 120 conceptualized components may operate to fulfill the instructions from the planning, scheduling, and monitoring 108 conceptualized component. The data configuration values may require image capture by the imager 106 every hour. Thus, the autonomous operation 104 may direct the imager 106 to capture images of each plate and may direct the transport system 112, barcode readers 114, incubator 116, laminar flow 118, and liquid handling 120 to move the plates in a specified order to capture the images. In some embodiments, different plates may be used for different experiments, such as using deep well plates for cell collection of 3D organoids as an intermediate plate when passaging intestinal organoids between two 24 well plates. Additionally, other experiments may use 24 well plates for treatment wells at differing dosages. The imager 106 may image specific plates of experiments to automatically capture data reliably. The images may be analyzed by the image analysis 110 conceptualized component and presented to the user through the user interface 102. Various automated tasks may be performed through the autonomous operation 104, including user notification, based on the results of the image analysis 110 and other conditions evaluated to be true.

[0046] Figure IB schematically illustrates a computer-based scientific discovery system 100 that performs the conceptual components illustrated in Figure 1 A, Figure IB illustrating networked system components including a user device 170, an automated experiment handling device 130, a core services handler 140, a data store 190, and an image data store 188, the system components connected through a network 128. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and soas not to obscure pertinent aspects of the example embodiments disclosed herein. In some embodiments, user device 170, automated experiment handling device 130, and core services handler 140 are computer-based components that may be interconnected by a network 128. Additional components of an example scientific discovery system 100, such as data store 190 and image data store 188, may also be connected to network 128.

[0047] In some embodiments, one or more networks 128 may be used to communicatively interconnect various components of scientific discovery system 100. For example, each component, such as user device 170, automated experiment handling device 130, core sendees handler 140, data store 190 and image data store 188, may include one or more network interfaces and corresponding network protocols for communication over network 128. Network 128 may include a wired and / or wireless network (e.g.. public and / or private computer networks in any number and / or configuration) which may be coupled in a suitable way for transferring data. For example, network 128 may include any means of a conventional data communication network such as a local area network (LAN), a wide area network (WAN), a telephone network, such as the public switched telephone network (PSTN), an intranet, the internet, or any other suitable communication network or combination of communication networks. In some embodiments, network 128 may comprise a plurality of distinct networks, subnetworks, and / or virtual private networks (VPN) may be used to limit communications among specific components. For example, user device 170 may be on a limited access network such that control data may only be transmitted between a user device 170 and automated experiment handling device 130, enabling the automated experiment handling device 130 to securely receive user input through the user device 170 and enable real-time data interaction and experiment configuration through a user interface on the user device 170.

[0028] User device 170 may be any suitable computer device, such as a computer, a computer server, a laptop computer, a tablet device, a netbook, an internet kiosk, a personal digital assistant, a mobile phone, a smart phone, a gaming device, or any other computing device. User device 170 is sometimes called a host, client, or client system. In some embodiments, user device 170 may host or instantiate one or more applications for interfacing with scientific discovery system 100. For example, user device 170 may be a personal computer or mobile device running a experiment management application configured to provide a user interface for automated experiment handling device 130. In some embodiments, user device 170 may be configured to access data accessible by the automated experiment handling device 130 directly through network 128. In someembodiments, one or more functions of automated experiment handling device 130 may be instantiated in user device 170 and / or one or more functions of user device 170 may be instantiated in automated experiment handling device 130.

[0029] User device 170 may include one or more processors 172 for executing compute operations or instructions stored in memory 174 for accessing experiment data and other functions of automated experiment handling device 130 through network 128. In some embodiments, processor 172 may be associated with memory 174 and input / output device 176 for executing both data display operations and scientific discovery system management operations. Processor 172 may include any type of processor or microprocessor that interprets and executes instructions or operations. Memory 174 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 172 and / or a read only memory (ROM) or another ty pe of static storage device that stores static information and instructions for use by processor 172 and / or any suitable storage element. In some embodiments, user device 170 may allocate a portion of memory 174 and / or another local storage device (in or attached to user device 170) for storing various data for user device 170, such as user-inputted data for planning, scheduling and monitoring of experiments as well as user-downloaded experiment data (e.g., reports, acquired images, etc.) for further processing. For example, a user may export data and store data locally or on external storage. In some embodiments, user device 170 may include one or more input / output (I / O) devices 176. For example, a graphical display, such as a monitor and / or touch screen display, and / or other user interface components such as a keyboard, a mouse, function buttons, speakers, vibration motor, a track-pad, a pen, voice recognition, biometric mechanisms, and / or any number of supplemental devices to add functionality to user device 170. Network interface 178 may include one or more wired or wireless network connections to network 128. Network interface 178 may include a physical interface, such as an ethemet port, and / or related hardware and software protocols for communication over network 128, such as a network interface card, wireless network adapter, and / or cellular data interface.

[0030] User device 170 may include a plurality of modules or subsystems that are stored and / or instantiated in memory 174 for execution by processor 172 as instructions or operations. For example, memory 174 may include a data manager 180 configured to provide a user interface for selectively creating, manipulating, and displaying real-time, near real-time, and / or stored structured data for scientific discovery in the automated experiment handling device 130. Memory 174 may include user permission management 182 configuredto manage user permissions to edit experiments and / or access data responsive to user input received at a user interface 102. Memory 174 may include a user interface manager 184 configured to provide a user interface 102 for generating, modifying, and shaping data received through user device 170. Memory 174 may include other modules, not illustrated, configured to perform functionality of the user interface 102, including rendering data values as graphical user interface elements.

[0031] Automated experiment handling device 130 may include a housing and a bus interconnecting at least one computing device (e.g. embedded PC 134), at least one user interface system 132, at least one imager 106, at least one camera 136, a transport system 112, an incubator 116, one or more barcode readers 114, a liquid handling system 120, a laminar flow device 118, and at least one device service controller 138. In an embodiment, an imager 106 may include an integrated camera 136, movable stage for a plate, objectives, filter cubes, light sources such as LED, mechanisms to adjust the objectives, filter cubes, and light sources, and gripper to move the plate onto the stage. The housing (not shown) may include an enclosure for mounting the various subcomponents of automated experiment handling device 130. locating any physical connectors for the interfaces, and protecting the subcomponents. Some housings may be configured for mounting within a rack system. The bus (not shown) may include one or more conductors that permit communication among the components of automated experiment handling device 130. A user interface system 132 mayenable a user to directly interact with the automated experiment handling device 130 through a touchscreen or other user interface hardware device. The user interface system 132 may also enable a user to interact with the automated experiment handling device 130 through a web application available through the network 128, such as through a w eb API or web-based communications protocol. Embedded PC 134 may include any ty pe of processor or microprocessor that interprets and executes instructions or operations and memory that may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by a processor within the embedded PC 134 and / or a read only memory- (ROM) or another type of static storage device that stores static information and instructions for use by the processor and / or any suitable storage element.

[0032] In some embodiments, automated experiment handling device 130 may include an application programming interface within the embedded PC 134 configured to transfer data between the automated experiment handling device 130, data store 190, user device 170, core services handler 140, and / or image data store 188 through network 128. For example, the application programming interface may include functionality for data to be easily transferredbetween components of the scientific discovery system 100. In some embodiments, automated experiment handling device 130 may include an embedded PC 134 having multiple application programming interfaces for communication with different types of applications on user devices 170, image data stores 188, data stores 190, and / or core services handler 140 over network 128. Because the embedded PC 134 is included in the automated experiment handling device 130, communications with hardware devices such as the imager 106, the camera 136. the liquid handler 120. or the incubator 116 are considered internal as well. In one embodiment, the embedded PC 134 uses an Ethernet connection with the core sendees handler 140 such that the communications are handled through a private interface not intended to be accessed by the user.

[0033] A device service controller 138 may manage a device service that comprises a hardware abstraction layer to implement hardware access, a management component, and a component for workflow execution that implements the required logic to execute workflows such as feeding or trigger imaging at specific points in time. In this way, the device components within the automated experiment handling device 130, such as the camera 136, the imager 106, transport system 112, barcode readers 114. incubator 1 16, laminar flow 118. and liquid handling 120 are controlled by the device service controller 138. In an embodiment, the device service controller 138 is integrated into the embedded PC 134.

[0034] For example, a transport system 112 may include electronic components, technology and machinery for environment control and transport to manage plates of different sizes, such as deep well plates, 6 well plates, 12 well plates, 24 well plates, 96 well plates, 384 well plates, and so forth. A liquid handling system 120 may include an inventory7of supplies needed to carry out experiments, such as tips, media, liquid, drug treatments, and so forth, as another example. The device service controller 138 includes a device status monitoring w atchdog to ensure that a device is operational and will restart the device if the device is not operational. In this way, the automated experiment handling device 130 may reduce the amount of manual intervention needed to perform experiments. This is beneficial because experiments may be completed without the need for manual intervention due to a failure of a device. In other embodiments, different configurations of devices may be used, such as a freezer and a plate hotel attached to the automated experiment handling device 130. For example, the system could be expanded with a hotel for spare plates or tips to reduce manual handling. Various configurations of the automated experiment handling device 130 may be implemented to meet user needs. A decision making process may determine, on its own, when to start with the assay in this configuration, thus becoming a completelyautonomous screening device. In other embodiments, the device service controller 138 includes support for external devices, such as a clone picker, flow cytometer, high resolution imaging, and so forth. Additionally, other devices, such as imaging devices, may be upgraded to dark field, phase contrast, and differential interference contrast DIC imaging. As another example, other detection devices may be used to read consumable ty pe automatically (e.g., RFID, QR code, camera + detection system, etc ). Media supply may also be modified, in some embodiments, such that a user may check the pH of medium by color (phenol red) using a sensor or color detection system, for example.

[0035] Core services handler 140 may be instantiated on a computing device that includes a housing and a bus interconnecting at least one processor 142, and at least one application programming interface gateway 146 configured to transfer data between the automated experiment handling device 130, data store 190, user device 170, core services handler 140, and / or image data store 188 through network 128. The housing (not shown) may include an enclosure for mounting the various subcomponents of core services handler 140, locating any physical connectors for the interfaces, and protecting the subcomponents. Some housings may be configured for mounting within a rack system. The bus (not shown) may include one or more conductors that permit communication among the components of the core sendees handler 140. For example, the application programming interface may include functionality for data to be easily transferred between components of the scientific discovery system 100. In some embodiments, core services handler 140 may include an API gateway 146 having multiple application programming interfaces for communication with different ty pes of applications on user devices 170, image data stores 188, data stores 190, and / or device components within the automated experiment handling device 130 over network 128. In an embodiment, the core services handler 140 may be instantiated on a virtual machine on a server having the components described above.

[0036] Network interface 148 may include one or more wired or wireless network connections to network 128. Network interface 148 may include a physical interface, such as an ethemet port, and related hardware and software protocols for communication over network 128. such as a network interface card.

[0037] Storage devices 150 may include one or more non-volatile memory devices configured to store data, such as a hard disk drive (HDD), solid state drive (SSD), flash memory-based removable storage (e.g., secure data (SD) card), embedded memory chips, etc. In some embodiments, storage device 150 is, or includes, a plurality of solid-state drives.

[0038] In some embodiments, a respective data storage device 150 may include a single medium device, while in other embodiments the respective data storage device 150 includes a plurality of media devices. In some embodiments, media devices include NAND-type flash memory orNOR-type flash memory . In some embodiments, storage device 150 may include one or more hard disk drives. In some embodiments, storage devices 150 may include a flash memory device, which in turn includes one or more flash memory’ die, one or more flash memory packages, one or more flash memory channels or the like. However, in some embodiments, one or more of the data storage devices 150 may have other types of nonvolatile data storage media (e.g., phase-change random access memory’ (PCRAM), resistive random access memory' (ReRAM), spin-transfer torque random access memory' (STT-RAM), magneto-resistive random access memory (MRAM), etc.).

[0039] Core services handler 140 may include a plurality of modules or subsystems that are stored and / or instantiated in memory’ 144 for execution by processor 142 as instructions or operations. For example, memory' 144 may include a control subsystem 152 configured to control execution of instructions sent to the automated experiment handling device 130, including planning, scheduling, and monitoring as described above. Memory' 144 may include a data monitoring subsystem 154 configured to monitor data generated from the automated experiment handling device 130. For example, the status of supplies, a consumable supply, may be monitored by the data monitoring subsystem 154. The control subsystem 152 may generate a schedule of workflows based on a real-time status of supplies needed to carry out an experiment. Memory 144 may include a data storage subsystem 156 configured to store received data in storage device(s) 150 and / or data store 190 over the network 128. Memory 144 may include a data analysis subsy stem 158 configured to analyze data for defined patterns, such as growth of cells, detected effects of treatments as delivered by liquid handling, and combinations thereof. Memory 144 may include a data display subsystem 160 configured to selectively' display' data on user device 170, directly attached to the automated experiment handling device 130 or through a user interface displayed on a web application connected via network 128. In some embodiments, the functions of user authentication may be integrated in core services handler 140 and instantiated in memory 144 as a user authentication subsystem and / or a subset of functions of display' subsystem 160.

[0040] In some embodiments, scientific discovery’ system 100 may include one or more remote and / or cloud-based resources for supporting the functions of automated experiment handling device 130 and / or user device 170. For example, scientific discovery system 100 may include a data store 190 configured to host some, all, or select portions of the storageY1functions of automated experiment handling device 130, such as a cloud-based network attached storage system, distributed storage system, or on-premise data storage system. In some embodiments, the majority of functions described above for automated experiment handling device 130 may reside in automated experiment handling device 130 and select functions may be configured to leverage additional resources in a network server (not pictured) and / or data store 190. For example, a network server may be configured to support specialized and / or processing intensive imaging algorithms to supplement data analysis subsystem 156, and / or data store 190 or image data store 188 may be configured to support archiving of image data for longer term storage. As another example, experiment data may be uploaded into an analytics-centric and cloud-native software service so that experiment evaluation may be performed asynchronously over several devices and lab data (such as a consumable batch) may be linked, programmatically, with the experiment results. In this way, through the data analytics software service, experiment data may be captured, analyzed, reported, and shared in a more streamlined manner, beneficially contributing to the user experience and increasing efficiency of experiment evaluation.

[0041] Figure 2 schematically illustrates a computer-based scientific discovery system 200 that may be implemented by the computer-based scientific discover}7system 100 of Figures la-lb. For example, a core services handler 140 may include an automated experiment handler 246 that further includes a planning and scheduling manager 202, a decision making module 204, an image file management module 206, a user management module 208, an experiment management module 210 and an image analysis control module 212.

[0042] The planning and scheduling manager 202 determines a scheduling plan to address a scheduling problem based on the real-time status of experiments being performed in the automated experiment handling device 130. The planning and scheduling manager 202 may provide a user interface to schedule and plan the experiments to be performed. The experiment management module 210 may also operate in conjunction with the planning and scheduling manager 202 to receive user input for the experiment protocols and actions triggered by conditions. A protocol assembles a sequence of templates, or phases, of a process. An experiment protocol defines the steps taken to conduct the experiment, including context and configuration that describes what machines can do what process and how. For example, an example experiment protocol includes a media change, harvesting (from wells to container / deep well plates), seeding (from container / deep well plates to well plates).passaging cells between plates, feeding, reagent addition, incubate (hold plate in the incubator), imaging a plate, running an end point assay, and so forth.

[0043] An example user interface of an experiment management module 210 may include a user interface that enables a user to create an experiment, select protocol to use, and assign plates to the experiment. For example, a user may select a start date and time for the experiment. In another embodiment, a suggested start date may be provided to a user depending on other running experiments to reduce overlap. The user may optionally autoselect the suggested start date, for example. Additionally, the user interface may also include a heat map in which tiles represent the wells of a plate used in an experiment, the amount of supplies needed to complete the phase of the experiment based on protocols inputted by the user, and the existing real-time inventory of the supplies. For example, the experiment protocol may involve analysis of all wells used in the experiment, which may include several plates that have deep wells for 3D organoids. An experiment protocol may provide the user an option to randomly spot check wells in the experiment if there are large numbers of wells, in an embodiment. The example experiment management interface may include a heat map interface for each plate used and the wells being used represented by a tile as an example visualization. Other visualizations may be provided by the experiment management interface, including histograms, stacked lines, line plots, scatter plots, filtering of data, and other data visualization techniques. For example, a cell journey visualization may depict the past images and data of a user-selection of cells, including when they were passaged and thus travelling through different plates. This visualization may be presented as a timeline, as a carousel of images, as an interactive experience when displaying an image, as well as a combination of other data visualizations as described above. Other visualizations may include filtering by well or plate annotations, such as excluding control groups or to show images which are labeled by decision making to surface and discover interesting images out of a large dataset of images, such as 10000 images or greater. Another visualization may involve a plot matrix or an image matrix that displays plots or images depending on a certain parameter, such as drug concentration. For example, a 10x10 image matrix may be displayed, with each image associated with a plot of time versus heart-beating frequency, and a 10x10 plot matrix associated with the image matrix, where the plot matrix displays all the plots in x-axes sorted by drug concentration, on y axes sorted by drug type. Other data visualization techniques may be used by the experiment management module 210, including thumbnails, box plots, violin plots, videos showing kinetics or growth, other types of mapsthat show plates / wells over time and how they are connected, and a timeline visualization that shows what happened to the wells and when.

[0044] Various protocols may be implemented by administrators of an experiment management module 210, including protocols for system resilience and flexibility. However, as an example of robustness, a default protocol may include handling a situation where a repository of tips is only half full where the system has designated the repository to be completely full. In this case, the system may be programmed to check the next available repository for tips when it comes across the empty repository of tips. In an embodiment, this technique of continuing to a next container when a current container is empty is applied to liquids and spare plates. For example, if there are three containers of a liquid such as mTESRI and one container runs empty, the system is programmed to continue with the next container of mTESRI. Other examples of experiment management include error handling that includes user notification that may require user intervention, such as a plate stuck or lid lost. Another example of experiment management includes measuring a height of liquid during every feeding while running and updating the expected volume based on the geometry of the plate. Some reasons for a volume mismatch may include user error in data entry and evaporation. In an embodiment, the experiment management module 210 may enable a user to manually exclude wells through the user interface. This enables the system to handle empty positions when performing functions such as feeding, imaging, and analysis. In another embodiment, one or more decision-making processes, as configured by a user through the experiment management module 210 and / or decision making module 204, may make a determination to automatically exclude wells based on a set of conditions or criteria being satisfied. In this way, the protocols may adapt to a particular worktable as well as adapt a protocol to other types of plates having different numbers of wells.

[0045] As another example, scheduling, whether static or dynamic, may be required to process eight (8) plates, three of which are used in the experiment protocol for 3D organoids, and the remaining 5 plates used for other experiments with overlapping timelines. Because the transport system device includes electronic components, technology, and machinery to handle different types of plates and a unique identifier (plate ID) attached to each plate is used to identify plates, the example experiment management interface may enable the user to schedule the 8 plates efficiently based on the expected times for actions to be performed, such as imaging or passaging. In another embodiment, a barcode scanner may be implemented to scan barcodes that identify the plate ID for each plate. In other embodiments, a design of experiment (DOE) may be suggested by the system within the experiment managementinterface to reduce the effect of noise while reducing the number of necessary wells. These suggestions may be based on past experiments, in an embodiment. The suggested DOEs may include positioning of plates in the incubator (random and switch positions), time of treatment, and so forth.

[0046] In another embodiment, optimization strategies may be suggested by the scientific discover}’ system 200, such as positioning of tips which are used more often, starting times for experiments to avoid overlap with other experiments, and other suggested design of experiment parameters based on the current state of the system 200 and historical data on past experiments.

[0047] A decision making module 204 provides a user interface to receive conditions that may trigger various actions within the automated experiment handling device 130. For example, a user may enter expressions that include Boolean logic operators, metrics determined by image processing and / or other devices such as confluence, actual area available on a plate or well, and number of cells on a plate or well. By evaluating the conditions in an expression, one or more actions may be performed. For example, images of cells matching a user-supplied expression of one or more conditions may be retrieved based on the evaluation of the conditions and the resulting images may be displayed on a user interface, such as image control and image acquisition UI 242.

[0048] As another example, various image analysis may be performed on images, such as classification of an image as growing a cancerous tumor. Such a condition may be inputted as a condition by a user, where the action may be annotating wells that exhibit samples whose images match the classification of an image growing a cancerous tumor. This enables a scientist to quickly identify which wells may be responding to a drug treatment and which wells are not responding to a drug treatment for cancer, as an example. This also enables scientific discover}' to occur in personalized medicine. For example, a certain drug combination may be effective in treating a certain ailment for a particular person based on patient-derived organoids. The decision making module 204 thus enables the user to generate any number of decision making algorithms to be applied to wells, plates, or all wells in an experiment. Additionally, a decision making algorithm may be designed to exclude wells that did not seed, or where cell cultures in wells have died. This beneficially reduces the amount of wasted resources that might have been spent taking images of wells of dead cells throughout the duration of an experiment. Additionally, because decision making algorithms may mark, or otherwise annotate a well or plate based on a condition being true, other decision-making algorithms may be generated based on the annotations, in an embodiment.In various embodiments, annotations may be used for filtering the large amounts of data generated during image analysis.

[0049] The core services handler 140 may retrieve and store data at a data store 190. For example, planning and scheduling data generated through a user interface system 132 and through the planning and scheduling manager 202 may be stored at the data store 190. Similarly, decision logic generated through a user interface and subsequently used in the decision making module 204 may be stored at the data store 190. A list of authenticated users may be stored in the data store 190 and used by the user management module 208. In an embodiment, user I group of users may be assigned specific rights and / or permissions that limit access to worktable or certain experiments.

[0050] In an embodiment, image analysis may be controlled through the image analysis control module 212 in coordination with the image control and acquisition service 216 of the embedded PC 134. Images may be temporarily stored in local data storage 222 in an embedded PC 134. Upon satisfaction of a condition, such as a liquid handler refreshing the media in the well, one or more images may be captured and stored by the image control and acquisition service 216. Later, another workflow item may request a file server 214 copy the newly acquired images to an image file storage 220. In some embodiments, an image stitching service 218 may be used to append images into a larger image file. For example, a cell may be so large that it spans across multiple images because some image analysis services may be limited to a particular image resolution, such as 2 megapixels. Some analysis tasks would require a larger image such that stitching is required to analyze the properties of the cell. By using an image stitching service 218, multiple images may be combined into a large image. In other embodiments, tiling may be used to set images side by side where stitching in comparison overlaps the images and combines them. Another technique that may be used to handle large areas is to use an objective with a lower numerical aperture (NA), such as a 2x objective. This would provide a wider field of view.Additionally, an image may be divided into smaller images to increase throughput. For example, an image of a 384 well plate may be acquired but then divided to analyze 4 wells of the 384 well plate.

[0051] The image analysis control module 212 manages the images associated with each well for each experiment. In some embodiments, an image analysis service 232 may be used to determine metrics about the cell culture in the well, such as confluence, area, and number of cells, based on image processing. Various image classification techniques may be used in the analysis of images by the image analyzer 234, such as segmentation, analysis, andclassification. For example, neural networks may be used for segmentation to identify which portions of the image represent a cell culture and which portions represent empty surface area on the plate. As another example, classical image classification techniques as well as AI- based classification techniques may be used in the analysis of the images by the image analyzer 234. The resulting images are stored in an image file storage 220, such as a networked data store or a cloud-based data store. In an embodiment, curve fitting may be performed in the data analysis of images, such as EC50 / IC50 curve fitting. Because many dose-response curves follow a familiar sigmoidal shape, curve are often defined by four parameters: top, bottom, hill slope and the EC50 (or IC50). The top and bottom parameters describe the values at which the curve reach a plateau - coming infinitely close, but never quite reaching these values. The Hill slope describes the slope of sigmoidal curve between these two plateaus. The EC50 (or IC50) refer to concentration of agonist (or antagonist) required to increase (or reduce) the measured response to half - or 50% - of its maximal value. EC50 / IC50 curve fitting may be implemented to provide additional analysis based on the image processing.

[0052] Experiments may be created, edited, and managed through a user interface system 132 such that the data configuration of the experiments are stored in the data store 190 or through the experiment management service 254. An experiment management service 254 may be communicatively coupled to the core services handler 140. The experiment management service 254 may include an audit trail module 250 for recording an audit trail of the data flows described above, a document workflow module 252 for managing the documents associated with experiments, and a data store 256. In an embodiment, the experiment management service 254 may be linked to a big data analysis service, such as STRATOMINER. Using the large data sets, protocols for experiments may be improved, such as using historical data to design better protocols for better outcomes. In another embodiment, the experiment management service 254 may include support for electronic lab notebooks (ELN), including automated upload of selected information to the ELN system. For example, the audit trail module 250 may provide a user interface with a prompt for ELN number and link a particular experiment to a whole study based on the ELN number in the ELN system. This beneficially enables the user to comply with GxP, a set of rules that are required for the safety and quality of pharmaceutical products, as well as compliance with various regulations (e.g. 21 CFR part 11) on how electronic records, electronic signatures, and handwritten signatures executed to electronic records are considered to be trustworthy,reliable, and generally equivalent to paper records and handwritten signatures executed on paper.

[0053] In an embodiment, medium from one plate may be used as an assay for another plate. This may require a special action to be performed at the end of an experiment to save a medium and assign it as available for use as an assay for another plate. In other embodiments, new plate types beyond standard micro titer plates may be used in the automated experiment handling device 130. such as plate types that support organ-on-a-chip devices.

[0054] An embedded PC 134 may communicate with the core services handler 140 through a file server 214 within the embedded PC 134. For example, the file server 214 may retrieve data from a local data storage 222 and transmit the data to the core services handler 140. The embedded PC 134 may also include a device service 224 that communicates directly with the core services handler 140. For example, the device service 224 may generate data, such as image data, and provide it to an image control and acquisition service 216 within the embedded PC 134 and store the image data within local data storage 222.

[0055] A separate image stitching service 218 may receive data from the core services handler 140 and store the results of the process in an image file storage 220. For example, an image stitching service 218 may stitch together separate images to create a larger image. Additionally, the core services handler 140 may store image files directly to the image file storage 220.

[0056] An image server 230 may request image files from an image file storage 220. Additionally, image data may be received from the core services handler 140, such as newly captured image data. The image server 230 may send the newly captured image data to an image analysis service 232 that includes an image analyzer 234, image processing algorithms 236, and a database 238. The processed image data may be sent to the core services handler 140 or the image server 230. In an embodiment, an image server 230 may serve images in the International Image Interoperability Framework (IIIF) format. In other embodiments, other interfaces may be used to serve images. The image server 230 may be extended with functionality for blending and contrast adjustment as well as support for multiple storage locations, in an embodiment. In an embodiment, image analyzer 234 may include an image processing algorithm 236 that tracks a singular object, not just a well. By tracking a singular object within an image, new discoveries may be made, especially with multiple domes of organoids in one 6 well plate. In another embodiment, multiple domes may be created in one well, where the exact position of each dome is tracked such that the imager and camera image directly at the domes. In a further embodiment, one or more image processing algorithms 236may divide an image into smaller images for better performance in image processing. For example. 4 wells of a 384 well plate may be captured and acquired in one image. The image may then be divided to analyze the 4 wells separately, beneficially increasing throughput of the image analysis process.

[0057] A user interface system 132 includes an image analysis user interface (UI) 240 for displaying an image analysis, an image control and image acquisition UI 242 for interacting with a planning user interface to configure when to capture images, and an automated experiment handling device UI 244 for presenting a user interface at the touchscreen or other hardware interface on the automated experiment handling device. In an embodiment, the image analysis UI 240 may enable a user to zoom in on an image and quickly identify various structures and / or objects within the image based on a library of images. Additionally, the image analysis UI 240 may enable a user to archive images to cloud storage to save on local storage, for example, and then also enable a user to restore the images. The user interface system 132 may communicate with the core services handler 140 through an API gateway 146. The user interface system 132 receives image files through the core services handler 140.

[0058] Figure 3 schematically illustrates an example computer-based scientific discovery system 300 that includes various functional components that may be implemented by the computer-based scientific discovery system 200 of Figure 2. For example, a core services handler 140 may communicate with a device manager 302, workflow execution 306. a file server 214, and a transport device module 314 of a hardware abstraction layer 304. A core sendees handler 140 may send a query' for devices 350 from the device manager 302 and may receive data regarding available devices 352 from the device manager 302. Additionally, the device manager 302 may send device status information 354 to the core services handler 140. The core services handler 140 may issue a workflow step 356 which is handled by workflow execution 306, and subsequently the workflow step result 358 may be returned to the core sendees handler 140.

[0059] A device manager 302 may include hardware (HW) monitoring 312 to monitor the status of devices in the hardware abstraction layer 304. Workflow execution 306 may include executing 308 the current step and error handling 310, such as moving a plate to incubator. In another embodiment, an example of error handling 310 may include handling a situation where a repository of tips is only half full where the system has designated the repository to be completely full. In this case, the system may be programmed to check the next available repository when it comes across the empty repository of tips.

[0060] As another example, in case of workflow failures or connection outages, local error handling strategies on the embedded PC 134 ensure that all plates are moved back to the incubator when the server PC is unable to send subsequent workflow steps. This is beneficial because if the server is being turned off, the workflow execution will continue in the automated experiment handling device 130 to automatically transport samples to the incubator to prevent them from dying. This ensures that extracted samples finish their treatment. The device manager 302 and workflow execution 306 are components of the device service 224 within the embedded PC 134. The hardware abstraction layer 304 includes a transport device module 314, a media supply device module 316, a laminar flow device module 318, a liquid handling device module 320, an incubator device module 322. a barcode device module 324, and an image control and acquisition device module 326. Hardware devices include a transport system 334 communicatively coupled with the transport device module 314, media supply 336 communicatively coupled with the media supply device module 316, laminar flow device 338 communicatively coupled with the laminar flow device module 318, liquid handler 340 communicatively coupled with a runtime environment 328, such as a software runtime environment for automated liquid handling systems, in the device service 224 that is connected to the liquid handling device module 320, an incubator 342 communicatively coupled with the incubator device module 322, a barcode scanner 344 communicatively coupled with the barcode device module 324 as well as an imager 346 and camera 348. both communicatively coupled with an image control and acquisition service 21 in the embedded PC 134 that is connected to the image control & acquisition device module 326.

[0061] The image control & acquisition device module 326 interfaces with the imager and implements basic plate acquisition, integrating the imager and hardware based autofocus for image capture. For example, an image of a plate may be captured by a camera 348 integrated within an imager 346 and processed by the image control & acquisition service 216. The imager 346 includes a movable stage to position the plate over the camera, different objectives (optical elements that gather light from the object being observ ed and focuses the light rays to produce a real image), filter cubes for confocal and widefield imaging, mechanisms to move the objectives and filter cubes, and light sources, in an embodiment. Additionally, the imager 346 includes a gripper to move the plate from the transport system to the stage. Various configurations of an imager 346 may be implemented in various embodiments. Then, the processed image data may be stored in local data storage 222 at the embedded PC 134. The file server 214 allows image files to be downloaded from theembedded PC 134 and sent to the core services handler 140 for more analysis. The file server then deletes the files after successful transmission, in an embodiment.

[0062] Figure 4 schematically illustrates an example computer-based scientific discovery system 400 that is implemented by elements of the computer-based scientific discovery system 200 of Figure 2. Scientific discovery system 400 may include a bus 410 interconnecting at least one processor 412, at least one memory 414, and at least one interface, such as application programming interface 416 and network interface 418. Bus 410 may include one or more conductors that permit communication among the components of scientific discovery system 400. Processor 412 may include any type of processor or microprocessor that interprets and executes instructions or operations. Memory' 414 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 412 and / or a read only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor 412 and / or any suitable storage element such as a hard disk or a solid state storage element. In some embodiments, processor 412 and memory 414 may be compute resources available for execution of logic or software instructions stored in memory 414 and computation intensive tasks, such as scheduling engine 436, may be configured to monitor and share these resources.

[0063] Application programming interface 416 may be configured for connection with one or more user devices and / or data modeling platforms. For example, application programming interface 416 may include a software interface that enables data transfer and / or communications between applications operating on user devices 170, automated experiment handling device 130 and / or core services handler 140.

[0064] Network interface 418 may include one or more wired or wireless network connections to network, similar to network 128. Network interface 418 may include a physical interface, such as an ethemet port, and related hardware and software protocols for communication over the network, such as a network interface card or wireless adapter.

[0065] Scientific discovery’ system 400 may include one or more non-volatile memory devices 420 configured to store data. For example, non-volatile memory devices 420 may include a plurality of flash memory packages organized as an addressable memory array and / or one or more solid state drives or hard disk drives. In some embodiments, non-volatile memory devices 420 may include a plurality of storage devices within, attached to. or accessible by a data modeling platform for storing and accessing data.

[0066] Scientific discovery system 400 may include a plurality of modules or subsystems that are stored and / or instantiated in memory 414 for execution by processor 412 as instructions or operations. For example, memory 414 may include a device controller 430 configured to send control instructions to at least one device within an automated experiment handling device 130, provide user control of at least one device through a device control interface 432, provide real-time device status from a device state manager 434, and capture and store data from that device through an event capture module 434. 1. Memory 414 may include an image file manager 442 configured to manage image files. Memory 414 may include a user manager 448 for managing user authorization for the scientific discovery' system 400 through user authorization module 448. 1. Memory 414 may also include an experiment manager 450 for receiving configuration protocols for experiments through a configuration protocol module 452, as well as managing experiment data through an experiment data module 454 that receives experiment data values 454. 1 from various components of the scientific discovery system 400 as well as extrapolated data values 454.2 based on algorithms.

[0067] Device controller 430 may include interface protocols, functions, parameters, and data structures for connecting to and controlling devices within or connected to an automated experiment handling device 130, capturing and storing data from those devices, and interfacing with other systems through the application programming interface 416. For example, device controller 430 may be an application and / or corresponding hardware in a core services handler 140 configured for network and / or direct communication with a set of associated user devices as well as an automated experiment handling device 130. Device controller 430 may be configured as a central point for determining instructions for device control from the associated user devices that enables user input to modify data configuration values for experiment protocols, planning, and scheduling. In some embodiments, device controller 430 may be divided among one or more servers and / or user devices. In an embodiment, device controller 430 may track the execution of the same or similar functions of the control subsystem 152 of Figure IB.

[0068] In some embodiments, device controller 430 may include a plurality of hardware and / or software modules configured to use processor 412 and memory 414 to handle or manage defined operations of device controller 430. For example, device controller 430 may include a device control interface 432, a device state manager 434, a scheduling engine 436, a decision making engine 438. and an image analysis and control engine 440.

[0069] Device control interface 432 may include device interface protocols and a set of functions, parameters, and data structures for using, configuring, communicating with, and providing command messages to devices within the automated experiment handling device 130 through application programming interface 416 and / or network interface 418. For example, device control interface 432 may include an API and command set for interacting with applications in each device to access one or more device functions. In some embodiments, device control interface 432 may be configured to set configuration parameters for device-specific operations, and / or otherwise manage operation of the devices. For example, device control interface 432 may maintain a device configuration table, pages, or similar data structures that includes entries for each device being managed and their respective user device-specific configuration parameters, active control features, and other configuration and control information for managing the devices. For example, a liquid handler 340, as illustrated in Figure 3, may have device-specific configurations, such as an amount of liquid treatment to feed a sample on a plate. As another example, a media supply- 336 comprises a large container for a liquid to be used in liquid handling as well as a pump to fill a reservoir on the deck, including a device-specific configuration that may include a maximum amount of media that can be held in the reservoir. As a further example, various measurements may be taken by an incubator 342, such as temperature and oxygen levels. These measurements may be used for monitoring device-specific configuration values specific to the incubator 342 to ensure that the device is operating within expected parameters.

[0070] A device state manager 434 may include interface protocols and a set of functions, parameters, and data structures for managing state data from devices within or connected to the automated experiment handling device 130. For example, device state manager 434 may include a monitoring watchdog module that routinely checks the status of each device. When a service is unavailable, the watchdog module restarts the service. The device state manager 434 may further include an event capture interface 434.1 may include interface protocols and a set of functions, parameters, and data structures for managing a plurality of data capture events from associated devices. In some embodiments, each device may a dedicated data channel for continuously and / or selectively sending its event data to event capture module 434. 1. For example, received event data may be buffered by event capture interface 434. 1 before being processed as workflow data values 438.5. In some embodiments, event capture interface 434. 1 may be configured to transport data to the decision making engine 438. Insome embodiments, event capture interface 434. 1 may receive or generate data based on other received data analysis or image analysis results 440.4.

[0071] Scheduling engine 436 may include a set of functions, parameters, and data structures for planning and scheduling steps of experiments based on received user input from user devices and / or user interfaces to enable real-time or near-real-time scheduling of experiment protocols responsive to received user input. In some embodiments, scheduling engine 436 may include a planning module 436.1 configured to determine an experiment plan based on user input, an inventory manager 436.2 configured to manage a physical inventory of supplies needed to execute experiment protocols, a scheduling data handler 436.3 configured to handle scheduling data to ensure various actions are performed at specific times, and scheduling event logic 436.4 configured based on the user input received in configuring the experiment protocol. In an embodiment, scheduling engine 436 may rely on a scheduling library for executing code at specific points in time or at other triggers. The scheduling library7may be used for running pre-computed schedules.

[0072] In some embodiments, planning module 436. 1 may be configured to provide a user interface for receiving data configuration values in determining an experiment protocol. For example, an experiment protocol may require that twelve (12) plates of the same media containing the same amount of cells of an organoid be fed at differing times with different dosages of a cancer fighting treatment. Thus, the planning module 436. 1 may enable a user to select which wells of each plate is fed with a user-specified dosage at a user-specified periodic time (e.g., hourly, every 4 hours, daily, etc.). The inventory manager 436.2 ensures that enough supplies needed to execute the complete experiment protocol are available. The scheduling data handler 436.3 sends the instructions of the schedule based on the user input (e.g., feed wells 1-3 every hour, feed wells 4-6 every four hours, capture image data for each plate hourly, capture image data on wells 1-6 on plate ABC daily, etc.). Scheduling event logic 436.4 may be generated based on the instructions of the schedule (e.g., instruct liquid handler to feed wells 1-3 every7hour, instruct liquid handler to feed wells 4-6 every four hours, direct the imager and camera to capture image data on wells 1 -6 on plate ABC daily, etc.). In some embodiments, actions may be taken by default, such as the laminar flow device maintaining a sterile environment, such that scheduling event logic 436.4 may include these default scheduling actions. In an embodiment, multiple experiments may be scheduled in overlapping time periods and users may be notified via email or other notification mechanism on all unexpected events, events that need the user’s attention, and whenever user input is required.

[0073] Decision making engine 438 may include one or more decision making algorithms and a set of functions, parameters, and data structures for processing data received from devices and / or user interfaces to enable real-time or near-real-time response to received data satisfying conditions in the one or more decision making algorithms, where the response is an action automatically taken by the automated experiment handling device. In some embodiments, decision making engine 438 may include experiment data values 438. 1 received from different types of devices included within or connected to the automated experiment handling device, baseline control values 438.2 configured to set a baseline of expected control values, action parameter values 438.3 configured based on received user input and / or pre-programmed as part of the one or more decision making algorithms, decision logic 438.4 configured based on received user input and / or included within the one or more decision making algorithms, and workflow data values 438.5 produced as a result of executing the actions configured to be executed from the action parameter values 438.3 and / or the one or more decision making algorithms. In an embodiment, the scheduling engine 436 includes scheduling event logic 436.4 such that all actions performed in a workflow by the embedded PC 134 are performed as a completely serialized workflow, including all steps and configuration parameters. The scheduling data handler 436.3 ensures that the completely serialized workflow7is performed by the embedded PC 134. In case of workflow failures or connection outages, local error handling strategies on the embedded PC 134 ensure that all plates are moved back to the incubator when the server PC is unable to send subsequent workflow steps.

[0074] In some embodiments, decision making engine 438 may enable the design of one or more decision making processes through user input received at a user interface (UI). For example, uploaded data 420. 1 stored in non-volatile memory 420 may be used by a user to represent baseline control values 438.2 for a particular experiment protocol. Uploaded data 420. 1, such as image data of a healthy cell culture, may be copied to a captured data table 420.4 such that baseline control values 438.2 may be generated from the uploaded data 420. 1. Experiment data values 438.1 may be generated by image acquisition engine 444 having image capture data 444.2. Thus, the uploaded data 420.1, transformed into baseline control values 438.2, may be compared to experiment data values 438.1 captured daily. Using action parameter values 438.3, which may, in some embodiments, correspond to configuration values 420.3 stored in non-volatile memory 420, the decision making engine 438 may identify a decision making algorithm in which a condition is met that satisfies executing an action defined in the action parameter values 438.3, such as passaging the cells to a new plate(e.g., dividing the sample in half) when the volume of cells exceeds a certain threshold amount. In some embodiments, decision logic 438.4 may be received via a user device connected through a network interface 418 and stored in configuration values 420.3 in nonvolatile memory 420. For example, user input may include an instruction that includes decision logic 438.4 expressed in a Boolean logic statement, such as passaging the cells to a new plate when the volume of cells exceeds a predetermined threshold amount of cells. Determining when the amount of cells exceeds the predetermined threshold amount may include image analysis results 440.4 from an image analysis and control engine 440, in an embodiment. Users may generate their own decision making algorithms, in an embodiment. A user-defined microservice may be used in decision making via APIs, for example. Other workflows may be triggered as actions through a REST API, such as accessing an image analysis service, or instructing, through a device controller 430, a transport device service to run a passaging action. Workflow data values 438.5 may include confirmation of the successful passaging of cells to a new plate and a barcode identifier of the plates involved in the passaging, as well as image data received of the new plate and the old plate. This workflow data values 438.5 may also be stored as device event data 420.5 in the captured data table 420.4.

[0075] Image analysis and control engine 440 may include a set of functions, parameters, and data structures for image analysis and control of imaging devices, such as an imager and a camera, as well as image analysis systems, image stitching services, and / or image file servers to enable analysis and control of image capture devices and image analysis systems responsive to one or more decision making algorithms. Although illustrated here as being instantiated in memory 414 on scientific discovery' system 400, in other embodiments, multiple image analysis systems can be used that operate on a separate computer within a local network or accessible through the network interface 418 such as a cloud service. For example, image analysis and control engine 440 may use image analysis logic 440.1 to determine whether a condition in a decision making algorithm is satisfied based on image control values 440.2. analysis parameters 440.3. and image analysis results 440.4. In an embodiment, image analysis results 440.4 may be received by an image analysis system to implement decision making and notify the user on certain conditions. For example, a scheduled experiment may require that a plate be fed with a certain amount of dosages via a liquid handler until an amount of grow th is reached. Based on the image analysis results 440.4, unexpected results may be determined such that a new schedule is implemented based on a decision making algorithm. Thus, the image analysis and control engine 440 providesthe image analysis results 440.4 needed to satisfy a condition in the decision making algorithm. In some embodiments, modem 3D deconvolution algorithms may be implemented in the imager, camera, microscope(s) and image processing algorithms to restore an effective specimen representation for 3D microscopy images. This would beneficially provide the user with more detail and data in experiments for scientific discovery.

[0076] Image file manager 442 may include data interface protocols and a set of functions, parameters, and data structures for managing image file transfers for various purposes, including providing images for display at a user device through a user interface for a viewing user to analyze and determine data configuration values for experiment planning purposes, retrieving images from image file stores through an image file transfer module 446, acquiring image capture data 444.4 from plates through an imager and / or a camera connected to an image acquisition engine 444, and receiving image analysis results 440.4 from an image analysis and control engine 440. For example, image file manager 442 may include an image acquisition engine 444 that uses baseline data values 444.1, such as an image of an empty plate, to refine image capture data 444.2, in an embodiment. A user interface may be provided by device controller 430 to present the at a user interface system integrated within an automated experiment handling device 130 or through a web application on a user device. The rendering of the user interface on the user device or user interface system may include compressed images by the image file manager 442. For example, scientific discovery system 400 may support continuous display of processed images of an experiment at the user interface on a user device as provided by the image file manager 442. In some embodiments, configuration values 420.3 are used by the image file manager 442 to generate images on a user interface that are correctly formatted for the user device. In some embodiments, the image file manager 442 may include an option to flag data which will not be saved to save storage. The option may be presented to the user during the design of the experiment or at another stage of the experiment based on the data being captured.

[0077] User manager 448 may include interface protocols, functions, parameters, and data structures for providing a user access to experiment data, including images of active experiments, as well as for generating and modifying experiment planning and / or protocols in the scientific discovery system 300, such as through device controller 430. For example, user manager 448 may be a software application running on a user device integral to, connected to, or in network communication with device controller 430 and / or an automated experiment handling device. In some embodiments, user manager 448 may run on a separate computingdevice from device controller 430, such as a personal computer, mobile device, embedded PC, server or other user device.

[0078] In some embodiments, user manager 448 may include a plurality of hardware and / or software modules configured to use processor 412 and memory 414 to handle or manage defined operations of user manager 448. For example, user manager 448 may include a user authorization module 448.1.

[0079] In some embodiments, user manager 448 may be configured to include a user authorization module 448.1 that manages a list of authorized users with specified permissions. Some users may be given full authorization to make changes to experiment protocols whereas other user may be given view-only permissions to view image analysis results 440.4 through the image file manager 442. The user authorization module 448. 1 may receive user-selected user permissions through a user interface, in an embodiment, presented at a user device or user interface system.

[0080] Experiment manager 450 may include data interface protocols and a set of functions, parameters, and data structures for managing experiment protocols and scheduling tasks within experiments for planning purposes. For example, experiment manager 450 may operate a user interface upon which a device controller 430 generates a user interface to record user input at a user device or user interface system. In some embodiments, uploaded data 420.1 appears on the experiment manager 450 as an interactive data flow interface. For example, the experiment manager 450 may include a configuration protocol module 452 that enable a user to define configuration values for an experiment protocol. Additionally, the experiment manager 450 may include an experiment data module 454 that is configured to orchestrate and execute experiments within the automated experiment handling device 130 through the device controller 430. For example, scientific discovery’ system 400 may support continuous execution, display and / or capture of experiment data values 454.1 at a user interface on a user device operating the experiment manager 450. The experiment data module 454 includes, in some embodiments, experiment data values 454.1 that may be copied from experiment data values 438. 1 from a decision making engine 438 as well as workflow data values 438.5 that may be copied as extrapolated data values 454.2. In some embodiments, configuration values 420.3 are used by the experiment manager 450 to generate the extrapolated data values 454.2 based on an expected growth factor of cells responding to a dosage treatment, for example. In another embodiment, extrapolated data values 454.2 may be generated based on images taken every 24 hours to find the best date for a certain action to be performed (e.g., start passaging). Algorithms may be used to facilitatemore dynamic scheduling, such as extrapolating a best date for a trigger using current data, in an embodiment. For example, when an expected data is in between two imaging timepoints and the second time point would be too late, an extrapolating algorithm may optimize to select the earlier imaging timepoint and potentially suggest a date to trigger an action, such as start passaging. As another example, experiment manager 450 may monitor experiment data values 454.1 to track cell growth, such as monitoring cell diameter and bubble formations. Decision logic 438.4 within a decision making engine 438 may be triggered by the monitored experiment data values 454.1 reaching action parameter values 438.3, for example. This may cause a decision to passage the cells, as configured by a user through the experiment manager 450.

[0081] Experiment manager 450 may include interface protocols, functions, parameters, and data structures for providing an audit trail and document workflows for recording unexpected events as well as recording modified experiment protocols and / or storing experiment metadata in non-volatile memory7420, such as through device controller 430. This may beneficially enable compliance and conformity with various regulations, such as 21 CFR part 1 1 for GxP compliance. For example, experiment manager 450 may be a software application running on a device integral to, connected to, or in network communication with device controller 430, an automated experiment handling device, and / or a core services handler. In some embodiments, experiment manager 450 may run on a separate computing device from device controller 430. such as a personal computer, server, embedded PC. mobile device, or other user device. Tn some embodiments, experiment manager 450 may be configured to interact with APIs presented by an API gateway to receive and transmit experiment data to other systems, such as a user interface on a user device operating a web application or a user interface presented on a user interface system.

[0082] Figure 5 schematically illustrates a data flow interaction of elements of the computer-based scientific discovery system 200 of Figure 2. For example, within a server PC 500, an action trigger may be scheduled at block 502. A processor on the server PC 500 mayread the action parameters at block 504 after retrieving the action parameters 532 based on the scheduled action trigger from the schedule data store 530. Conditions for decision making may be configured for different actions. A single condition may be defined as an expression that, when evaluated, yields either “true’’ or “false.” When “true”, the corresponding action is taken.

[0083] A user interface may be constructed and implemented to facilitate building up expressions. Relying on disjunctive normal form, every7expression may be constructed froma disjunction of conjunctions in Boolean logic. For example, symbols may be used where (|| == OR, && == AND). Thus, an example expression may be (Confluency > 0.8 && Area > 15) || (NumberOfCells < 2), where “Confluency’’ refers to the percentage of the surface of a culture dish that is covered by adherent cells, “Area” refers to an actual measurement of surface area of cell culture vessels, and “NumberOfCells” is a calculated estimate of the number of cells in a well or plate. Groups of conditions that use a feature and a threshold may be created, and those groups may then be combined with an “AND” Boolean operator. Then, other groups may be combined via an “OR” operator, enabling various Boolean expressions to be created through a user interface.

[0084] Actions may be performed in response to conditions. For example, after imaging every plate, using all images that belong to the current experiment and the current phase, a corresponding condition may be evaluated to determine whether a passaging action is run. If the condition holds “true”, the passaging action is run. In another example, conditions may be run on all images at a certain point in time of a specific well. The conditions are run after imaging of a single plate. The result of the imaging determines if the well is to be excluded from further processing, e.g., if the cells within the well have died. In a further example, a user may be sent notifications based on a condition being met for various situations, including a condition run on all plates, on a single plate, or for each well. As a result, the user may receive an email notification on issues with single wells, single plates, or the complete experiment.

[0085] Decision making processes can be implemented in different aspects, such as generating the input data for decision making using image processing, configuration of decision making, and evaluating conditions and reacting to the corresponding decisions, in an embodiment. A user may be presented with a user interface to select and provide user input to configure the experiment protocol and one or more decision making processes for the automated experiment handling device to execute as a series of workflows. For example, a user may select classifiers for classical image classification analysis and / or Al-based classification. Various metric thresholds may be included, such as selecting protocol and metric, selecting operator (< >), and entering a threshold. A well threshold may be defined for a well ensemble (a percentage of wells) or for a well solo (for each well individually). An action may be defined for a well ensemble to start 2D passaging, start 3D passaging, end current phase and begin next phase, add plate annotation to the experiment document, and notify the user. Similarly, an action may be defined for a well solo to notify the user, add well annotation to the experiment document, re-scan the well with a higher magnification,and remove the well from the experiment. The above actions and conditions may be combined using Boolean operators to generate expressions that can be inputted via a user interface, for example.

[0086] In block 506, the action is run, which includes sending workflow to device service at block 508. This is sent to block 562 in which the embedded PC 550 is waiting for workflow. At block 564, the workflow is run. Workflow updates are sent at block 566. These updates are received at block 510 in monitoring workflow. A status 534 may be sent to the schedule data store 530 as updates are received. In another embodiment, a status 534 may be sent periodically from the run action block 506. When the run action block 506 is completed, the processor of the server PC 500 is instructed to copy new images from the embedded PC in block 512. This involves retrieving image files from the image file storage 552 on the embedded PC 550 and storing the new images to the image storage 540 on the server PC 500.

[0087] Based on new images being copied from the embedded PC at block 512, a decision making block 514 may be executed by a processor at the server PC 500. At decision block 516, it is determined whether decision making is needed. If decision making is not needed, then the decision making process 514 ends. If decision making is needed, then a request for image analysis is performed at block 518. This initiates image analysis to begin at block 542. For example, an image analysis service, such as one depicted in Figure 2, may be used for image analysis at block 542. Image data may be retrieved from image storage 540, such as the new images copied from the embedded PC at block 512. The image analysis will produce image analysis results after completion of block 542. At block 520, the results are analyzed. For example, a number of cells may be determined based on image analysis algorithms. As another example, a positive reaction to dosage treatment may be determined, such as a reduction in cancer growth based on baseline values for growth and measured growth. Other types of analysis may be performed at block 520, including notifying the user of an unexpected result through an email or other user notification. Classical image analysis algorithms as well as artificial intelligence (Al) based classification algorithms may be used to generate analysis results.

[0088] At decision block 522, it is determined whether a rule is true. For example, the decision making that is needed as determined in block 516 may involve a rule, or a set of criteria, that triggers other actions. If the answer is no, then the decision making block 514 ends. If the answer is yes, then the schedule is updated and re-planned in block 524. A decision making algorithm may include various conditions that may assist with determining whether a rule is true in decision block 522. Based on the results of an analysis of an image,as performed in block 520. certain measurements may have been made that trigger an action based on the rule being true. For example, specific actions may be performed based on a condition that the cell confluence has reached a predetermined threshold, such as a “run passaging” action and re-scanning a well at a higher magnification. User notifications may be sent based on certain conditions, such as diameter reaching a certain measurement. In an embodiment, selected wells may be excluded from further processing based on decisionmaking logic. In other embodiments, a plate annotation may be added to the experiment document based on a set of user-inputted and / or experiment design criteria being evaluated as true. In further embodiments, a well annotation may be added to the experiment document. In yet another embodiment, the current phase of the experiment may be concluded and a new experiment may be started that is scheduled next, or the experiment may be finished. In some embodiments, passaging is performed on an experiment basis, or batch of plates basis. In other embodiments, passaging may be performed on a plate basis with individual passaging parameters, such as splitting ratio and volume, depending on the individual cell parameter, such as confluency. By enabling a sample to be passaged at the very best moment, treating plates differently based on individual cell parameters, beneficially enables further advances in scientific discoveries for patient derived organoids.

[0089] At block 526, the schedule may be generated. Schedule data 536 may be stored at the schedule data store 530. The scheduler problem may also be updated 538 as a result of the update and re-plan action executed at block 524. The scheduler problem may be updated 538 in the schedule data store 530, in an embodiment. For example, because workflows are scheduled to occur serially and because several experiments with overlapping time periods may be running in one automated experiment handling device, the scheduler problem is updated based on the new actions being taken based on the results of the image analysis in the decision making process block 514. The process may then repeat as needed to complete the experiment protocol as new action triggers are scheduled at block 502.

[0090] Other algorithms may be used to facilitate more dynamic scheduling, such as extrapolating a best date for a trigger using current data, in an embodiment. For example, when an expected data is in between two imaging timepoints and the second time point would be too late, an extrapolating algorithm may optimize to select the earlier imaging timepoint and potentially suggest a date to trigger an action, such as start passaging.

[0091] In an embodiment, relative timepoints of phase transitions and other data may be tracked per protocol. In this way, timepoints may be predicted for future experiments basedon this historical tracking data. Additionally, this information may be used to quantify robustness of protocols or new cell lines.

[0092] In yet another embodiment, curves may be overlayed with data from previous experiments (plural, same protocol). The minimum, maximum, mean, and other statistical calculations and / or differences may be shown. For example, data may be plotted based on time versus diameter. The statistical differences may be shown through curve overlayed with data from previous experiments. This may provide a user with similar information as saving data per protocol.

[0093] In an embodiment, a decision making process may enable the scientific discovery system 200 to detect unexpected structures, cells, and / or organoids and inform the user or make suggestions, such as what the unexpected items are and what would happen. For example, the system may suggest that unexpected items are mold, potential contamination, dust, an empty well, HeLa cells, and so forth, providing predictions on what would happen for each case. This would beneficially empower the user with more information as to the unexpected items detected by the system.

[0094] In a further embodiment, sensor devices may calculate partial oxygen and carbon dioxide pressure based on media volume and media-air surface, as well as time duration of samples spent outside incubator. This additional data may be used to warn the user if the plate has been outside the incubator for too long, for example. Additionally, this information may be used to generate hypoxia conditions for assays, which may be especially beneficial for tumoroids. In yet another embodiment, sensor devices may also determine kinetics, such as a frequency of a heart beat or movement of larvae. This additional sensor information on kinetics may be used in decision making processes to generate additional actions, such as recording the frequency of a heart beat after a treatment has been fed to a sample.

[0095] As shown in Figure 6, scientific discovery system 200 may be operated according to an example data flow interaction diagram illustrating image capture and storage elements of the scientific discovery7system 200 having a user device, a server PC and an embedded PC.

[0096] At block 602. a user device 600 may send snap command and parameters, meaning that the imager and / or camera are instructed to capture an image based on user- specified parameters. As a result, the server PC 630 relays imager commands at block 632 to the embedded PC 660 which further relays imager commands at block 672 from within the device service 670 to the image control and acquisition service 680. For example, a user interface may be generated to enable a user to request a plate be re-scanned at a highermagnification level, where the user interface is presented on the user device 600 through a web application or through a user interface system attached to the automated experiment handling device 130. The commands are relayed to the server PC 630 and to the embedded PC 660 through a REST API. Other example commands that may be communicated through a REST API between a user device 600, server PC 630, and embedded PC 660 include user login and user management, interactive instrument control for the imager, liquid handler, incubator, and transport system, system configuration (including temperatures, gas concentrations, etc ), plate management for plates in the incubator, liquid handler worktable configuration, labware management to manage various physical items (e.g., plate t pes, liquid classes, etc.), and experiment and protocol management. Other APIs may be used for other purposes, such as using an IIIF Image API for serving images in the in tiles and allowing for blending images and contrast adjustment. A separate REST API may be used for image analysis requests and for receiving image analysis results.

[0097] Returning to Figure 6, at block 604, the user device 600 requests an image update. The result of the imaging run at block 682 in the image control and acquisition service 680 is sent to the device service 670 which is sent at block 674 from the embedded PC 660 to the server PC 630 where the result is relayed at block 634 to the user device 600. The server PC then copies the images at block 636 from the image file storage 662 on the embedded PC 660 and stores the images at the image storage 640 on the server PC or within cloud storage.

[0098] At block 606. the user device 600 shows the image, retrieving the image data from the image storage 640 on the server PC 630 or from cloud storage.

[0099] As shown in Figure 7, scientific discovery system 200 may be operated according to an example data flow interaction diagram illustrating experiment data elements of the scientific discover)’ system 200 having a user device and a server PC.

[0100] At block 702, a user device 700 may get experiment data from experiment data storage 732 at a server PC 730 or stored in cloud storage. Experiment data may include an audit trail of the experiment, experiment protocols as configured by user input, and experiment documents, including annotations on wells and plates.

[0101] At block 704. a user device 700 may request analysis results. At block 706. post processing may occur at the user device 700 after receiving experiment data from experiment data storage 732, image analysis results from an image analysis sendee 740, and / or an image from image storage 750. Next, at block 708, the results of the analysis and the post processing may be visualized at the user device 700. As a result, data may be retrieved from the experiment data storage 732 at the server PC 730 or at cloud storage. Additionally, imageanalysis results may be retrieved from image storage 750, in an embodiment. In another embodiment, image analysis is required to be run at block 742 based on the request for analysis results in block 704. Associated image data may be retrieved from image storage 750 by an image analysis service 740 executing the run image analysis block 742. When the image analysis is complete, the results may be sent to the user device 700 at block 706 for post processing which would then move to block 708 to visualize the results on the user device 700 at a user interface, such as a web application, or through a user interface system attached to an automated experiment handling device 130.

[0102] Figure 8 is a flowchart of an example method of automating decision making for experiment-based scientific discovery. Method 800 begins with generating input data for decision making using image processing at step 810. For example, images are acquired for a plate, and analysis is run on each plate to generate analysis results that are stored in a database. Both classical image analysis algorithms as well as Al based classification algorithms may be used to generate the analysis results. Analysis on the plate includes segmentation of the images, analysis based on a selected classifier, and results based on the analysis. For example, a scientist may wish to identify cell cultures that have both a confluency greater than 0.80 and have an actual area of greater than 15 mmA2, or where the number of cells is less than 2. This step of analysis generates input data for decision making using image processing such that metrics are determined according to the configuration parameters (i. e. , the example expression above, (Confluency > 0.80 && Area > 15) || (NumberOfCells < 2)).

[0103] At step 812, user input is received to configure decision making. For example, the example expression above may be entered as user input, and a selected action would be to retrieve image analysis results for display on the user device.

[0104] At step 814, one or more conditions are determined based on the received user input. In this example, the conditions determined would include the Confluency metric meeting a threshold of 0.80 and an actual Area on the plate greater than 15, or a number of cells less than 2. Based on the retrieved processed images, the conditions would be determined at step 814.

[0105] At step 816, one or more actions would be generated based on the one or more conditions evaluated with the generated input data. In this example, the images meeting the conditions in the expression would be retrieved and presented for display in a user interface on a user device. The method 800 may then repeat as new input data is continuouslygenerated as a result of ongoing experiments. In another embodiment, the method 800 may end after step 816.

[0106] Figure 9 is a flowchart of an example method of generating image analysis for experiment-based scientific discovery. Method 900 begins at step 910 where images are acquired for plate. For example, an imager and a camera may be connected to an image acquisition and control service configured to acquire images for each plate at a specified time or upon completion of an action.

[0107] At step 912, an analysis protocol is retrieved based on the plate. For example, the analysis protocol may be to use segmentation and a classical image classification to analyze each image associated with the plate. As another example, the analysis protocol may use segmentation and an Al-based classifier to analyze all wells in the plate. In other embodiments, an analysis protocol may be retrieved based on the entire experiment (i.e., batches of plates), or by individual well.

[0108] At step 914, the analysis is run on the plate using the analysis protocol. For example, the images may be processed through an image analysis service such that each image associated with the plate is processed with the analysis protocol.

[0109] At step 916, one or more results are generated based on the analysis on the plate. For example, the one or more results generated may include a conclusion that a certain metric has been achieved, such as confluence of 0.80, or number of cells being less than 4.

[0110] At step 918, the one or more results are stored in a data store. For example, the one or more results may be stored in a data store based on the experiment and plate.

[0111] Figure 10 is a flowchart of an example method of performing actions based on conditions for experiment-based scientific discovery. Method 1000 begins at step 1010 where a user interface is provided to receive one or more configuration instructions associated with a condition to render a decision.

[0112] At step 1012, user input is received at the user interface where the user input comprises the one or more configuration instructions. For example, expressions may be entered in a user interface constructed and implemented to facilitate building up expressions. Relying on disjunctive normal form, even’ expression may be constructed from a disjunction of conjunctions in Boolean logic. Other algorithms may be used, such as extrapolating a best date for a trigger using current data.

[0113] At step 1014, the one or more configuration instructions are evaluated based on the received input data. For example, the expressions inputted by the user include configuration instructions defining the types of conditions to be evaluated.

[0114] At step 1016, the decision is determined based on the condition and the received input data. For example, the image processing performed on the images as well as the image analysis may be used to evaluate whether the conditions are satisfied.

[0115] At step 1018, one or more actions are performed based on the decision. For example, the action may be retrieving images that match the conditions. As another example, the action may be to re-plan the schedule based on the conditions being satisfied. Other actions may include ending the phase of the experiment and beginning the next phase, or finishing the experiment. Any number of actions may be configured by the user through the expressions inputted by the user.

[0116] Some examples relate to a computer-implemented method, the method comprising receiving image data associated with a well in a plate within an automated experiment handling device based on a schedule of tasks associated with an automated workflow.Further, the method comprises generating input data associated with the automated workflow, the input data generated using an image processing service to process the image data. Additionally, the method comprises receiving user input to configure a new workflow and determining one or more conditions of the new workflow based on the received user input. Furthermore, the method comprises updating the schedule of tasks in the automated workflow with the new workflow based on existing experiment protocol timing requirements and providing a control signal capable of triggering one or more actions to be executed in the automated experiment handling device according to the updated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

[0117] By an automatic update of workflows based on existing requirements, the costs, the time effort and / or the error rate of complex experiments may be significantly reduced.

[0118] An automated workflow may include a list of scheduled tasks or actions, conditions of tasks or actions, and / or timing requirements of tasks or actions. A task or action may be any task or action executable by an automated experiment handling device as described above or below.

[0119] Input data associated with the automated workflow is generated based on an image analysis of the image data, for example. Examples for input data generated by an image analysis are mentioned above (e.g. in connection with Fig. 1-10) or below.

[0120] The new workflow may include one or more tasks or actions, which should be added to the automated workflow and / or replace one or more tasks or actions of the automated workflow or the new workflow may include one, more or all tasks or actions ofthe automated workflow and includes one or more additional tasks or actions. The one or more conditions of the new workflow may be one or more conditions mentioned with respect to tasks or actions described above or below.

[0121] The existing experiment protocol timing requirements may comprise timing requirement of tasks or actions of the automated workflow and / or the new workflow and / or timing requirement mentioned above or below.

[0122] The control signal may comprise instructions, which cause the automated experiment handling device to execute one or more actions based on the updated schedule of tasks in the automated workflow. The control signal may be transmitted to a computing device (e.g. an embedded PC, a microcontroller, a processor, an application-specific integrated circuit ASIC or a Field Programmable Gate Array FPGA) of the automated experiment handling device. The computing device may control the execution of the one or more actions based on the control signal.

[0123] The image data may be associated with a well in a plate received from an imager included in the automated experiment handling device.

[0124] A location of the plate within the automated experiment handling device may be determined based on the schedule of tasks in the automated workflow. For example, the location can be determined, since the automated workflow may define for each plate which task is done at which time resulting in a defined location for each plate at any time.

[0125] The method may further comprise generating a data visualization that depicts processed image data, the data visualization including a history of recorded actions performed on the well in the plate, a history of cell passaging across one or more plates, a plurality of images associated with the well, and / or a plurality of image analysis results over time. Further, the method may comprise providing or causing the data visualization to be displayed in a user interface at a user device. The history of recorded actions may comprise a time and / or date of actions and / or the sequence of actions. The history of cell passaging may comprise a time and / or date of cell passagings and / or the sequence of cell passagings.

[0126] The user input may comprise an expression having the one or more conditions associated with the one or more actions to be executed when the one or more conditions are satisfied.

[0127] The schedule of tasks in the automated workflow may include the existing experiment protocol timing requirements. The method may further comprise determining a set of new tasks associated with the new workflow, determining one or more new timing requirements associated with the set of new tasks, and generating the updated schedule oftasks based on the one or more new timing requirements associated with the set of new tasks and the existing experiment protocol timing requirements.

[0128] The one or more actions may be triggered to be executed in the automated experiment handling device by sending a request through an application programming interface (API) gateway connected to a computing device of the automated experiment handling device.

[0129] The automated experiment handling device may comprise a plurality of platehandling devices, including a transport system to handle a plurality of plates associated with one or more experiment protocols, an incubator providing an environment for cell culture growth in the plurality of plates, an imager configured to capture periodic images of the plurality of plates, a liquid handling system providing periodic liquid maintenance on the plurality of plates, a laminar flow system providing sterile air for the plurality of plates outside of the incubator, and / or one or more component identifiers to uniquely identify the plurality7of plates and one or more inventory7supply containers.

[0130] The method may further comprise sending one or more instructions to the automated experiment handling device, the one or more instructions including the one or more actions to be executed in the automated experiment handling device. Further, the method may comprise receiving a status update from the automated experiment handling device, the status update including a progress indication of the one or more actions. Additionally, the method may comprise, responsive to receiving an indication of completion of the one or more actions, retrieving one or more new images from the automated experiment handling device associated with the one or more actions. Furthermore, the method may comprise storing the one or more new images in a data store and determining a next task in the updated schedule of tasks in the automated workflow.

[0131] More details and aspects are mentioned in connection with the embodiments described above or below. The method may' comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g. Fig. 1-10) or below.

[0132] Some examples relate to a system comprising one or more processors and one or more storage devices. The system is configured to receive image data associated with a plate within an automated experiment handling device based on a schedule of tasks associated with an automated workflow and generate input data associated with the automated workflow, the input data generated using an image processing service to process the image data. Further, the system is configured to receive user input to configure a new workflow' and determine one ormore conditions of the new workflow based on the received user input. Additionally, the method is configured to update the schedule of tasks in the automated workflow with the new workflow based on existing experiment protocol timing requirements and providing instructions capable of triggering one or more actions to be executed in the automated experiment handling device according to the updated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

[0133] The user input to configure the new workflow may be received through a user interface provided by a user interface system communicatively coupled to the system, the user interface provided for display.

[0134] The tasks performed by the system may be executed by one or more software components (e.g. a device controller as mentioned above or below) running on the one or more processors.

[0135] More details and aspects are mentioned in connection with the embodiments described above or below. The method may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g. Fig. 1-10) or below.

[0136] Some examples relate to a computer-implemented method, the method comprising providing a user interface for display on a user device communicatively coupled to the server, the user interface including a set of data values corresponding to physical attributes of components used in an automated experiment handling device. Further, the method comprises receiving, from the user device, user input to define an experiment protocol including one or more configuration instructions associated with the components used in the automated experiment handling device and an automated workflow including one or more conditions associated with actions in the automated workflow, the user input received through the user interface. Additionally, the method comprises scheduling a first action in the automated workflow, the first action having one or more action parameters and evaluating the one or more conditions associated with the first action in the automated w orkflow .Furthermore, the method comprises, based on the one or more conditions associated with the first action evaluated as being satisfied, causing the first action to be performed at the automated experiment handling device according to the one or more action parameters, and sending a notification to the user device indicating that the first action has been triggered based on the one or more conditions being satisfied.

[0137] By automatically evaluating if conditions for an action are satisfied, the costs, the time effort and / or the error rate of complex experiments may be significantly reduced.

[0138] The one or more configuration instructions may be any configuration mentioned in connection with components of the automated experiment handling device described above or below.

[0139] The set of data values corresponding to physical attributes of components used in the automated experiment handling device may comprise at least one of a quantity of inventory supplies required by the experiment protocol, a quantity’ of inventory supplies available in the automated experiment handling device, a first identifier associated with a plate having a standard number of deep wells, a second identifier associated with a plate having a standard number of shallow w ells, a third identifier associated with a plate having a small number of shallow w ells, a fourth identifier associated with a plate having a medium number of shallow wells, a fifth identifier associated with a plate having a large number of shallow wells, a plurality of unique well identifiers, each unique well identifier associated with an individual well in the automated experiment handling device, a plurality of unique plate identifiers, each unique plate identifier associated with an individual plate in the automated experiment handling device, a media supply identifier associated with a media supply used in cell culture cultivation, a plurality of liquid handling identifiers, each liquid handling identifier associated with a liquid fed in the automated experiment handling device, a plurality of environmental parameters, each environmental parameter describing a unique environmental attribute within the automated experiment handling device, or a plurality of timing parameters, each timing parameter indicating a period of time expected to complete an associated action in the automated experiment handling device.

[0140] The user input may comprise an expression of the one or more conditions, the expression using one or more Boolean operators to describe the one or more conditions.

[0141] Evaluating the one or more conditions may comprise, based on the one or more conditions, retrieving one or more image files, each image file associated with a well in a plate in the automated experiment handling device. Further, evaluating the one or more conditions may comprise generating one or more image analysis results using an image processing service and evaluating the one or more conditions based on the one or more image analysis results.

[0142] The method may further comprise receiving an indication that a first resource of inventory supplies needed to complete a current task has been depleted and determining a second resource of inventory supplies that is available in the automated experiment handling device. Additionally, the method may comprise modifying the current task to use the second resource of inventory' supplies and updating one or more scheduled tasks based on acalculated data value corresponding to a quantity of inventory supplies based on the first resource being depleted.

[0143] More details and aspects are mentioned in connection with the embodiments described above or below. The method may comprise one or more optional additional features corresponding to one or more aspects mentioned in connection with the proposed concept or one or more examples described above (e.g. Fig. 1-10).

[0144] While at least one exemplary embodiment has been presented in the foregoing detailed description of the technology, it should be appreciated that a vast number of variations may exist. It should also be appreciated that an exemplary7embodiment or exemplary embodiments are examples, and are not intended to limit the scope, applicability, or configuration of the technology in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary7embodiment of the technology7, it being understood that various modifications may be made in a function and / or arrangement of elements described in an exemplary7embodiment without departing from the scope of the technology, as set forth in the appended claims and their legal equivalents.

[0145] As will be appreciated by one of ordinary skill in the art, various aspects of the present technology7may be embodied as a system, method, or computer program product. Accordingly, some aspects of the present technology may take the form of an entiretyhardware embodiment, an entirely software embodiment (including firmware, resident softw are, micro-code, etc ), or a combination of hardware and software aspects that may all generally be referred to herein as a circuit, module, system, and / or network. Furthermore, various aspects of the present technology may take the form of a computer program product embodied in one or more computer-readable mediums including computer-readable program code embodied thereon.

[0146] Any combination of one or more computer-readable mediums may be utilized. A computer-readable medium may be a computer-readable signal medium or a phy sical computer-readable storage medium. A physical computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, crystal, polymer, electromagnetic, infrared, or semiconductor system, apparatus, or device, etc., or any suitable combination of the foregoing. Non-limiting examples of a physical computer-readable storage medium may include, but are not limited to, an electrical connection including one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), readonly memory (ROM), an erasable programmable read-only memory (EPROM), anelectrically erasable programmable read-only memory' (EEPROM), a Flash memory, an optical fiber, a compact disk read-only memory (CD-ROM), an optical processor, a magnetic processor, etc., or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program or data for use by or in connection with an instruction execution system, apparatus, and / or device.

[0147] Computer code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to, wireless, wired, optical fiber cable, radio frequency (RF), etc., or any suitable combination of the foregoing. Computer code for carry ing out operations for aspects of the present technology may be written in any static language, such as the C programming language or other similar programming language. The computer code may execute entirely on a user’s computing device, partly on a user’s computing device, as a stand-alone software package, partly on a user’s computing device and partly on a remote computing device, or entirely on the remote computing device or a server. In the latter scenario, a remote computing device may be connected to a user’s computing device through any type of network, or communication system, including, but not limited to, a local area network (LAN) or a wide area network (WAN), Converged Network, or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider).

[0148] Various aspects of the present technology may be described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products. It will be understood that each block of a flowchart illustration and / or a block diagram, and combinations of blocks in a flowchart illustration and / or block diagram, can be implemented by computer program instructions. These computer program instructions may be provided to a processing device (processor) of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which can execute via the processing device or other programmable data processing apparatus, create means for implementing the operations / acts specified in a flowchart and / or block(s) of a block diagram.

[0149] Some computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device(s) to operate in a particular manner, such that the instructions stored in a computer- readable medium to produce an article of manufacture including instructions that implement the operation / act specified in a flowchart and / or block(s) of a block diagram. Some computerprogram instructions may also be loaded onto a computing device, other programmable data processing apparatus, or other device(s) to cause a series of operational steps to be performed on the computing device, other programmable apparatus or other device(s) to produce a computer-implemented process such that the instructions executed by the computer or other programmable apparatus provide one or more processes for implementing the operation(s) / act(s) specified in a flowchart and / or block(s) of a block diagram.

[0150] A flowchart and / or block diagram in the above figures may illustrate an architecture, functionality, and / or operation of possible implementations of apparatus, systems, methods, and / or computer program products according to various aspects of the present technology. In this regard, a block in a flowchart or block diagram may represent a module, segment, or portion of code, which may comprise one or more executable instructions for implementing one or more specified logical functions. It should also be noted that, in some alternative aspects, some functions noted in a block may occur out of an order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or blocks may at times be executed in a reverse order, depending upon the operations involved. It will also be noted that a block of a block diagram and / or flowchart illustration or a combination of blocks in a block diagram and / or flowchart illustration, can be implemented by special purpose hardw are-based systems that may perform one or more specified operations or acts, or combinations of special purpose hardware and computer instructions.

[0151] While one or more aspects of the present technology have been illustrated and discussed in detail, one of ordinary skill in the art will appreciate that modifications and / or adaptations to the various aspects may be made without departing from the scope of the present technology, as set forth in the following claims.

Claims

Claims:

1. A computer-implemented method, comprising: receiving, at a server, image data associated with a well in a plate within an automated experiment handling device based on a schedule of tasks associated with an automated workflow; generating, at the server, input data associated wi th the automated workflow, the input data generated using an image processing service to process the image data; receiving, at the server, user input to configure a new workflow : determining, at the server, one or more conditions of the new7workflow based on the received user input; updating, at the server, the schedule of tasks in the automated workflow with the new w orkflow7based on existing experiment protocol timing requirements; and causing, at the server, one or more actions to be executed in the automated experiment handling device according to the updated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

2. The method of claim 1, wherein the image data associated with a well in a plate is captured using an imager included in the automated experiment handling device.

3. The method of one of the previous claims, w herein a location of the plate w ithin the automated experiment handling device is determined based on the schedule of tasks in the automated workflow.

4. The method of one of the previous claims, further comprising: generating, at the server, a data visualization that depicts processed image data, the data visualization including a history of recorded actions performed on the w ell in the plate, ahistory of cell passaging across one or more plates, a plurality of images associated with the w ell, and a plurality of image analysis results over time; and causing the data visualization to be displayed in a user interface at a user device.

5. The method of one of the previous claims, wherein the user input comprises an expression having the one or more conditions associated with the one or more actions to be executed when the one or more conditions are satisfied.

6. The method of one of the previous claims, wherein the schedule of tasks in the automated workflow7includes the existing experiment protocol timing requirements, the method further comprising: determining a set of new tasks associated with the new workflow; determining one or more new timing requirements associated with the set of new7tasks; andgenerating the updated schedule of tasks based on the one or more new timing requirements associated with the set of new tasks and the existing experiment protocol timing requirements.

7. The method of one of the previous claims, wherein the one or more actions are caused to be executed in the automated experiment handling device by sending a request through an application programming interface (API) gateway connected to an embedded PC within the automated experiment handling device.

8. The method of claim 7, wherein: the automated experiment handling device comprises a plurality of plate-handling devices, including: a transport system to handle a plurality of plates associated with one or more experiment protocols, an incubator providing an environment for cell culture growth in the plurality' of plates, an imager configured to capture periodic images of the plurality of plates, a liquid handling system providing periodic liquid maintenance on the plurality of plates, a laminar flow system providing sterile air for the plurality of plates outside of the incubator, and one or more component identifiers to uniquely identify the plurality of plates and one or more inventory' supply containers, the method further comprising: sending one or more instructions in the request to the embedded PC w ithin the automated experiment handling device, the one or more instructions including the one or more actions to be executed in the automated experiment handling device; receiving a status update from the embedded PC, the status update including a progress indication of the one or more actions; responsive to receiving an indication of completion of the one or more actions. retrieving one or more new' images from the embedded PC associated with the one or more actions; storing the one or more new images in a data store; and determining a next task in the updated schedule of tasks in the automated workflow'.

9. A system, comprising: a processor; anon-volatile memory; a device controller, operable by the processor on the non-volatile memory, the device controller configured to: receive image data associated with a plate within an automated experiment handling device based on a schedule of tasks associated with an automated workflow; generate input data associated with the automated workflow, the input data generated using an image processing service to process the image data; receive user input to configure a new workflow; determine one or more conditions of the new workflow based on the received user input; update the schedule of tasks in the automated workflow with the new workflow based on existing experiment protocol timing requirements; and cause one or more actions to be executed in the automated experiment handling device according to the updated schedule of tasks in the automated workflow based on the one or more conditions evaluated with the generated input data.

10. The system of claim 9, wherein the image data associated with a plate is captured using an imager included in the automated experiment handling device.

11. The system of claim 9 or 10. wherein a location of the plate within the automated experiment handling device is determined based on the schedule of tasks in the automated workflow.

12. The system of claim 9, 10 or 11, wherein the user input to configure the new workflow is received through a user interface provided by a user interface system communicatively coupled to the device controller, the user interface provided for display.

13. The system of claim 9, 10, 11 or 12, wherein the user input comprises an expression having the one or more conditions associated with the one or more actions to be executed when the one or more conditions are satisfied.

14. The system of claim 9.

10.

11. 12 or 13, wherein the schedule of tasks in the automated workflow includes the existing experiment protocol timing requirements, and wherein the device controller is further configured to: determine a set of new tasks associated with the new workflow: determine one or more new timing requirements associated with the set of new tasks; andgenerate the updated schedule of tasks based on the one or more new timing requirements associated with the set of new tasks and the existing experiment protocol timing requirements.

15. The system of claim 9, 10, 11, 12, 13 or 14, wherein the one or more actions are caused to be executed in the automated experiment handling device by sending a request through an application programming interface (API) gateway connected to an embedded PC within the automated experiment handling device, and wherein the automated experiment handling device comprises a plurality of plate-handling devices, including: a transport system to handle a plurality of plates associated with one or more experiment protocols, an incubator providing an environment for cell culture growth in the plurality of plates, an imager configured to capture periodic images of the plurality' of plates, a liquid handling system providing periodic liquid maintenance on the plurality of plates, a laminar flow system providing sterile air for the plurality of plates outside of the incubator, and one or more component identifiers to uniquely identity' the plurality of plates and one or more inventory supply containers, wherein the device controller is further configured to: send one or more instructions in the request to the embedded PC within the automated experiment handling device, the one or more instructions including the one or more actions to be executed in the automated experiment handling device; receiving a status update from the embedded PC, the status update including a progress indication of the one or more actions; responsive to receiving an indication of completion of the one or more actions, retrieving one or more new images from the embedded PC associated with the one or more actions; storing the one or more new images in a data store; and determining a next task in the updated schedule of tasks in the automated workflow.

16. A computer-implemented method, comprising: providing, by a server, a user interface for display on a user device communicatively coupled to the server, the user interface including a set of data values corresponding to physical attributes of components used in an automated experiment handling device;receiving, at the server from the user device, user input to define an experiment protocol including one or more configuration instructions associated with the components used in the automated experiment handling device and an automated workflow including one or more conditions associated with each action in the automated workflow, the user input received through the user interface; scheduling a first action in the automated workflow through the user interface, the first action having one or more action parameters; evaluating the one or more conditions associated with the first action in the automated workflow; based on the one or more conditions associated with the first action evaluated as being satisfied, causing the first action to be performed at the automated experiment handling device according to the one or more action parameters; and sending a notification to the user device indicating that the first action has been triggered based on the one or more conditions being satisfied.

17. The computer-implemented method of claim 16, wherein the set of data values corresponding to physical attributes of components used in the automated experiment handling device comprises at least one of: a quanti of inventory' supplies required by the experiment protocol; a quantity of inventory supplies available in the automated experiment handling device; a first identifier associated with a plate having a standard number of deep wells; a second identifier associated with a plate having a standard number of shallow wells; a third identifier associated with a plate having a small number of shallow wells; a fourth identifier associated with a plate having a medium number of shallow wells; a fifth identifier associated with a plate having a large number of shallow wells; a plurality of unique well identifiers, each unique well identifier associated with each well in the automated experiment handling device; a plurality of unique plate identifiers, each unique plate identifier associated with each plate in the automated experiment handling device; a media supply identifier associated with a media supply used in cell culture cultivation; a plurality of liquid handling identifiers, each liquid handling identifier associated with a liquid fed in the automated experiment handling device; a plurality of environmental parameters, each environmental parameter describing a unique environmental attribute within the automated experiment handling device; anda plurality of timing parameters, each timing parameter indicating a period of time expected to complete an associated action in the automated experiment handling device.

18. The computer-implemented method of claim 16 or 17, wherein the user input comprises an expression of the one or more conditions, the expression using one or more Boolean operators to describe the one or more conditions.

19. The computer-implemented method of claim 16, 17 or 18, wherein evaluating the one or more conditions comprises: based on the one or more conditions, retrieving one or more image files, each image file associated with a well in a plate in the automated experiment handling device; generating one or more image analysis results using an image processing service; and evaluating the one or more conditions based on the one or more image analysis results.

20. The computer-implement method of claim 16, 17, 18 or 19, further comprising: receiving an indication that a first resource of inventory7supplies needed to complete a current task has been depleted; determining a second resource of inventory supplies that is available in the automated experiment handling device; modifying the current task to use the second resource of inventory supplies; and updating one or more scheduled tasks based on a calculated data value corresponding to a quantity of inventory supplies based on the first resource being depleted.

Images