Computer-implemented method, computer program product, control device and system for controlling chromatography system
Through the deep reinforcement learning algorithm generation control strategy, the problem of many parameters and strong variability in the chromatography process is solved, and the automated control of the chromatography system is realized, the purity and yield of the target components are improved, and the cost is reduced.
Patent Information
- Application Number
- CN202380032272.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-31
- Filing Date
- 2023-05-31
- Publication Date
- 2024-11-22
AI Technical Summary
Due to the many parameters and strong variability during chromatography, the separation of the target components is difficult to control, especially during the protein purification process, and the yield is difficult to predict. In particular, there is a variability problem in the protein A capture step of monoclonal antibodies.
Using machine learning algorithms, especially deep reinforcement learning, through the training generation control strategy of state-action pairs, adjust control parameters such as valve position and pump speed in the chromatographic system, adapt to the variability of the chromatographic system, and realize automatic control of the chromatographic process. .
Improves control accuracy and efficiency of the chromatographic process, reduces costs, and can adapt to the variability of the chromatographic system throughout the process, achieving higher target component purity and yield.
Smart Images

Figure CN119013556A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer implemented methods, computer program products, control devices and systems for controlling a chromatography system. Background Art
[0002] Chromatography is used to separate and purify chemical components in a mixture. For example, in order to separate components (such as chemicals), a mixture of components can be dissolved in a fluid medium (such as a liquid or gas) called a mobile phase. The components in the mobile phase can then be separated by interacting with a material called a stationary phase, which is usually fixed on a chromatographic bed (such as a column, capillary, plate or sheet, etc.). Since different components in a mixture tend to have different affinities for the stationary phase, different components are usually kept in the chromatographic bed for different lengths of time. Therefore, different components of a mixture dissolved in the mobile phase can pass through the chromatographic bed at different speeds, which allows separation of components.
[0003] For example, in resin liquid chromatography, a solid phase (as a stationary phase) can be applied to the inner surface of a porous resin particle that is usually spherical in shape. The particles prepared in this way can be filled in a cylindrical column to be able to accommodate a liquid mobile phase. In the initial stage of the separation process, the mobile phase can be injected or loaded into the column. When the mobile phase flows through the column, the component with affinity for the solid phase will be transferred at a lower speed. This can allow highly concentrated components to be collected at the column outlet.
[0004] Chromatographic processes can be used to purify a target substance, such as a specific protein. For example, a monoclonal antibody can be captured using protein A in a resin binding matrix using a bind-elute chromatography process.
[0005] In a chromatographic process, many factors and / or parameters can affect the resulting yield and / or concentration of a target component separated from other components in a mixture. Therefore, it is difficult to control the chromatographic process for obtaining the desired results.
[0006] In order to determine the parameter set that can lead to the desired results of the chromatographic process, physical experiments of the chromatographic process can be performed. However, physical experiments can be time-consuming and / or expensive. For a specific example, it may be too expensive to purify proteins using affinity chromatography due to many factors, including the high cost of raw materials. For example, monoclonal antibodies (mAbs) are a class of drugs in which mammalian cells are genetically engineered to produce antibody proteins that catalyze the patient's immune system. mAbs include the most common biopharmaceutical product categories and are also the most expensive products to manufacture. A particularly expensive step in the process can be the so-called protein A capture step. In protein A affinity chromatography, mAbs can be separated by binding to solid phase receptors. In addition, many parameters and multiple input components caused by interactions in the production chain can make the output difficult to predict. In addition, one of the main challenges of the capture step may be variability in the resin column. Variability can come from manufacturing variability and changes in chromatographic capacity over time due to use.
[0007] As an alternative or in addition to physical experiments, in some cases simulations of chromatographic processes can be performed. Summary of the invention
[0008] According to one aspect, the problem relates to facilitating the control of a chromatography system with improved performance. The problem is solved by the features disclosed in the independent claims. Further exemplary embodiments are defined by the dependent claims.
[0009] According to one aspect, a computer-implemented method for controlling a chromatography system is provided, the chromatography system being configured to physically perform and / or simulate a chromatography process. The method comprises:
[0010] obtaining from the chromatographic system a current state of the chromatographic system, the current state comprising one or more values of one or more state parameters, the one or more state parameters comprising one or more amounts of one or more substances present in the chromatographic system;
[0011] determining one or more values of one or more control parameters according to a policy configured to map the current state to corresponding actions representing one or more values of one or more control parameters for the chromatography system, the one or more control parameters comprising at least: a position of a valve included in the chromatography system and / or a pump speed of a pump included in the chromatography system; and
[0012] controlling the chromatography system using the one or more determined values of the one or more control parameters,
[0013] wherein the strategy is generated according to a machine learning algorithm trained using state-action pairs; and
[0014] Wherein, the state-action pairs are obtained at least in part by physically performing and / or simulating the chromatographic process by the chromatographic system, each of the state-action pairs comprising:
[0015] The state of the chromatography system (S t ), comprising one or more values of said one or more state parameters at a specific point in time (t); and
[0016] Action t ), including one or more values of the one or more control parameters, the chromatography system responds to the state (S t ), using the action (A t ) is controlled by one or more values in ).
[0017] In some exemplary embodiments, the chromatographic system may include a chromatographic device configured to physically perform a chromatographic process. For example, the chromatographic device may include one or more chromatographic beds, containers for different types of media used in the chromatographic process, connectors between containers and / or chromatographic beds, pumps that allow media to flow through the chromatographic system, and / or control valves that control fluid flow. Therefore, in some exemplary embodiments, "one or more substances present in the chromatographic system" may include one or more connectors between containers and / or one or more substances that flow through one or more chromatographic beds. Alternatively or in addition, "one or more substances present in the chromatographic system" may include one or more substances inside one or more containers.
[0018] In the present disclosure, the term "chromatographic bed" may be understood as any of the various configurations containing a stationary phase for a chromatographic process. Examples of chromatographic beds may include, but are not limited to, columns, capillaries, plates, sheets, and the like.
[0019] In the case where the chromatography system comprises a plurality of chromatography beds, the chromatography beds may be connected in series. For example, in the case where a plurality of chromatography beds are connected in series, the chromatography system may perform periodic countercurrent chromatography.
[0020] The chromatographic system may also include a processor and a storage medium. The processor included in the chromatographic system may be configured to collect sensor information from different types of sensors provided on the chromatographic device and store the collected sensor information in a storage medium included in the chromatographic system. The storage medium may be configured to store the collected sensor information and / or values of other parameters of the chromatographic system.
[0021] As an alternative or supplement to a chromatographic device configured to physically perform a chromatographic process, the chromatographic system may include a simulation system configured to simulate a chromatographic process. The simulation system may be implemented by a processor and a storage medium included in the chromatographic system, wherein the processor is configured to perform operations and / or calculations necessary for the simulated chromatographic process, and the storage medium stores information necessary for the simulated chromatographic process.
[0022] Furthermore, in some exemplary embodiments, the chromatography system may be configured to perform multiple physical and / or simulated chromatography processes in parallel and / or in series.
[0023] In various aspects and embodiments described herein, the chromatography system can be configured to communicate its internal state (e.g., represented by one or more state parameters) to another device (e.g., a control device) at a given point in time. For example, the communication of the internal state can be performed by a processor included in the chromatography system. In addition, in various aspects and embodiments described herein, the chromatography system can be configured to accept program control to change the value of one or more control parameters.
[0024] In the present disclosure, one or more status parameters may be parameters representing the condition of the chromatography system.
[0025] In the case where the chromatographic system physically performs the chromatographic process, one or more values of one or more state parameters can be obtained by measuring these values with sensors corresponding to the state parameters. The sensor can be set at an appropriate position in the chromatographic system. For example, in the case where there are a large number of substances in the chromatographic system, an ultraviolet (UV) sensor (e.g., UV chromatogram), an infrared (IR) sensor, a mass spectrometer sensor, a conductivity sensor (e.g., for measuring ion concentration), a scattered light signal and / or a Raman spectrum can be used to measure the value. Additionally or alternatively, in some cases, a software sensor configured to estimate the value of the state parameter based on the value measured by the sensor can be used. The estimated value can be derived from physical principles, data-driven methods, or a combination of the two.
[0026] In case the chromatography system simulates a chromatography process, one or more values of one or more state parameters may be obtained by calculations performed on the simulation of the chromatography process.
[0027] In various aspects and embodiments described herein, the control of the chromatography system with one or more determined values of one or more control parameters can be performed by generating one or more control signals to instruct the chromatography system to set one or more control parameters to one or more determined values. Then, the generated control signal can be transmitted to the chromatography system. The chromatography system can set one or more control parameters to one or more determined values after one or more control signals. For example, in the case of the position of a control valve, the control signal can include a signal indicating that the valve sets the valve position to a desired value. In an exemplary embodiment in which the chromatography system includes a chromatography device for physically performing a chromatography process, for example, the valve can be a control valve configured to change the size of a flow channel according to a control signal. In an exemplary embodiment in which the chromatography system includes a simulation system, the valve can be a virtual valve, and operations and / or calculations involving the virtual valve can be performed according to the position of the valve indicated by the control signal, for example.
[0028] Any of the various aspects and embodiments described herein can provide a data-driven method for controlling a chromatography system that can produce a control strategy that can adapt to variability in the chromatography system (e.g., in one or more chromatography beds (e.g., one or more columns) and thereby achieve improved performance.
[0029] More specifically, the strategy for controlling the chromatography system in any of the various aspects and embodiments described herein can allow for control of the chromatography process over the entire duration of the process, including the timing when the flow of substances into specific components of the chromatography system (e.g., chromatography beds and / or different containers) is switched, because the control parameters include at least the position of the valve, and the state parameters include the amount of substances present in the chromatography system. Thus, the control performed in any of the various aspects and embodiments described herein can be free from the constraints of a fixed strategy, such as derived from experimentally generated breakthrough curves of the chromatography process, and can accommodate variability between different chromatography beds and / or during a chromatography run.
[0030] In some exemplary embodiments, the method according to the above aspects may further include:
[0031] receiving the state-action pair; and
[0032] The policy is generated according to the machine learning algorithm using the received state-action pairs.
[0033] The method according to the above aspect may further include:
[0034] The policy is updated using the current state and a corresponding action including the one or more determined values of the one or more control parameters.
[0035] In various aspects and embodiments described herein, the one or more amounts of the one or more substances present in the chromatography system include one or more of:
[0036] one or more amounts of one or more substances flowing into one or more chromatographic beds included in the chromatographic system;
[0037] one or more amounts of one or more substances eluting from the one or more chromatographic beds;
[0038] one or more amounts of one or more substances within the one or more chromatographic beds;
[0039] one or more amounts of one or more substances within one or more of the containers included in the chromatography system,
[0040] Therein, the one or more state parameters may include at least one parameter based on two or more of the quantities listed above.
[0041] "At least one parameter based on two or more of the quantities listed above" may include, for example, the ratio between two different quantities among those listed above, the sum of different quantities among those listed above, the difference between different quantities among those listed above, etc.
[0042] In some exemplary embodiments, one or more amounts of one or more substances within one or more chromatographic beds may be derived from amounts of one or more substances flowing into and out of the one or more chromatographic beds.
[0043] In various aspects and embodiments described herein, the one or more control parameters may also include one or more flow rates of one or more media flowing into and / or out of one or more of the following:
[0044] at least one of the one or more chromatographic beds of the chromatographic system;
[0045] at least one of the containers included in said chromatography system;
[0046] At least one of the one or more flow controllers included in the chromatography system.
[0047] In addition, the one or more control parameters may also include one or more of the following items:
[0048] the temperature in the chromatography system;
[0049] the pH of the mobile phase in the chromatography system;
[0050] The salinity of the mobile phase in the chromatography system.
[0051] In the present disclosure, a "flow controller" included in a chromatography system may be understood as a component configured to control the flow rate of fluids from different fluid sources (e.g., containers) into a chromatography bed. For example, each chromatography bed of a chromatography system may have a corresponding flow controller to control the flow of fluids into the chromatography bed.
[0052] In the case where the one or more control parameters include one or more flow rates of one or more media flowing into and / or out of at least one of the one or more chromatographic beds of the chromatographic system, at least one of the containers included in the chromatographic system, and / or at least one of the one or more flow controllers included in the chromatographic system, a control signal can be generated to control various pumps and / or valves for regulating the flow rates of the various media, and the control signal can be transmitted to the chromatographic system. In the case where the chromatographic system physically performs the chromatographic process, the controlled pumps and / or valves can be physical pumps and / or valves, and in the case where the chromatographic system simulates the chromatographic process, the controlled pumps and / or valves can be virtual pumps and / or valves.
[0053] Furthermore, where one or more control parameters include temperature, pH and / or salinity, a control signal may be generated to control various components in the chromatography system for regulating temperature, pH and / or salinity and transmitted to the chromatography system.
[0054] In addition, in various aspects and embodiments described herein, the one or more state parameters may also include one or more of the following items:
[0055] the temperature at a specified point of the chromatographic system;
[0056] the pH of the medium in the specified portion of the chromatography system;
[0057] one or more parameters related to specifications of the one or more chromatographic beds, one or more vessels included in the chromatographic system, and / or one or more connections between the vessels;
[0058] one or more maximum flow rates of one or more media into the one or more chromatographic beds;
[0059] one or more upstream parameters;
[0060] one or more parameters related to the feed medium, wash medium and / or elution medium used in the chromatographic process;
[0061] Conductivity:
[0062] absorption of effluent;
[0063] Target protein content;
[0064] concentration of coeluting contaminants;
[0065] Product concentration;
[0066] purity;
[0067] Yield.
[0068] The "specified point" and "specified portion" of the chromatography system as described above may be any point and portion, respectively, within the chromatography system. For example, the "temperature" at a "specified point" of the chromatography system may be the temperature within one of the containers (e.g., a container for feed medium) included in the chromatography system. Furthermore, for example, the "pH" of the medium in a "specified portion" of the chromatography system may be the pH at the outlet of the chromatography bed (e.g., the column outlet), the concentration in one of the containers (e.g., a container for target product), etc.
[0069] Examples of "one or more parameters related to the specifications of the one or more chromatographic beds, one or more containers included in the chromatographic system and / or one or more connectors between the containers" as described above may include, but are not limited to, volume, size (e.g., the inner diameter of a tube), etc.
[0070] Examples of "one or more parameters related to the feed medium, wash medium and / or elution medium used in the chromatography process" as described above may include, but are not limited to, product titer, flow rate, temperature, pH, etc.
[0071] In various aspects and embodiments described herein, the machine learning algorithm may include one, or a combination of two or more of the following:
[0072] Reinforcement learning;
[0073] Deep reinforcement learning;
[0074] Supervised learning;
[0075] Semi-supervised learning;
[0076] Self-supervised learning;
[0077] Imitation learning;
[0078] Transfer learning.
[0079] In the present disclosure, "reinforcement learning" may be understood as a machine learning algorithm involving training an intelligent agent (hereinafter also referred to as an "agent") to take a series of actions in an environment to maximize a cumulative reward. In "reinforcement learning", an intelligent agent may learn a policy by trial and error that maps the state of the environment to the action to be taken by the agent.
[0080] In this disclosure, "deep reinforcement learning" may be understood as a machine learning algorithm that combines reinforcement learning and deep learning. "Deep learning" may use one or more neural networks to obtain representations (e.g., features) of raw, input data. In deep reinforcement learning, the strategy of mapping states to actions and / or other learned functions may be implemented by a neural network.
[0081] In the present disclosure, "supervised learning" may be understood as a machine learning algorithm that involves learning a function that maps an input to an output based on exemplary input-output pairs. Each exemplary input-output pair may include a specific input and an expected output for that specific input. The exemplary input-output pairs may be referred to as labeled training data.
[0082] In the present disclosure, "semi-supervised learning" may be understood as a machine learning algorithm that uses labeled and unlabeled training data (e.g., no information about what the "expected" output is for a certain input) to learn a function that maps an input to an output. Typically, in semi-supervised learning, the amount of labeled training data is less than the amount of unlabeled training data.
[0083] In this disclosure, "self-supervised learning" may be understood as a machine learning algorithm that can learn from unlabeled data. In some examples, self-supervised learning may involve a neural network.
[0084] In the present disclosure, "imitation learning" may be understood as a machine learning algorithm that enables retrieval of a policy that mimics the state-action pairs presented that may be defined by an expert (e.g., an expert policy). In some implementations, "imitation learning" may be used to train an agent in reinforcement learning.
[0085] In this disclosure, “transfer learning” may be understood as a machine learning algorithm that applies knowledge acquired while solving a problem to solve a different but similar problem.
[0086] In an exemplary combination of machine learning algorithms as described above, to train an agent in reinforcement learning, imitation learning can be used to pre-train a neural network representing a policy, and then transfer learning is used.
[0087] In another exemplary combination of the machine learning algorithms described above, imitation learning may be introduced in semi-supervised learning in order to train an agent in reinforcement learning.
[0088] In some exemplary embodiments, the machine learning algorithm may include reinforcement learning or deep reinforcement learning. In such exemplary embodiments, the rewards in reinforcement learning or deep reinforcement learning may be calculated using one or more of the following:
[0089] at least one of the one or more values of the one or more state parameters;
[0090] a value representing the flow rate of a target compound into the one or more chromatographic beds of the chromatographic system;
[0091] a value representing the flow rate of a target compound out of the one or more chromatographic beds of the chromatographic system;
[0092] a value representing an amount of a target compound for the chromatography process that is contained in or flows into a product container contained in the chromatography system;
[0093] a value representing an amount of a target compound included in or flowing into a waste container included in the chromatography system;
[0094] a value representing an amount of spent medium in or flowing into the product container, the spent medium including materials other than the target compound;
[0095] A value representing the amount of the spent media in or flowing into the waste container.
[0096] In cases where values representing the flow rate of a target compound into and out of one or more chromatographic beds of a chromatographic system are used to calculate the reward, the amount of the target compound currently in the one or more chromatographic beds can be approximated and incorporated into the reward to punish the reinforcement learning or deep reinforcement learning agent for retaining too much target compound in the one or more chromatographic beds at the end of the chromatographic process.
[0097] In the present disclosure, "spent media" can indicate waste products from the production of a target molecule. For example, in the case of isolating a certain type of protein (e.g., mAb-type protein), the target protein can be cultured in mammalian egg cells. In downstream processing, cells can initially be grown in some containers. This may become the feed medium when the cells are broken down into their components at the end of their life cycle. "Spent media" can represent all remaining products of the cell culture process that do not include elution buffer or target protein. Spent media can include, for example, non-target proteins, depleted nutrients, metabolites, etc.
[0098] Thus, the reward calculated as described above can focus on the overall efficiency of the chromatography system, e.g., the amount of target product and / or spent media in or flowing into the product container and / or waste container, rather than relying on properties of the chromatography system, such as the binding capacity of the column, which can vary as the chromatography process proceeds. Thus, the reward calculated as described above can enable the generation of a strategy that allows the variability of the chromatography system to be accommodated when controlling the chromatography system.
[0099] Furthermore, in exemplary embodiments where the machine learning algorithm includes reinforcement learning or deep reinforcement learning, the following reward function may be used to calculate the reward:
[0100] R=(M×Δtc+Δsm)×pure+cp×done,
[0101] Where M is a factor that indicates how much material other than the target compound is used in the chromatography process compared to the total amount of the target compound fed into the chromatography system;
[0102] Where Δtc is calculated by the following formula:
[0103] Δtc=tc prod -tc waste ,
[0104] where tc prod represents the amount of target compound in the product container, and tc waste Indicates the amount of target compound in the waste container;
[0105] The Δsm is calculated by the following formula:
[0106] Δsm=sm waste -sm prod ,
[0107] Among them, sm waste Indicates the amount of spent media in the waste container and sm prod Indicates the amount of spent media in the product container;
[0108] Where pure is the purity term calculated by the following formula:
[0109]
[0110] Where x is the minimum value of the purity of the target compound in the product container;
[0111] Where p is the penalty term determined by:
[0112]
[0113] where tc col is the amount of the target compound within the chromatographic bed, and t is a specified threshold value;
[0114] where c is the clean term calculated by:
[0115] c=max(tc prod -sm prod , 0); and
[0116] where done is set to 1 in the case of a terminal state of the chromatographic process and to 0 otherwise.
[0117] Furthermore, in example embodiments where the machine learning algorithm includes deep reinforcement learning, the deep reinforcement learning may involve using the following actor-critic approach:
[0118] an actor network comprising a first neural network to be trained to represent the policy, the first neural network configured to take a state as input and an action as output; and
[0119] An evaluator network includes a second neural network trained to estimate rewards obtained from state-action pairs, the evaluator network being used to train the actor network to output actions that produce high rewards in response to states input to the actor network.
[0120] In this disclosure, the term "neural network" may refer to an artificial neural network.
[0121] Furthermore, in some exemplary embodiments, the machine learning algorithm may include supervised learning or semi-supervised learning, and at least a portion of the state-action pairs may be defined by an expert in the chromatographic process.
[0122] In some exemplary embodiments, a chromatography system may include:
[0123] a chromatography device configured to physically perform a chromatography process; and
[0124] a simulation system configured to simulate the chromatography process and implemented by a processor and a storage medium,
[0125] Therein, the controlling of the chromatography system may include controlling a chromatography device included in the chromatography system using one or more determined values of the one or more control parameters.
[0126] According to another aspect, which may be used in particular in conjunction with any computer-implemented method for controlling a chromatography system as described herein, there is provided a computer-implemented method for configuring a control device for controlling a chromatography system, the chromatography system being configured to physically perform and / or simulate a chromatography process. The method comprises:
[0127] Receiving state-action pairs obtained at least in part by physically performing and / or simulating a chromatography process with a chromatography system, each state-action pair comprising:
[0128] The state of the chromatography system (S t ) comprises one or more values of one or more state parameters at a specific point in time (t), the one or more state parameters comprising one or more amounts of one or more substances present in the chromatographic system; and
[0129] Action t ), including at least one or more values of one or more control parameters for the chromatography system, in response to the state (S t ), use the included action (A t ) to control the chromatography system, wherein the one or more control parameters include at least a position of a valve included in the chromatography system and / or a pump speed of a pump included in the chromatography system;
[0130] generating, based on a machine learning algorithm and using the received state-action pairs, a policy that maps a current state including one or more values of the one or more state parameters to a corresponding action including one or more values of the one or more control parameters; and
[0131] The generated policy is stored in a storage medium included in the control device.
[0132] According to another aspect, a computer program product is provided, comprising computer-readable instructions, which, when loaded and executed on a computer, enable the computer to execute the method according to any one of the above aspects and various embodiments thereof.
[0133] According to yet another aspect, a control device for controlling a chromatography system is provided, wherein the chromatography system is configured to perform and / or simulate a chromatography process. The control device comprises:
[0134] A processor configured to execute the method according to any one of the above aspects and various embodiments thereof; and
[0135] A storage medium is configured to store the policy.
[0136] According to another aspect, a system is provided. The system comprises:
[0137] a chromatography system configured to perform and / or simulate a chromatography process; and
[0138] The control device according to the above aspect is connected to a chromatography system.
[0139] Various aspects and embodiments described herein can be applied to chromatography systems configured to physically perform and / or simulate different types of chromatography processes. Examples of different types of chromatography processes can include, but are not limited to, resin liquid chromatography processes, high pressure liquid chromatography processes, gas chromatography processes, ion exchange chromatography processes, affinity chromatography processes, membrane chromatography processes, continuous chromatography processes, etc.
[0140] The subject matter described in this application may be implemented as a method or system, possibly in the form of one or more computer program products. The subject matter described in this application may be implemented in a data signal or on a machine-readable medium, wherein the medium is included in one or more information carriers, such as a CD-ROM, a DVD-ROM, a semiconductor memory, or a hard disk. Such a computer program product may cause a data processing device to perform one or more operations described in this application.
[0141] In addition, the subject matter described in this application can also be implemented as a system including a processor and a memory connected to the processor. The memory can encode one or more programs so that the processor performs one or more methods described in this application. In some examples, the system can be a general-purpose computer system. In other examples, the system can be a special-purpose computer system including an embedded system. BRIEF DESCRIPTION OF THE DRAWINGS
[0142] The details of one or more implementations are described in the following exemplary drawings and descriptions. Other features will be apparent from the specification, drawings, and claims. However, it should be understood that even if the embodiments are described separately, individual features of different embodiments may be combined with other embodiments.
[0143] Figure 1 A functional block diagram of an exemplary system according to the present disclosure is shown.
[0144] Figure 2 A schematic diagram illustrating an exemplary simulation environment for simulating a chromatography process is shown.
[0145] Figure 3 A schematic diagram illustrating a reinforcement learning agent interacting with an environment is shown.
[0146] Figure 4 Schematic diagram showing agents and environments for deep reinforcement learning involving an actor-critic approach.
[0147] Figure 5 A schematic diagram illustrating multi-environment interactions of a reinforcement learning agent is shown.
[0148] Figure 6 Shown by Figure 1 A flow chart of an exemplary process performed by a control device is shown.
[0149] Figure 7 Shown as Figure 1 A flow chart of an exemplary process for configuring a control device strategy is shown.
[0150] Figure 8 An exemplary benchmark artifact strategy for separation of mAb proteins and for controlling spent media of an exemplary chromatography system is shown.
[0151] Fig. 9 Shown is the control process of an exemplary chromatography system, as performed by a reinforcement learning agent after training 657 events in an exemplary use case.
[0152] Fig.10 Shown is the control process of an exemplary chromatography system, performed by a reinforcement learning agent after training 10 events in an exemplary use case.
[0153] Fig.11 Shown is the reward in reinforcement learning obtained from a term related to the purity of the target compound obtained during chromatography.
[0154] Fig.12 The control process of a simulated chromatography device by a reinforcement learning agent in another exemplary use case is shown.
[0155] Fig.13 The control process of a simulated chromatography device by a reinforcement learning agent in yet another exemplary use case is shown.
[0156] Fig.14 Another exemplary simulated benchmark strategy for separation of mAb proteins and for controlling spent media of an exemplary chromatography system is shown.
[0157] Fig.15 A schematic diagram illustrating another exemplary simulation environment for simulating a chromatography process is shown.
[0158] Fig.16An exemplary hardware configuration of a computer that can be used to implement at least part of a system according to the present disclosure is shown. DETAILED DESCRIPTION
[0159] In the text below, a detailed description of the examples will be given with reference to the accompanying drawings. It should be understood that various modifications may be made to the embodiments. In particular, one or more elements of an example may be combined and used in other examples to form new examples.
[0160] System Configuration
[0161] Figure 1 A functional block diagram of an exemplary system according to the present disclosure is shown. Figure 1 The exemplary system shown includes a chromatography system 10 and a control device 20.
[0162] like Figure 1 As shown, the chromatography system 10 may include a chromatography device 102 , a processor 104 , and / or a storage medium 104 .
[0163] Chromatographic device 102 may be a device configured to physically perform a chromatographic process. Figure 1 102 may include a chromatographic bed, containers for different types of media used in the chromatographic process, connectors between the containers, pumps for passing the media through the chromatographic system, and / or control valves for controlling the flow of fluids. The chromatographic bed may be a configuration containing a stationary phase for the chromatographic process, and may include columns, capillaries, plates, sheets, etc. In some exemplary embodiments, the chromatographic device 10 may include multiple chromatographic beds, such as for multi-column chromatography. For example, in the case where the chromatographic device 10 includes multiple columns as chromatographic beds, the chromatographic device 10 may be used to perform periodic countercurrent chromatography.
[0164] In some exemplary embodiments, the processor 104 of the chromatography system 10 can be configured to simulate a chromatography process. For example, the simulation environment can be implemented using the processor 104 and the storage medium 106. The processor 104 can be configured to perform operations and / or calculations necessary to simulate the chromatography process. The storage medium 106 of the chromatography system 10 can store information necessary for the processor 104 to simulate the chromatography process. Figure 2 Describe the details of the simulation environment, Figure 2 A specific example of a simulation environment is shown.
[0165] In addition, the processor 104 can be configured to collect the internal state of the chromatography system 10 at a given point in time and transmit the collected information to another device, such as the control device 20. The internal state can be represented by one or more values of one or more state parameters. The one or more state parameters can represent the condition of the chromatography system 10.
[0166] For example, the one or more state parameters may include one or more amounts of one or more substances present in the chromatography system 10. For example, the one or more amounts of the one or more substances present in the chromatography system 10 may include: one or more amounts of the one or more substances flowing into one or more chromatography beds included in the chromatography system 10; one or more amounts of the one or more substances flowing out of the one or more chromatography beds; one or more amounts of the one or more substances within one or more chromatography beds and / or one or more amounts of the one or more substances within one or more of the containers included in the chromatography system 10. Furthermore, in some exemplary embodiments, the one or more state parameters may include at least one parameter based on two or more of the above amounts (e.g., ratios, differences, totals, etc.).
[0167] Other examples of one or more state parameters may include, but are not limited to: temperature at a specified point in the chromatography system; pH of the medium in a particular portion of the chromatography system; one or more parameters related to the specifications of one or more chromatography beds, one or more containers included in the chromatography system and / or one or more connections between containers; one or more maximum flow rates of one or more media flowing into one or more chromatography beds; one or more upstream parameters; one or more parameters related to the feed medium, wash medium and / or elution medium used in the chromatography process; conductivity; absorbance of the effluent; target protein content; concentration of co-eluting contaminants; product concentration; purity; yield, etc.
[0168] The processor 104 can obtain at least some values of one or more state parameters from one or more sensors disposed on the chromatographic device 102. Sensors corresponding to the state parameters of interest can be disposed at appropriate locations within the chromatographic device 102. For example, in the case where the state parameter includes the amount of a substance present in the chromatographic system, an ultraviolet (UV) sensor (e.g., UV chromatogram), an infrared (IR) sensor, a mass spectrometer sensor, a conductivity sensor (e.g., for measuring ion concentration), a scattered light signal, and / or a Raman spectrum can be used to measure the value. In addition, for example, in the case where the state parameter includes temperature and / or pH, a temperature sensor and / or a pH sensor can be used. Additionally or alternatively, in some cases, a software sensor configured to estimate the value of the state parameter based on the value measured by the sensor can be used. The estimated value can be derived from physical principles, data-driven methods, or a combination of the two. The software sensor can provide the estimated value to the processor 104.
[0169] In some cases, some values of one or more state parameters may be provided to the processor 104 via user input. For example, where the state parameters include specifications of components of the chromatographic apparatus (e.g., volumes of vessels, connections between vessels and / or chromatographic beds), the values of the state parameters may be input to the processor 104 by a user using an input device (not shown).
[0170] For a simulated chromatographic process, the processor 104 may obtain values of one or more state parameters during the simulation of the chromatographic process, for example, through calculations performed with respect to the simulation of the chromatographic process.
[0171] In addition, the chromatography system 10 may be configured to accept program control to change the value of one or more control parameters of the chromatography process. The control device 20 may perform the control.
[0172] from Figure 1 It can be seen that the control device 20 may include a processor 202 and a storage medium 204. The control device 20 may be connected to the chromatography system 10 via a wired and / or wireless communication network.
[0173] The processor 202 of the control device 20 may be configured to obtain the current state of the chromatography system 10 from the chromatography system 10. For example, the processor 202 of the control device 20 may receive the current state from the processor 104 of the chromatography system 10. The current state may include one or more values of one or more state parameters as described above.
[0174] The processor 202 may also be configured to determine one or more values of the one or more control parameters according to a strategy configured to map a current state to a corresponding action representing one or more values of the one or more control parameters for the chromatography system 10. The one or more control parameters may include at least: a position of a valve included in the chromatography system 10 and / or a pump speed of a pump included in the chromatography system 10. Furthermore, for example, the one or more control parameters may include one or more flow rates of one or more media flowing into and / or out of one or more of: at least one of the one or more chromatography beds of the chromatography system 10; at least one of the containers included in the chromatography system 10; at least one of the one or more flow controllers included in the chromatography system 10. Furthermore, the one or more control parameters may include a temperature in the chromatography system, a pH of a mobile phase in the chromatography system 10, and / or a salinity of the mobile phase.
[0175] The strategy may be generated based on a machine learning algorithm trained using state-action pairs. The state-action pairs may be obtained at least in part by physically performing and / or simulating a chromatographic process by the chromatography system 10. Each of the state-action pairs may include:
[0176] The state (S) of the chromatography system 10 t ), comprising one or more values of one or more state parameters at a particular point in time (t); and
[0177] Action t ), including one or more values of the one or more control parameters, in response to the state (S t ), using the action (A t ) to control the chromatography system 10.
[0178] The details of the machine learning algorithm used to generate the policy will be referred to below. Figures 3 to 5 Give a description.
[0179] In addition, the processor 202 can be configured to use one or more determined values of the one or more control parameters to control the chromatography system 10. For example, the processor 202 can be configured to generate one or more control signals to instruct the chromatography system 10 to set the one or more control parameters to one or more values determined according to the strategy and the current state obtained from the chromatography system 10. The generated control signal can be transmitted to the chromatography system 10, and the chromatography system 10 can set the one or more control parameters to the one or more determined values after the one or more control signals.
[0180] For example, in the case of controlling the position of a valve in the chromatography system 10, the control signal may include a signal instructing the valve to set the valve position to a desired value. For example, in the case of the chromatography device 102 physically performing a chromatography process, the valve may be a control valve configured to change the size of a flow channel according to the control signal. In the case of the processor 104 simulating a chromatography process, the valve may be a virtual valve, and operations and / or calculations involving the virtual valve may be performed according to the position of the valve indicated by the control signal, for example.
[0181] In addition, for example, in the case of controlling one or more flow rates of one or more media used in the chromatographic process, the control signal may include a signal indicating the various pumps and / or valves used to adjust the various media. Similarly, in order to control the position of the above-mentioned valves, the pumps and / or valves may be physically controllable pumps and / or valves included in the chromatographic device 10 in the case of physically performing the chromatographic process. In the case of simulating the chromatographic process, the generated control signal may be used to control a virtual pump and / or valve in the simulated environment.
[0182] In addition, for example, in the case of controlling temperature, pH and / or salinity, the control signal may include instructions to various components in the chromatography system for adjusting temperature, pH and / or salinity. In the case of physically performing the chromatography process, the various components may be physical components included in the chromatography device 10. In the case of simulating the chromatography process, the various components may be virtual components in the simulation environment.
[0183] The storage medium 204 of the control device 20 may store information necessary for the processor 202 to control the chromatography system 10. For example, the storage medium 204 may store a strategy for determining one or more values of one or more control parameters in response to receiving a current state from the chromatography system 10. In addition, for example, the storage medium 204 may store information used to generate a strategy in a machine learning algorithm.
[0184] Thus, the control device 20 may provide an automated controller for the chromatography system 10 and may learn how to operate itself through interaction with real and / or simulated data.
[0185] It should be noted that Figure 1 The configurations of the chromatography system 10 and the control device 20 shown are merely examples used in some exemplary embodiments. In other exemplary embodiments, the chromatography system 10 and / or the control device 20 may have different configurations.
[0186] For example, despite Figure 1 The chromatography system 10 and the control device 20 are shown as separate devices, but the control device 20 may be implemented as a part of the chromatography system 10, in other words, the control device 20 may be integrated within the chromatography system 10. In this case, the processor 104 and the storage medium 106 included in the chromatography system 10 may be configured to implement the functions of the processor 202 and the storage medium 204 of the control device 20 as described above.
[0187] Furthermore, in some exemplary embodiments, the chromatography system 10 may include the processor 104 and the storage medium 106 without the chromatography device 102. In such exemplary embodiments, the chromatography system 10 may utilize the processor 104 and the storage medium 106 to perform a simulation of a chromatography process, but without physically performing the chromatography process.
[0188] In some other exemplary embodiments, the processor 104 of the chromatography system 10 is not configured to perform a simulation of a chromatography process. In such exemplary embodiments, the chromatography system 10 may physically perform a chromatography process via the chromatography device 102, but does not simulate the chromatography process.
[0189] Example Simulation Environment
[0190] Figure 2 A schematic diagram illustrating an exemplary simulation environment implemented by the processor 104 of the chromatography system 10 for simulating a chromatography process is shown. Figure 2 In the specific example shown, the simulation environment 50 provides a chromatography system simulation 500, which includes modules representing: a column simulation 502, a feed container 510, an elution container 512, a wash container 514, a flow controller 516, an output valve 530, a product container 540, and a waste container 542. The column simulation 502 includes a model of a column 520 as a chromatography bed.
[0191] Figure 2 The exemplary simulation environment 50 shown describes a single column system for the binding-elution process of a simulated liquid chromatography process. The physical system (e.g., chromatographic device 102) recreated in the exemplary simulation environment 50 may include a feed container 510 to accommodate untreated media, an elution container 512 to accommodate elution buffer, and a wash container 514 to accommodate a wash solution for flushing impurities and spent media from a column 520. Containers 510, 512, and 514 may be connected to a flow controller 516, which is configured to individually control the flow from containers 510, 512, and 514 into the column 520. The flow controller 516 may be configured to provide a control signal to a controllable pump and / or valve for regulating the corresponding flow. For example, the flow controller 516 may obtain a target value for the corresponding flow, generate an appropriate control signal for the controllable pump and / or valve to obtain the target value, and provide the generated control signal to the controllable pump and / or valve. The column inlet can be fed by a common volume flow (e.g., referred to as a "column inlet flow") of one or more of the mixture species included in one or more of the feed container 510, the elution container 512, and the wash container 514. The flow controller 516 can also be configured to join the output of a separate container to the column inlet flow by controlling a controllable pump and / or valve. At the other end of the column, the outlet can be fed into an output valve 530, which can siphon the treated medium into a product container 540 or a waste container 542, which is designed to accommodate purified medium and waste, respectively.
[0192] In order to represent a physical system (e.g., a chromatographic device 102), a set of differential equations describing the flow between each container, valve, and column can be combined in the chromatographic system simulation 500 in the exemplary simulation environment 50. The column itself can be modeled by the implementation of a mechanism model called a general rate model (e.g., see Guiochon et al., "Fundamentals of Preparative and Nonlinear Chromatography", Second Edition, Elsevier, February 10, 2006; Leweke et al., "Chromatography Analysis and Design Toolkit (CADET)", Computers and Chemical Engineering, Volume 113 (2018), 274-294; See also URL: https: / / cadet.github.io / v4.3.0 / modelling / unit_operations / general_rate_model.html). The model of the column can be composed of a set of partial differential equations. The equation can describe the interstitial mobile phase, the stagnant mobile phase in the particles, and the solid phase concentration as a function of space and time. Various parameters can be set to reflect the specific characteristics of the column, particles and resin. This group of equations can be numerically solved using a technique known as orthogonal configuration, which belongs to a series of methods known as weighted residual methods (see, for example, Young, "Orthogonal collocation revisited", Computer Methods in Applied Mechanics and Engineering, Vol. 345, March 1, 2019, pp. 1033 to 1076, https: / / doi.org / 10.1016 / j.cma.2018.10.019). The weighted residual method also includes the Galerkin method (Id.). In the orthogonal configuration, the process can be defined by selecting points for discretization. These points can be obtained by finding the root points of the selected Jacobi polynomial (such as Legendre polynomial). The process can discretize the spatial domain of the equation and convert them into a system of ordinary differential equations.The column equations and the general system equations can be combined and solved in the time domain using the Euler method discretization technique (e.g., see Wikipedia, “Backward Euler method,” https: / / en.wikipedia.org / wiki / Backward_Euler_method; Wikipedia, “Euler method,” https: / / en.wikipedia.org / wiki / Euler_method).
[0193] The simulated chromatography system can simulate the control surfaces of a physical system (e.g., chromatography device 102). This can allow an operator to set the volume flow from feed container 510, elution container 512, and wash container 514 using flow controller 516. The simulator can calculate the flow into column 520 and the average flow of phase transfer through column 520. The inlet concentration can be defined by the container concentration and the flow from flow controller 516 and used for column simulation 502. Column simulation 502 can continuously calculate the concentration of components in column 520, and when the solution reaches the column outlet, output valve 530 can allow the operator to direct the flow to product container 540 or waste container 542.
[0194] In the chromatography system simulation 500, observations of the internal state of the chromatography system can be recorded in a time-indexed state vector. The state vector can include one or more state parameters, as described above with reference to Figure 1 The chromatographic system 10 is shown as described. Figure 2 In the specific example of the chromatography system simulation 500 shown, the chromatography system 10 may maintain one or more of the following current and historical records as one or more state parameters:
[0195] - feed container 510, elution container 512, wash container 514, product container 540 and / or waste
[0196] The concentration of the component in the material container 542;
[0197] - component concentration along the length of column 520;
[0198] - feed container 510, elution container 512, wash container 514, product container 540 and / or waste
[0199] The volume of the component solution in the material container 542;
[0200] - The approximate mass of components included in the column 520 and the feed container 510 , elution container 512 , wash container 514 , product container 540 and / or waste container 542 .
[0201] The flow controller 516 and the output valve 530 can be programmable. For example, the flow controller 516 can set the flow of each component to zero and a specified maximum flow rate (e.g., 2e -6 [m 3 / s]), and the output valve 530 can be configured to direct the flow to a waste container 542 or a product container 540.
[0202] It should be noted that the simulation environment 50 described above is only an example, and the chromatography system 10 may be implemented in a variety of ways. Figure 2 The exemplary simulation environment 50 shown is different from the simulation environment. For example, in the case of simulating different types of chromatography processes, the simulation environment implemented by the chromatography system 10 may have different components and structures suitable for simulating different types of chromatography processes.
[0203] Machine Learning Algorithms for Strategy Generation
[0204] As mentioned above, for Figure 1 The control device 20 shown can generate a strategy based on a machine learning algorithm trained using state-action pairs, and determine one or more values of one or more control parameters based on the strategy. For example, the machine learning algorithm can include one or a combination of two or more of reinforcement learning, deep reinforcement learning, supervised learning, semi-supervised learning, self-supervised learning, imitation learning, and transfer learning.
[0205] a) Reinforcement Learning
[0206] In some exemplary embodiments, reinforcement learning may be used as a machine learning algorithm. Reinforcement learning generally aims to optimize the behavior of an agent in an environment by defining a reward function that should increase in value for desired behavior and decrease in value for undesired behavior. Thus, a reinforcement learning algorithm may be based on an agent interacting with an environment, and may be directed to cause the agent to seek and repeat actions that produce higher rewards using a learned approach. Attempts to repeat actions that produce higher rewards must generally be balanced with exploration to avoid the agent from getting stuck in a local optimum. Figure 3 A schematic diagram illustrating a reinforcement learning agent interacting with an environment is shown.
[0207] Reference Figure 3 , the agent 30 can interact with the environment 40. During learning, the environment 40 can be in state S at time point t t , and the agent 30 may respond to obtaining the state S t To select action A t The environment 40 can utilize the action A taken by the agent 30 t To update its state and return the next state S t+1 and the state S t+1The corresponding reward R t+1 In the context of the present disclosure for controlling a chromatography system 10, the environment 40 may represent the chromatography system 10, the state S t may include one or more values of one or more state parameters of the chromatography system 10, and action A t One or more values of one or more control parameters for controlling the chromatography system 10 may be included. The reward R may be calculated using the values of some of the parameters. t+1 , these parameters may represent the execution of the chromatographic process, for example, the amount of the target compound obtained at the end of the chromatographic process. The interaction between the agent 30 and the environment 40 may continue until the environment 40 reaches a terminal state or for a specified number of iterations. Typically, in order for the agent 30 to learn an optimal policy, a plurality of sets of interactions between the agent 30 and the environment 40 must be run from start to finish, for example, from an initial state of the environment 40 until the environment 40 reaches a terminal state or until a specified number of iterations are completed. Such a set of interactions is often referred to as an "episode" or "trajectory".
[0208] b) Deep Reinforcement Learning
[0209] In some exemplary embodiments, deep reinforcement learning can be used as a machine learning algorithm. Deep reinforcement learning can be considered as a combination of classical reinforcement learning as described above and deep learning using neural networks. In a deep reinforcement learning algorithm, a strategy for mapping states to actions and / or other learned functions can be implemented by an artificial neural network.
[0210] An exemplary algorithm for deep reinforcement learning may involve an actor-critic approach using components called an "actor" and a "critic," both of which are implemented as neural networks. Figure 4 A schematic diagram of agents and environments for deep reinforcement learning involving an actor-critic approach is shown.
[0211] Reference Figure 4 , the agent 30 may include an actor network 32 and an evaluator network 34. The actor network 32 may include a first neural network to be trained to represent a policy, the first neural network being configured to transform the state S t as input and take action A t As output. The evaluator network 34 may include a second neural network that is trained to estimate the reward obtained by the state-action pair. The evaluator network 34 may be used to train the actor network 32 to output an action that produces a high reward in response to the current state St input to the actor network 32.
[0212] As a specific example, a deep reinforcement learning algorithm called double-delay DDPG (TD3) (e.g., see Fujimoto et al., “Addressing Function Approximation Error in Actor-Critic Methods,” Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018, URL: https: / / arxiv.org / abs / 1802.09477) can involve an agent 30 including an actor network 32 and a critic network 34, and double-delay DDPG is built on top of deep deterministic policy gradient (DDPG) (e.g., see Lillicrap et al., “Continuous Control with Deep Reinforcement Learning,” URL: https: / / arxiv.org / abs / 1509.02971), such as Figure 4 shown.
[0213] TD3 can be thought of as a temporal difference (TD) learning algorithm, where estimates of future rewards can be bootstrapped by estimating future rewards using a value function, rather than running the agent 30's interaction with the environment 40 to completion and collecting all future rewards. The value function estimates can be made by a critic network 34, which can iteratively improve these estimates. To train the critic network 34, the performance of the critic network 34 can be estimated in a bootstrapped manner by calculating the TD error defined as follows:
[0214] TD error =R t +γQ(S t+1 , A t+1 , w Q )-Q( S t,A t , w Q ),
[0215] Where R t , S t and A t can be the reward, state, and action of the current time step t, respectively, and S t+1 and A t+1 can be the state and action of the next time step t+1 respectively. Q can be represented by the evaluator network 34 with weights wQ is calculated and can be used to estimate the future reward (also known as the Q-value) for the state-action pair. γ can be a discount factor used to discount the rewards of future actions, as they are generally more uncertain than the current reward. Ideally, as the network gets better, the TD error should approach zero, because the Q-value at time step t should be equal to the estimated reward at time step t plus the Q-value at time step t+1, giving an ideal Q-value estimate as follows:
[0216] Q(S t , A t , w Q )=R t +γQ(S t+1 , A t+1 , w Q )
[0217] T Derror can be used as a loss / cost function to train the critic network 34 using standard deep learning optimization techniques such as gradient descent. The Q value can be estimated by using the critic network 32 as follows To train the actor network 32 to produce the optimal policy:
[0218]
[0219] Among them, action A t Can be based on state S t and weight w π The output of the actor network 32 under the policy π:
[0220] A t =π(S t , w π )
[0221] Since the goal of the actor network 32 may be to take the action that will generate the highest possible reward, and the critic network 34 may provide such a differentiable estimate, gradient ascent may be used on the actor network 32 to update the network weights w π , resulting in a better policy π. In practice, this can be achieved by freezing the network weights of the critic network 34 and using the estimated negative future reward for the current action-state pair The gradient descent is implemented as follows:
[0222] L actor =-Q(S t ,π(S t , w π ), w Q )
[0223] By exploiting bootstrapping in TD learning, for time step t, only the state, action, reward, next state, and next action may be needed to compute all losses to train a reinforcement learning agent (e.g. Figure 4 Agent 30 shown). This is often referred to as state-action-reward-state-action (SARSA). In practice, this may mean that SARSA can be saved for each time point as the agent 30 interacts with the environment 40, and these experiences can be used for training.
[0224] When training the agent 30, the agent 30 may run the simulation (and / or conduct real-world interactions) for a certain number of iterations, or until a terminal condition is reached. During training, SARSAs (in other words, experiences) may be recorded and stored in a storage buffer. After a specified number of steps of interacting with the environment 40, the agent 30 may sample the storage buffer a fixed number of times and use those experiences to train the agent networks (e.g., the actor network 32 and the evaluator network 34). Thus, the drawn training examples may be from current and past events, which may result in more stable training because the drawn experiences are typically not as interrelated as when training with the following experiences in chronological order (see, e.g., Mnih et al., "Playing Atari With Deep Reinforcement Learning", NIPS Deep Learning Workshop 2013; URL: https: / / arxiv.org / abs / 1312.5602).
[0225] In some exemplary embodiments, a storage buffer as described above can be used to run multiple experiments in parallel and collect all experiences in the storage buffer. In order to make the agent 30 explore new possible actions, noise can be added to these actions, where the noise can decay over time to make the agent 30 explore more at the beginning and exploit the environment 40 more at the end. In this way, the same agent 30 can interact with multiple versions of the exact same environment 40 and state, but take various actions due to the detection noise. Figure 5 A schematic diagram illustrating this multi-environment interaction of a reinforcement learning agent is shown. Figure 5 , for each time step t, the agent 30 may output an action vector from the state vector.
[0226] In some variants of deep reinforcement learning, only one of the actor network 32 or the critic network 34 is used (see Figure 4), and the heuristic algorithm is used in another role.
[0227] c) Supervised Learning
[0228] In some exemplary embodiments, supervised learning can be used as a machine learning algorithm. In a supervised learning algorithm, states (e.g., including values of one or more state parameters) and corresponding actions (e.g., including values of one or more control parameters) taken according to a strategy made by a human expert can be collected, and the states and corresponding actions are used to train a strategy network (e.g., corresponding to the "actor" of the above-mentioned actor-evaluator method) to mimic the actions taken by the human expert. This training can be performed on a series of chromatographic beds (e.g., columns) using a variety of expert strategies, and a series of chromatographic beds exhibit variability in binding capacity, for example, to obtain strategies that can adapt to new situations.
[0229] In some exemplary embodiments, supervised learning can be combined with deep reinforcement learning. For example, the actor network 32 can first be trained to imitate the expert strategy using a supervised learning algorithm and then in a deep reinforcement learning setting (e.g., Figure 4 The trained actor network 32 is used in the example shown in FIG. 3 for further exploration and learning.
[0230] In a further exemplary embodiment, the actor network 32 can be trained by accessing both the environment interaction and the expert policy state-action pairs, for example, sampling heavily from the expert system experience at the beginning and interacting almost exclusively with the environment after training. This approach can also be referred to as imitation learning.
[0231] d) Imitation Learning
[0232] In the case where a strategy for supervised learning developed by a human expert is available as described above, in some exemplary embodiments, imitation learning can be used as a machine learning algorithm. In an imitation learning algorithm, an agent can be trained to take the same action as an expert strategy given a specific state through a supervised learning algorithm.
[0233] In some cases, combining imitation learning of expert policies from simulated data and real data obtained by physically performing the chromatographic process and deep reinforcement learning may be advantageous for bridging the gap between training using real data (e.g., obtained by physically performing the chromatographic process) and training in a data-efficient manner using simulated data.
[0234] e) Transfer Learning
[0235] In some exemplary embodiments, transfer learning can be used as a machine learning algorithm. In a transfer learning algorithm, knowledge gained while solving a problem can be applied to solve a different but similar problem.
[0236] Thus, in the case of generating a strategy for controlling a chromatography system, a reinforcement learning agent may first be pre-trained using state-action pairs obtained by simulating a chromatography process, and then, using the knowledge gained during pre-training, the agent may be further trained using state-action pairs obtained by physically performing the chromatography process by the chromatography system. For example, pre-training and further training may be performed according to deep reinforcement learning, supervised learning, or a combination of both. By pre-training the agent as much as possible using simulated data, the need for training using real data by physically performing the chromatography process may be reduced. This may help control and / or improve the overall efficiency of a chromatography system configured to physically perform a chromatography process because the expensive operation of the chromatography apparatus used to train the agent may be reduced.
[0237] Control process
[0238] Figure 6 Shown by Figure 1 Flow chart of an exemplary process performed by the control device 20 shown in FIG. For example, when the control device 20 receives a user instruction to start the exemplary process and / or when the control device 20 receives a notification from the chromatography system 10 that the chromatography system 10 is starting a chromatography process, Figure 6 The exemplary process shown may begin.
[0239] In step S10, the control device 20 may generate a strategy for controlling the chromatography system 10. The strategy may map the state of the chromatography system 10 represented by one or more values of one or more state parameters obtained from the chromatography system 10 at a given time point to an action representing one or more values of one or more control parameters for controlling the chromatography system 10.
[0240] Figure 7 shows the use of Figure 6 This is a flowchart of an example of the detailed process of step S10. Figure 6 In step S10, the control device 20 may execute Figure 7 An exemplary process is shown.
[0241] Reference Figure 7, in step S100, the control device 20 may receive a state-action pair obtained at least in part by physically executing and / or simulating a chromatographic process. For example, the processor 202 of the control device 20 may receive a state-action pair recorded when the chromatographic system 10 physically executes and / or simulates a chromatographic process. Such a state-action pair may be received directly from the chromatographic system 10 and / or from another computing device such as a database server, which is configured to collect and store state-action pairs from the chromatographic system 10. Additionally or alternatively, for state-action pairs originating from the chromatographic system 10, the processor 202 may receive the state-action pair according to a strategy formulated by a human expert. For example, a state-action pair according to an expert strategy may be received from a computing device other than the chromatographic system 10.
[0242] Each of the state-action pairs received at step S100 may include a state including one or more values of one or more state parameters as described above and an action taken in response to the state including one or more values of one or more control parameters as described above. The state-action pairs received at step S100 may be stored in the storage medium 204 of the control device 20.
[0243] After step S100, Figure 7 The exemplary process may proceed to step S102.
[0244] In step S102, the control device 20 may generate a strategy based on a machine learning algorithm and using the received state-action pairs. The machine learning algorithm may be any of the exemplary algorithms described above. For example, the processor 202 of the control device 20 may generate a strategy based on the state-action pairs described above. Figures 3 to 5 The described deep reinforcement learning algorithm is used to train the reinforcement learning agent to generate a strategy. In addition, for example, in the case where the state-action pair according to the above expert strategy is received in step S100, the processor 202 can generate a strategy according to the supervised learning algorithm as described above.
[0245] After step S102, Figure 7 The exemplary process may proceed to step S104.
[0246] In step S104, the control device 20 may store the generated policy in the storage medium 204 of the control device 20. After step S102, Figure 7 The exemplary process may end, and Figure 6 Step S10 of the illustrated exemplary process may end.
[0247] Reference again Figure 6 , the exemplary process may proceed to step S12 after step S10.
[0248] In step S12, the control device 20 can obtain the current state of the chromatography system 10 from the chromatography system 10. For example, the processor 104 of the chromatography system 10 can collect the current state, and the processor 202 of the control device 20 can receive the current state from the processor 104. The current state may include one or more values of one or more state parameters at the current time point. The values of the same set of state parameters may be included in the current state and included in the state of each state-action pair used to generate the strategy in step S10. As described above, the one or more state parameters may include one or more amounts of one or more substances present in the chromatography system. In addition, the one or more state parameters may additionally include the above reference Figure 1 One or more of any other examples of state parameters described.
[0249] After step S12, the exemplary process may proceed to step S14.
[0250] In step S14, the control device 20 may determine one or more values of one or more control parameters according to the strategy generated in step S10. For example, the processor 202 of the control device 20 may identify the action mapped to the current state by the strategy, and use one or more values of one or more control parameters included in the identified action as the determined value. In step S14, the values of the same set of control parameters as those in the action of each state-action pair used to generate the strategy in step S10 may be determined. As described above, the one or more control parameters may include at least: the position of a valve included in the chromatography system 10 and / or the pump speed of a pump included in the chromatography system 10. In addition, the one or more control parameters may include the above reference Figure 1 One or more of any other examples of control parameters described.
[0251] After step S14, the exemplary process may proceed to step S16.
[0252] In step S16, the control device 20 may use one or more determined values of one or more control parameters to control the chromatography system 10. For example, the processor 202 of the control device 20 may generate one or more control signals, which represent one or more determined values of one or more control parameters to instruct the chromatography system 10 to set the determined values of the control parameters. The processor 202 may transmit the generated control signals to the chromatography system 10. In response, the chromatography system 10 may set the one or more control parameters to the one or more determined values after the one or more control signals, as described above with reference to FIG. Figure 1 described.
[0253] After step S16, the exemplary process may proceed to step S18.
[0254] In step S18, the control device 20 can determine whether to end the exemplary process. This determination can be based on whether a specified termination condition is met. For example, in the case where the chromatography system 10 is configured to notify the control device 20 when the chromatography process ends, the specified termination condition can be whether the chromatography system 10 has notified the chromatography process to end. In addition, for example, the specified termination condition can be that a specified duration has passed since the chromatography process began. In addition, for example, the specified termination condition can include conditions related to at least some of the state parameters. Alternatively or additionally, the specified termination condition can include one or more conditions related to one or more things related to safety, facility conditions, production suspension, etc. The occurrence of such things can be detected using external sensor data obtained, for example, from one or more sensors, which are arranged on one or more devices and / or facilities outside the chromatography system 10.
[0255] In the event that it is determined not to end the exemplary process ("NO" in step S18), the process may return to step S12.
[0256] In the event that it is determined to end the exemplary process ("YES" in step S18), the process may end.
[0257] In some exemplary embodiments, the control device 20 need not perform Figure 6 For example, the strategy generation step S10 may be performed by a computing device other than the control device 20. In this case, the control device 20 may obtain the generated strategy from the computing device and then perform Figure 6 Steps S12 to S18 are used to control the chromatography system 10.
[0258] Case Study 1 - Experiment
[0259] As a case study, experiments were performed using deep reinforcement learning in the context of controlling a chromatography system to successfully capture and purify a monoclonal antibody (mAb). This case study 1 focused on processing simulated data from a chromatography process.
[0260] Bind-elute process chromatography can be considered one of the major downstream purification unit operations in most biopharmaceutical manufacturing processes. In these units, the medium to be treated can be passed through a column containing a fixed solid phase (usually resin beads) with binding sites that chemically retain the product of interest. When most of the binding sites have been filled, unbound impurities in the column can be washed away. Finally, the conditions of the column (usually pH or salinity) can be changed to reverse the binding and release (in other words, elute) the purified product.
[0261] Process chromatography unit operations in manufacturing facilities are typically operated in batch mode, where a series of stages are repeated following a predetermined (and therefore, unadapted) recipe. A typical recipe may involve preparing a column with a wash step, "loading" the column with product, washing impurities, eluting the product, and cleaning the column before the cycle begins again. The flow rate of material and the duration of each of these steps are typically determined by a combination of experimental work and historical process knowledge. For example, column capacity at different flow rates is typically determined by performing so-called breakthrough experiments, where, in each trial, a solution of known product concentration is fed at a constant flow rate until the product concentration leaving the column is equal to the product concentration fed. From the breakthrough experiment, the "optimal" flow rate and loading duration for a given product concentration can be estimated.
[0262] The "optimal" flow rate and loading duration can vary with column variability and reduced capacity due to use. The method for controlling a chromatography system according to the present disclosure can account for this variability and adjust the control scheme based on real-time monitoring data of the chromatography process.
[0263] a) Simulation environment
[0264] In this case study 1, a simulation environment was used to easily run many experiments in a short period of time. The simulation system was combined with a deep reinforcement learning algorithm whose task was to learn how to control the process in a way that separated the sought product (e.g., target compound, mAb) from the rest of the medium.
[0265] Specifically, in this case study 1, the Figure 2 The exemplary simulation environment 50 is shown. Figure 2 As described above, the simulation environment 50 describes a single column system to simulate the bind-elute process of a liquid chromatography process. Thus, in this case study 1, the chromatography system 10 (see Figure 1 )'s processor 104 and storage medium 106 implement Figure 2 An exemplary simulation environment 50 is shown. In addition, experiments were performed to generate strategies for controlling the device 20 and to control the chromatography system 10 according to the generated strategies.
[0266] b) Machine Learning Algorithms and Reward Functions
[0267] As mentioned above, deep reinforcement learning was used in this case study 1. Specifically, the above reference Figure 4 The TD3 algorithm described above is used to generate the strategy. Figure 5 As described above, train an agent using a multi-environment setup.
[0268] In this case study 1, the state S includes a vector containing the concentration values of mAb (e.g., target compound), spent media (SM), elution, and washings entering the flow controller and flowing out of the column. The mass transfer through the column inlet and outlet is also calculated using the concentration data of the phases at the inlet / outlet and the total system flow. The calculated mass transfer is provided to the agent as additional observation data. The calculation of the initial state may require knowing the starting concentrations in the feed and elution volumes. In the experiments in this case study 1, the values of the state S are normalized to place them in the range close to [0, 1] because the original concentrations are in the range of 1e -6 , and such normalization can be beneficial to neural network performance.
[0269] Furthermore, in this case study 1, the action comprises a vector with values between [-1, 1] (hyperbolic tangent (TanH) activation), which is normalized to the range [0, 1] and used to control valves and flows of the chromatography system (see, for example, Figure 2 , chromatography system simulation 500). More specifically, the actions correspond to valve and pump control for feed, elution and wash media, and output valve 530 that controls the flow of media from the column to a waste container 542 or a product container 540.
[0270] There are many ways to construct and utilize reward functions for deep reinforcement learning algorithms. For example, for a game like board games, it may be sufficient to give the agent a loss reward of -1 and a win reward of +1 at the end of an episode, and the rewards for all states up until that point are set to 0. In this case study 1, the reward for each state is calculated and then subtracted from the previously recorded reward to obtain a reward that reflects the changes made on a step-by-step basis, which can be represented by the following expression:
[0271] Δ reward =reward {t} -reward {t-1}
[0272] For case study 1 of controlling a chromatography system, the reward function must take into account multiple factors. First and foremost, the purpose of the chromatography process may be to separate the product (e.g., target compound such as mAb) into product container 540 and the waste (e.g., remaining medium, in other words, used medium) into waste container 542. For a smooth reward function, it may be important to not only reward the product (e.g., in this case study 1, target compound mAb) for reaching the correct location (e.g., product container 540), but also to penalize the agent when it places the product in the wrong location (e.g., waste container 542). This can be reflected by the following term:
[0273] Δtc=tc prod -tc waste ,as well as
[0274] Δsm=sm waste -sm prod
[0275] where tc prod The amount of the target compound (e.g., mAb) in the product container 540 can be indicated, tc waste The amount of target compound in the waste container 542 can be indicated, sm waste The amount of waste (e.g., spent media) in the waste container 542 may be indicated, and the sm prod The amount of waste in the product container 540 may be indicated. First, these two terms are combined to generate a reward function as follows:
[0276] Reward=(M×Δtc+Δsm),
[0277] Where M can be a factor that represents how much waste is used in the chromatography process compared to the target compound. In other words, the term Δtc is multiplied by M because there is M times more waste in the system. In the experiments for this particular case study 1, M=5. Through experimental work, this initial reward function was shown to be insufficient to obtain the desired behavior of the agent. Specifically, with this initial reward function, the agent obtained a much higher score than a perfect separation of the two compounds, but at the expense of a lower purity product. This is likely because, although a perfect separation requires less feed to be put into the system after loading, if the agent just keeps the feed high, the absolute reward for the agent is greater because it has accumulated more product at the end.
[0278] Therefore, the product with the following purity term pure is introduced into the reward function:
[0279]
[0280] In this particular case study 1, the purity term pure described above has a minimum value of x=0.1, but can increase as the purity of the product mAb in the product container 540 increases. The purity term pure is introduced in this particular case study 1 because it is noticed that without this term, the agent has difficulty starting to learn and the minimum value of x=0.1 is chosen as the value that provides the best results after testing with different values (e.g., x=0.1, 0.2, 0.3, etc.). The reward function for the purity term pure can be expressed as follows:
[0281] Reward=(M×Δtc+Δsm)×pure
[0282] The above reward function for the term pure introduces another interesting behavior of the agent. Specifically, the agent starts putting all of the target compound mAb into column 520 and never washes it out, only accumulating scores from waste being correctly put into waste container 542, and not putting anything into product container 540. To explain this behavior, the following additional penalty p for target compounds left in column 520 is introduced:
[0283]
[0284] where tc col is the amount of target compound mAb in column 520, and t is a specified threshold. If at the end of a run (e.g., an event), the threshold t of target compound mAb in column 520 is exceeded, a penalty is incurred. To further give an explicit reward for each unit of "clean" target compound in product container 540, the following clean term "c" is added:
[0285] c=max(tc prod -sm prod ,0)
[0286] Therefore, the final reward function can be defined as follows:
[0287] Reward=(M×Δtc+Δsm)×pure+cp×done,
[0288] where done is set to 1 in the case of the terminal state of the chromatography process and to 0 otherwise, which may result in a penalty for the agent of the mAb remaining in column 520 at the end if the amount of mAb is above a specified threshold t. In the experiment of this particular case study 1, the reward is further multiplied by a factor 1e 6 , in order to keep the reward within a reasonable range.
[0289] c) Training
[0290] In the experiments of this particular case study 1, to train the agent, the critic network 34 and the actor network 32 (see Figure 4) is initiated by two convolutional layers connected to a hidden fully connected layer consisting of 300 neurons. The convolutional layers that process the input are provided to support a cascaded representation of the input state over time. Using convolutional layers in this way is often referred to as a Temporal Convolutional Network (TCN) and is a common way to process time series data. TCNs have improved performance compared to fully connected networks with only 1 observation time point as input. The first layer is parameterized with channels 128 and 64, with a kernel size of 3 for channels 128 and 64, and a stride of 1. The output of the second layer is flattened and then propagated to the fully connected third layer, which propagates the signal to the fully connected output layer.
[0291] In this case study 1, both the actor network 32 and the critic network 34 involving TCN take as time series input a normalized vector representing the simulator state (e.g., the state provided by the chromatography system 10 performing a simulation of the chromatography process). The critic network 34 also merges the vector representing the action in the third (fully connected) layer with the state vector. This can be done by connecting the output of the second hidden layer (convolutional) with the action vector, each of which is activated using a ReLU (rectified linear unit) (e.g., see also Fujimoto et al., "Addressing Function Approximation Error in Actor-Critic Methods", Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018, URL: https: / / arxiv.org / abs / 1802.09477).
[0292] In this case study 1, the output of the actor network 32 is a vector representing actions that can be taken during the simulated spectrum based on the input state. In addition, in this case study 1, the output of the critic network 34 is a single value representing the Q-value (estimated future reward) based on the state-action pair propagated through the critic network 34.
[0293] In this case study 1, both the actor network 32 and the critic network 34 involving TCN are trained using gradient descent with the ADAM optimizer (e.g., see Kingma et al., “Adam: A Method for Stochastic Optimization,” published as a conference paper at the 3rd International Conference on Learning Representations, San Diego, 2015, URL: https: / / arxiv.org / abs / 1412.6980). For the actor network 32 and the critic network 34, 3e -4 The learning rate. The priority experience storage buffer is used to store historical experience-tuple format (state t 、action t 、reward t 、state t+1 、action t+1 ), the buffer size is 1e 6 The storage buffer is sampled 6 times, each time interacting with the environment 4 times. Each sampling is done with a batch size of 200 (number of experiences) and is used to train the network to better approximate the correct policy (action) and Q-values. The actor network 32 is trained every other sampling, while the critic network 34 is trained whenever sampling occurs, which is a technique introduced by the TD3 algorithm as an improvement to DDPG.
[0294] In the experiments for this case study 1, training was performed for 1000 events, where each event consisted of 4 simulations run in parallel for a total of 160 iterations (4000 simulation runs in total). Each simulation run approximated a chromatographic device interacting with the agent every 10 seconds, for a total of 1600 seconds. The best model was saved for each run and used for plotting and evaluating performance.
[0295] d) Evaluation
[0296] To evaluate performance, rewards defined by the final reward function described above are calculated for the agent and the benchmark. The amount of mAb (target compound) captured in the product container 540 and the waste container 542 and the purity of the mAb in the product container 540 are compared between the agent and the benchmark.
[0297] Figure 8 An exemplary benchmark artifact strategy for separation of mAb proteins and for controlling spent media of an exemplary chromatography system is shown. Figure 8 a) and b) show the mAb, SM (spent medium) and eluent concentrations at the column inlet and outlet, respectively. Figure 8 a) and b) show the internal state of the chromatography system. The inlet concentrations of mAb and SM are determined by their concentrations in the feed container 510. Figure 8 c) shows the controllable flow of feed, elution and wash into column 520. Figure 8 d) shows the flow out of column 520, into product container 540 or waste container 542, which can be switched by controlling output valve 530. Thus, Figure 8 c) and d) represent actions taken to control the chromatographic system. Figure 8 e) and f) show the quality of mAb and SM in product container 540 and waste container 542 over time, respectively. In other words, Figure 8 e) and d) show the separation of product mAb and waste.
[0298] Table 1 below shows the comparison of the agent-derived strategies with the human baseline.
[0299] Table 1
[0300]
[0301] The "score" shown in Table 1 is a value calculated using the final reward function used to train the agent as described above. Table 1 also shows the amount of mAb captured in the product container 540 and the waste container 542 and the cleanliness of the mAb in the product container 540 (the concentration of mAb).
[0302] The best performing agent selected from the training in the experiment of this case study 1 was trained for 657 events and scored 328.4. As shown in Table 1, the selected agent produced a higher score than the human baseline. This may be due to the agent capturing more mAbs in a given time, even though it is not as clean as in the baseline model (67% vs. 91%), which can be seen from Table 1.
[0303] Fig. 9 An exemplary chromatography system control process performed by a selected agent is shown. Fig. 9 a) and b) show the concentrations of mAb, SM and eluate at the column inlet and column outlet, respectively. Fig. 9 c) shows the column inlet flow rates of feed, elution and wash media, which the agent can control. Fig. 9 d) shows the column outlet flow to the product container 540 or to the waste container 542 , which the agent can control by controlling the output valve 530 . Fig. 9 e) and f) show the amounts of mAb and SM in the product container 540 and the waste container 542, respectively.
[0304] from Fig. 9 As can be seen from e) and f), the agent has learned the strategy to separate SM into waste container 542 and mAb into product container 540. Even though the separation may not be perfect, Fig. 9 c) shows that the agent can exhibit behavior associated with a classical chromatography strategy with a loading phase (where the feed flow is high, while the wash and elution flow rates are low), followed by a wash phase and an elution phase. Fig. 9 c) even with Figure 8 These phases are shown in a similar cyclic behavior to the baseline strategy shown in c). Fig. 9 As can be seen in c) and d), the generated phases are controlled synchronously with the output valve 530, directing the feed to the waste container 542 during the loading phase and the washing phase, and redirecting it to the product container 540 during the elution phase. Fig. 9 a) and b) clearly show how the agent manipulates the chromatographic system to change the concentration of mAb and SM from the original feed mixture concentration (see Fig. 9 a)) changes the phase of the mAb or SM at the outlet of column 520 (see Fig. 9 b)).
[0305] Fig. 9 The behavior of the selected agent shown can be contrasted with the behavior of an earlier agent that was trained on only 10 events and had 14 scores. Fig.10 The control process of an exemplary chromatography system performed by this earlier agent after training 10 events is shown. Fig.10 a) and b) show the concentrations of mAb, SM and eluate at the column inlet and column outlet, respectively. Fig.10 c) shows that the agent can control the column inlet flow rates of feed, elution and wash media. Fig.10 d) shows the column outlet flow to a product container 540 or to a waste container 542 , the agent can control the column outlet flow by controlling the output valve 530 . Fig.10 e) and f) show the amount of mAb and SM in the product container 540 and the waste container 542, respectively. Fig.10 c) and d) show that Fig. 9 Compared to the behavior of the selected agents shown in c) to f) of FIG. 1 and the separation achieved with the selected agents, the earlier agents exhibited a more irregular and "random" behavior, with little to no separation of mAb and SM (see Fig.10 e) and f)). This presents a stark contrast in how much learning has taken place.
[0306] Reference above Figures 8 to 10The experimental results of this case study 1 described above show that a reinforcement learning agent can successfully learn how to generate a control policy by interacting with a simulated chromatography device and without any prior knowledge, starting from completely random actions to a complex control scheme. Although the agent in this case study 1 did not produce clean mAbs like the human policy, the agent still produced a higher score by collecting more products (mAbs) in a fixed time. Modifying the reward function may lead to further improvements in the purity of the collected products.
[0307] It is emphasized that agents exhibit recurrent behavior and that the policy generated by training the agent (see Fig. 9 c) and d)) include the same stages as those used in the current chromatographic strategy (e.g. Figure 8 As the agent loads the feed into the column, the mAb will begin to leak as it approaches the column binding capacity. When the agent encounters this challenge, it appears to have discovered structured and periodic behavior.
[0308] Thus, the experiments in Case Study 1 demonstrated that methods and / or systems according to the present disclosure can automatically generate strategies that allow control of a chromatographic process during the entire process, including the timing when media flowing into one or more chromatographic beds and / or different containers (e.g., product containers and waste containers) are switched.
[0309] In this Case Study 1, strategies generated using data obtained from a simulation of a chromatography process can be used to control one or more chromatography devices configured to physically perform the chromatography process and having the same corresponding components as in the simulation environment.
[0310] Case Study 2 - Experiment with Column Variability
[0311] Case Study 1 described above demonstrates the ability of a deep reinforcement learning (RL) algorithm to operate a simulated chromatography system after interactively learning from the simulated chromatography system.
[0312] As a further case study, in this case study 2, we further study how a deep RL agent according to the present disclosure learns to operate a simulated liquid chromatography system with stochastic variability that introduces a chromatography column. Today, users of liquid chromatography are usually required to perform experiments to find out the binding capacity of a column and thus adjust the process based on the variability of the column. Variability can come from manufacturing conditions as well as usage over time.
[0313] The hypothesis used in this case study 2 is that a deep RL agent according to the present disclosure is able to adapt to column variability and is able to control the chromatography system by acting on the measured signal. To test this hypothesis, a set of test columns with random properties that affect binding capacity were generated. The agent was then trained on randomly initialized columns and evaluated on predefined test columns. The results presented below demonstrate the promising ability of the RL agent to successfully control and adapt to chromatography systems with column capacity variability.
[0314] In the following, the experimental setup and results of Case Study 2 are described, focusing on the differences compared to Case Study 1.
[0315] a) Column variability
[0316] To simulate column variability, a set of test columns with different performances are created. In these columns, the selected parameters are adjusted to create a unique simulation environment. The randomized parameters of the column are generated by adding the product of the initial values scaled with uniformly distributed errors:
[0317] f(x)=x+x·S·X
[0318] Here, X~U(-1,1) is a uniformly distributed random variable, S is a scaling variable, and x is some input parameter. In the experiments of Case Study 2, this treatment was performed for axial dispersion, radial dispersion, column porosity, and resin porosity. The scaling term was kept common for all the columns tested, thereby introducing the maximum relative deviation for each parameter. The environmental variation was then implemented by sampling from X and multiplying with the error term.
[0319] Eight test columns were generated, of which seven were sampled from the random starts described above, scaled to 5% (S=0.05), and one was started as the one in Case Study 1 described above.
[0320] b) Simulation
[0321] To represent the binding interactions between the solute components and the stationary phase, an adjustment of the mobile phase modifier Langmuir model (see Karlsson et al., "Model-based optimization of a preparative ion-exchange step for antibody purification", Journal of Chromatography A, Vol. 1055, No. 1-2, Nov. 5, 2004, pp. 29-39, URL: https: / / doi.org / 10.1016 / j.chroma.2004.08.151) was used. This yielded a dependence of the mAb concentration in the stationary phase on the elution buffer concentration. This dependence was evaluated for each individually modeled particle along the column axis. The interaction was governed by the following equation:
[0322]
[0323] Here, q is the stationary phase concentration, c p is the mobile phase concentration of the stagnant fluid in the particle, q max is the saturation constant, k a and k d is the adsorption / desorption rate constant, and β and γ are the modulation constants. It is assumed that only mAb (target compound) can bind to the stationary phase and the elution buffer concentration is the only component that affects this interaction.
[0324] c) RL Agent
[0325] First, a new environment with randomly initialized columns was run with the same deep RL agent setup as in Case Study 1, but without success. It was observed that the agent could reach a certain concentration of about 35% fairly easily and then had difficulty progressing further. Therefore, the reward function was modified from the function used in Case Study 1 and defined as:
[0326] Reward=(M×Δtc+Δsm)×pure+cp×done
[0327] More details of the reward function used in Case Study 1 are described in Section b) of Case Study 1, “Machine Learning Algorithm and the Reward Function”.
[0328] In order to modify the reward function used in case study 1, a new term pure_reward is introduced. The new term pure_reward gives a linearly increasing score based only on purity (ignoring the amount of product) after 30% labeling.
[0329]
[0330] This term is theorized to help agents get over the barrier of reaching their upper purity limit, and the reward from this term can be found in Fig.11 Seen in.
[0331] When the pure_reward term is used to calculate the total reward to play a more important role in the total reward, the pure_reward term is scaled by a factor of 100, resulting in the following new reward equation:
[0332] Reward=(M×Δtc+Δsm)×pure+c+pure_reward×100-p×dome
[0333] As in Case Study 1, in this Case Study 2 the factor M was also set to M=5. The new equation showed good results in overcoming the previously observed upper purity limit. Further performance improvements were obtained by normalizing the observed states of the agent to the range [0, 1]. This was done by experimentally running the chromatography simulator and identifying the highest values for the different measurements, and then dividing the state vector by those maximum values.
[0334] d) Training
[0335] In case study 2, the same TD3 agent was trained as in case study 1, but the number of episodes was modified to 2000 runs on 8 parallel simulation environments. The duration of the simulation was increased to 3000 seconds, with the agent interacting with the environment every 15 seconds, for a total of 200 time steps per episode. Both the actor network 32 and the critic network 34 were trained at 4e -4 The learning rate was started with a learning rate of , where the network weights were updated four times in succession, with the interaction frequency with the environment being every four times. The batch size (e.g., number of experiences) used to update the network each time was set to 200. While training, the column parameters were randomly started for each new event with a scaling factor S of 0.2 according to the method described in section a) “Column variability” above.
[0336] e) Results and evaluation
[0337] When evaluating the RL agent trained on randomly initialized columns on predetermined test columns, an average purity of 81% (over all test columns) was observed, which can be compared to the human benchmark with a purity of 99.12% (see Table 2 below). However, the RL agent managed to produce a higher reward score of 36.29, while the human benchmark achieved 26.78. In addition, it can be observed that the RL agent has a much higher productivity, with a yield of 1.33e of mAb -5 g, while the yield of the mAb from the human strategy was 0.59e -5 g. This makes the RL agent 125% more productive at the expense of purity. Based on these findings, it can be concluded that the high productivity of the RL agent equates to higher rewards than the human policy, even though it is 18% less pure.
[0338] Table 2 below shows the comparison of an agent trained in the environment using pillars with random initialization with a human baseline policy on a predefined test set of 8 pillars.
[0339] Table 2
[0340]
[0341] In Table 2 above, "score" is a value calculated using the reward function defined for agent training. The amount of mAb captured in the product container 540 and the waste container 542 is also shown in Table 2. Table 2 further shows how pure the mAb in the product container is (concentration of mAb / (sm+mAb)). The recycled mAb recovered in Table 2 indicates how much target protein is recovered in the product container 540 after recycling the raw material. The total mAb recovered in Table 2 indicates how much target protein is recovered relative to the initial target protein amount in the feed container 510.
[0342] Observing how an agent controls and affects a simulated chromatographic system can help understand the agent's behavior.
[0343] Fig.12 The control process of a simulated chromatography device by an RL agent in this case study 2 is shown. Fig.12 a) and b) show the column inlet and outlet concentrations of mAb, spent medium (SM) and eluate, respectively. Fig.12 c) shows that the agent can control the column inlet flow rates of feed, elution and wash. Fig.12 d) shows the column outlet flow to the product container 540 or the waste container 542 , which the agent can control by controlling the output valve 530 . Fig.12 Figures e) and f) show how much mAb and spent media end up in the product container 540 and waste container 542, respectively.
[0344] exist Fig.12In the example, the agent acts on a system with the same columns as the initial use case in Case Study 1 above (see Fig. 9 ). For example, from Fig.12 c) It can be seen that the recurrent mode, which performs better in flow control, has a control scheme that is less unstable than the RL agent in Case Study 1 (see Fig. 9 c)). From Fig.12 c), the agent appears to complete four load-wash-elute cycles within the specified time frame. Note further that the agent in this case study 2 appears to have managed to increase the column loading rate by combining the wash flow with the feed flow. Fig.14 This results in faster loading allowing shorter cycle times when compared to the reference case described below. Fig.14 Compared to the conventional process with constant flow described in c), the agent in this case study 2 is able to intelligently use the various control interfaces provided to improve productivity.
[0345] To highlight the RL agent’s adaptation to the new column, Fig.13 shows that when the RL agent is presented with Fig.12 The control process of the RL agent during the different columns of the experiment. Specifically, Fig.13 The control process of the RL agent over a simulated chromatography device is shown when the RL agent is presented with different columns. Fig.13 a) and b) show the column inlet and column outlet concentrations of mAb, spent medium (SM) and eluate, respectively. Fig.13 c) shows that the agent can control the column inlet flow rates of feed, elution and wash. Fig.13 d) shows the column outlet flow to the product container 540 or the waste container 542 , which the agent can control by controlling the output valve 530 . Fig.13 Figures e) and f) show how much mAb and spent media end up in the product container 540 and waste container 542, respectively.
[0346] from Fig.13 It can be clearly seen that Fig.12 Compared to the experimental results shown, the agent has now produced a control scheme that completes only three (and half) load-wash-elute cycles, where the agent Fig.12 The experimental results shown have been completed over four cycles.
[0347] Fig.14 A simulated benchmark strategy is shown for reference. Fig.14 Column inlet and outlet concentrations of mAb, spent medium (SM) and eluate are shown in a) and b), respectively. Fig.14c) shows the constant flow switching between the three stages of feed, wash and elution. Fig.14 d) shows the valve control of the column outlet flow to the product container 540 or the waste container 542. Fig.14 Figures e) and f) show the recovered weight in the product container 540 and the waste container 542, respectively. Note that the reference case with a lower flow rate achieves only three cycles in the time frame.
[0348] This case study2 has shown that an RL agent can learn to control and adapt to a chromatography system that has variability in column properties. This is an important finding because conventional recipes are experimentally determined per column and cannot be generalized to new columns. This deep RL workflow highlights the possibility of a generally applicable control scheme that adapts to column variability by relying on sensor data.
[0349] It is also noted that unforeseen behavior of the agent was obtained in this case study 2, where the agent used flow control to increase the amount of processed product in the experiment. Therefore, the control device according to the present disclosure is able to adapt autonomously to various conditions. Subsequent adjustments of the reward can be used to optimize the purity or recovery of the product, where the current reward weights are 125% increase in productivity and 18% decrease in purity compared to the expert strategy. It can be assumed that further improvements in purity are not limited by the ability of the algorithm to learn how to control the system, but rather by the shaping of the reward function.
[0350] Such results can be improved by further exploration of deep learning architectures, RL algorithms, hyperparameter tuning, and perhaps most importantly, further engineering of the reward function. Furthermore, it can be assumed that the RL agent benefits from the randomized bar parameters.
[0351] Variants
[0352] It should be noted that the methods and / or systems according to the present disclosure can be applied to control chromatographic processes other than the specific LC process used in the above case studies. For example, the final reward function used in the above case studies 1 or 2 can also be used to generate a strategy to control the LC process of target compounds other than mAbs in case studies 1 and 2.
[0353] In addition, for example, chromatographic processes other than liquid chromatography processes can also be controlled by methods and / or systems according to the present disclosure. Examples of different types of chromatographic processes may include, but are not limited to, high pressure liquid chromatography processes, gas chromatography processes, ion exchange chromatography processes, affinity chromatography processes, membrane chromatography processes, continuous chromatography processes, affinity monolithic chromatography (AMC) liquid chromatography using monolithic carriers and bio-related binding agents as stationary phases, etc. When controlling different types of chromatographic processes, states and / or control parameters other than those used in case studies, or those states and / or control parameters described above as exemplary states and / or control parameters, may be considered. For example, for ion exchange chromatography, salt ion concentration may be included as a state parameter and / or control parameter. In addition, for example, for affinity chromatography, pH may be included as a state parameter and / or control parameter. In addition, for high pressure liquid chromatography, particularly for anion exchange high pressure liquid chromatography, separation of mRNA / pDNA / dsRNA / ssRNA synthesis, salt gradient and pH may be included as state parameters and / or control parameters. On the other hand, for controlling membrane chromatography, the state and control parameters used for the case study can be used similarly.
[0354] In addition, in exemplary embodiments using reinforcement learning or deep reinforcement learning, the reward function may be different from the final reward function used in the case study described above. For example, the reward function may also include a penalty term for an action that is too different from a previous action to drive the agent to take a smoother action (e.g., slowly adjusting a valve rather than quickly opening and closing it). Other factors may also be considered in the reward function, such as the time required to process a certain amount of feed and / or the use of compounds with specific interaction properties (e.g., the use of elution buffer, less use of which may be preferred due to its cost). When such other factors are incorporated into the reward function, specific process measurements may be made (e.g., in addition to the value of the state parameter and the amount of material in the product container and the waste container) and used to calculate the reward.
[0355] Furthermore, although the above exemplary embodiments relate to Figure 2 The exemplary simulation environment 50 shown in includes a single column (chromatographic bed), but methods and / or systems according to the present disclosure may be applied to control a chromatography system, which may be simulated using another exemplary simulation environment that models multi-column chromatography. Fig.15 A schematic diagram illustrating such a further exemplary simulation environment is shown. The exemplary simulation environment may be implemented by the processor 104 of the chromatography system 10. Fig.15In the specific example shown, the simulation environment includes a module representing a column simulation, which includes four columns (column 1, column 2, column 3 and column 4) connected to each other, four flow controllers (flow controller 1, flow controller 2, flow controller 3 and flow controller 4) corresponding to the four columns, four output valves (output valve 1, output valve 2, output valve 3, output valve 4) corresponding to the four columns, a feed medium container, an elution buffer container, a wash buffer container, a product container and a waste container. Fig.15 The physical system (e.g., chromatographic device 102) recreated in the exemplary simulation environment shown may include Fig.15 The physical components of the modules in the exemplary simulation environment shown. For example, Fig.15 The illustrated exemplary simulation environment or corresponding physical system may be used to simulate or physically perform periodic countercurrent chromatography.
[0356] Note that although Fig.15 Although not shown, in some exemplary embodiments, the simulation environment and corresponding physical system may include more than one product container and / or more than one waste container. Fig.15 The exemplary simulation environment shown includes four columns and corresponding flow controllers and corresponding output valves, but in other cases, the simulation environment and corresponding physical system may include more or less than four columns and corresponding flow controllers and corresponding output valves.
[0357] In addition, despite Figure 1 A single chromatography system 10 is shown, and case studies 1 and 2 involve a single chromatography system for training and control, but in some exemplary embodiments, the control device 20 can also control multiple chromatography systems 10. The multiple chromatography systems 10 may include one or more systems that physically perform the chromatography process (thus, they can be referred to as "physical systems") and one or more systems that simulate the chromatography process (thus, they can be referred to as "simulation systems"). Some of the multiple chromatography systems 10 can be used only to generate policies, while others can be controlled by the generated policies. As a specific example, among the multiple chromatography systems 10, all simulation systems and at least one physical system can be used to generate policies, and physical systems that are not used to generate policies can be controlled by the generated policies. As another specific example, multiple chromatography systems 10 may include a physical system and a simulation system, the simulation system uses a simulation environment (digital twin) that models the physical system, generates policies using the simulation system, and then uses the generated policies to control the physical system.
[0358] Hardware Configuration
[0359] Fig.16An exemplary hardware configuration of a computer that can be used to implement at least part of the above system is shown. For example, Figure 1 At least a portion of the chromatographic system 10 and / or the control device 20 shown may be used Fig.16 The computer 7 shown is used to implement it. Fig.16 The computer 7 shown includes a central processing unit (CPU) 70, a system memory 72, a network interface 74, a hard disk drive (HDD) interface 76, an external disk drive interface 78, and an input / output (I / O) interface 80. These components of the computer are connected to each other via a system bus 82. The CPU 70 can perform arithmetic, logic, and / or control operations by accessing the system memory 72. The system memory 72 can store information and / or instructions used in conjunction with the CPU 70. The system memory 72 can include volatile and non-volatile memory, such as a random access memory (RAM) 720 and a read-only memory (ROM) 722. A basic input / output system (BIOS) containing routines such as those that help transfer information between elements within the computer 7 during startup can be stored in the ROM 722. The system bus 82 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
[0360] The computer may include a network interface 74 for communicating with other computers and / or devices via a network.
[0361] In addition, the computer may include a hard disk drive (HDD) 84 for reading from and writing to a hard disk (not shown), and an external disk drive 86 for reading from and writing to a removable disk (not shown). The removable disk may be a disk for a disk drive or an optical disk (e.g., CD ROM) for an optical disk drive. The HDD 84 and the external disk drive 86 are connected to the system bus 82 via the HDD interface 76 and the external disk drive interface 78, respectively. The drive and its associated computer-readable medium provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for a general-purpose computer. The data structure may include relevant data for implementing the exemplary methods described herein and variants thereof. The relevant data may be organized in a database, such as a relational database or an object database.
[0362] Although the exemplary environment described herein employs a hard disk (not shown) and an external disk (not shown), those skilled in the art will appreciate that other types of computer-readable media capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, digital video disks, random access memory, read-only memory, etc., may also be used in the exemplary operating environment.
[0363] A number of program modules may be stored on the hard disk, external disk, ROM 722 or RAM 720, including an operating system (not shown), one or more application programs 7202, other program modules (not shown) and program data 7204. The application program may include at least part of the functionality described above.
[0364] The computer 7 may be connected to an input device 92 such as a mouse and / or a keyboard and a display device 94 such as a liquid crystal display via corresponding I / O interfaces 80a and 80b and a system bus 82. In the case where the computer 7 is implemented as a tablet computer, for example, a touch panel that displays information and receives input may be connected to the computer 7 via corresponding I / O interfaces and the system bus 82. In addition, in some examples, although Fig.16 Not shown, the computer 7 may also be connected to a printer and / or an imaging device such as a camera via corresponding I / O interfaces and the system bus 82.
[0365] Except for use Fig.16 In addition to the implementation of the computer 7 shown, or as an alternative to the implementation using the computer 7, some or all of the functions of the exemplary embodiments described herein may be implemented as one or more hardware circuits. Examples of such hardware circuits may include, but are not limited to: large-scale integrated circuits (LSI), reduced instruction set circuits (RISC), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).
Claims
1. A computer-implemented method for controlling a chromatography system (10), the chromatography system (10) being configured to physically perform and / or simulate a chromatography process, the method include: obtaining a current state of the chromatography system (10) from the chromatography system (10), the current state comprising one or more values of one or more state parameters, the one or more state parameters comprising one or more amounts of one or more substances present in the chromatography system (10); determining the one or more values of the one or more control parameters according to a strategy configured to map the current state to a corresponding action representing one or more values for the one or more control parameters of the chromatography system (10), the one or more control parameters comprising at least: a position of a valve included in the chromatography system (10) and / or a pump speed of a pump included in the chromatography system (10); and controlling the chromatography system (10) using the one or more determined values of the one or more control parameters, wherein the strategy is generated according to a machine learning algorithm trained using state-action pairs; and The state-action pairs are obtained at least in part by physically executing and / or simulating the chromatography process by the chromatography system (10), each of the state-action pairs comprising: The state (S) of the chromatography system (10) t ), comprising one or more values of said one or more state parameters at a specific point in time (t); and Action t ), including one or more values of the one or more control parameters, the chromatography system (10) responds to the state (S t ), using the action (A t ) is controlled by one or more values in ).
2. The method according to claim 1, further comprising: include: receiving the state-action pair; as well as The policy is generated according to the machine learning algorithm using the received state-action pairs.
3. The method according to claim 1 or 2, further comprising: include: The strategy is updated using the current state and the corresponding action including the one or more determined values of the one or more control parameters.
4. The method according to any one of claims 1 to 3, in, The one or more amounts of the one or more substances present in the chromatography system (10) include one or more of the following: one or more amounts of one or more substances flowing into one or more chromatography beds (520) included in the chromatography system (10); one or more amounts of one or more substances flowing out of the one or more chromatographic beds (520); one or more amounts of one or more substances within the one or more chromatographic beds (520); one or more amounts of one or more substances within one or more of the containers included in the chromatography system (10), Therein, the one or more state parameters may include at least one parameter based on two or more of the quantities listed above.
5. The method according to any one of claims 1 to 4, in, The one or more control parameters also include one or more flow rates of one or more media flowing into and / or out of one or more of the following: at least one of the one or more chromatography beds (520) of the chromatography system (10); at least one of the containers included in said chromatography system (10); at least one of the one or more flow controllers included in the chromatography system (10); and The one or more control parameters may further include one or more of the following items: the temperature in the chromatography system (10); the pH of the mobile phase in the chromatography system (10); The salinity of the mobile phase in the chromatography system (10).
6. The method according to any one of claims 1 to 5, in, The one or more status parameters also include one or more of the following: the temperature at a specified point of the chromatography system (10); the pH of the medium in the specified portion of the chromatography system (10); one or more parameters related to specifications of the one or more chromatography beds (520), one or more containers (510, 512, 514, 540, 542) included in the chromatography system (10), and / or one or more connections between the containers (510, 512, 514, 540, 542); one or more maximum flow rates of one or more media flowing into the one or more chromatographic beds (520); one or more upstream parameters; one or more parameters related to the feed medium, wash medium and / or elution medium used in the chromatographic process; Conductivity: absorption of effluent; Target protein content; concentration of coeluting contaminants; Product concentration; purity; Yield.
7. The method according to any one of claims 1 to 6, in, The machine learning algorithm includes one of the following, or a combination of two or more: Reinforcement learning; Deep reinforcement learning; Supervised learning; Semi-supervised learning; Self-supervised learning; Learning by imitation; Transfer learning.
8. The method according to claim 7, in, The machine learning algorithm includes the reinforcement learning or the deep reinforcement learning; as well as The reward in the reinforcement learning or deep reinforcement learning is calculated using one or more of the following items: at least one of the one or more values of the one or more state parameters; a value representing the flow rate of a target compound flowing into one or more chromatographic beds of the chromatographic system (10); a value representing the flow rate of the target compound out of the one or more chromatographic beds of the chromatographic system (10); a value representing an amount of a target compound for the chromatography process in or flowing into a product container (540) included in the chromatography system (10); a value representing the amount of a target compound in a waste container (542) included in the chromatography system (10) or flowing into a waste container (542) included in the chromatography system (10); a value representing an amount of spent medium in or flowing into the product container (540), the spent medium including substances other than the target compound; A value representing the amount of the spent media in or flowing into the waste container (542).
9. The method according to claim 8, in, The machine learning algorithm includes the deep reinforcement learning, which involves an actor-critic method that uses: an actor network (32) comprising a first neural network to be trained to represent the policy, the first neural network configured to take states as input and actions as output; as well as An evaluator network (34) comprising a second neural network to be trained to estimate the reward obtained by a state-action pair, the evaluator network (34) being used to train the actor network (32) to output actions that produce high rewards in response to states input to the actor network (32).
10. The method according to any one of claims 7 to 9, in, The machine learning algorithm includes the supervised learning or the semi-supervised learning; as well as Wherein at least part of the state-action pairs are defined by an expert of the chromatographic process.
11. The method according to any one of claims 1 to 10, in, The chromatography system (10) comprises: a chromatography device (102) configured to physically perform the chromatography process; and a simulation system configured to simulate the chromatography process and implemented by a processor (104) and a storage medium (106), Wherein, the controlling of the chromatography system (10) comprises: controlling the chromatography device (102) included in the chromatography system (10) using the one or more determined values of the one or more control parameters.
12. A computer-implemented method for configuring a control device (20) for controlling a chromatography system (10), the chromatography system (10) being configured to physically perform and / or simulate a chromatography process, the method include: receiving state-action pairs obtained at least in part by physically executing and / or simulating the chromatography process by the chromatography system (10), each of the state-action pairs comprising: The state (S) of the chromatography system (10) t ), comprising one or more values of one or more state parameters at a specific point in time (t), the one or more state parameters comprising one or more amounts of one or more substances present in the chromatography system (10); and Action t ), comprising at least one or more values of one or more control parameters for the chromatography system (10), the chromatography system (10) being responsive to the state (S t ), Use the action included in (A t ), wherein the one or more control parameters at least include: a position of a valve included in the chromatography system (10) and / or a pump speed of a pump included in the chromatography system (10); generating, based on a machine learning algorithm and using the received state-action pairs, a policy that maps a current state including one or more values of the one or more state parameters to a corresponding action including one or more values of the one or more control parameters; and The generated strategy is stored in a storage medium (204) included in the control device (20).
13. A computer program product comprising computer-readable instructions, which, when loaded on a computer and executed on the computer, cause the computer to perform the method according to any one of claims 1 to 12.
14. A control device (20) for controlling a chromatography system (10), the chromatography system (10) being configured to perform and / or simulate a chromatography process, the control device (20) include: A processor (202) configured to execute the method according to any one of claims 1 to 11; as well as A storage medium (204) is configured to store the policy.
15. A system, include: A chromatography system (10) configured to perform and / or simulate a chromatography process; as well as The control device (20) according to claim 14, wherein the control device (20) is connected to the chromatography system (10).
