Using causal models to make biopharmaceuticals

Abstract

The present invention provides methods, systems and apparatus for optimizing a process for manufacturing a biopharmaceutical, including a computer program encoded on a computer storage medium. In one aspect, the method includes repeatedly performing the following operations: i) selecting a configuration of input settings for manufacturing a batch of a biopharmaceutical based on a causal model that measures the current causal relationship between the input settings and a measure of quality of the batch of the biopharmaceutical; ii) determining a measure of quality of the batch of the biopharmaceutical manufactured using the configuration of the input settings; and iii) adjusting the causal model based on the measure of quality of the batch of the biopharmaceutical.

Description

Background Art

[0001] The present description relates to controlling one or more biopharmaceutical sites and to determining a cause-and-effect relationship between control settings used to manufacture a biopharmaceutical at the site and an environmental response received from the biopharmaceutical site.

[0002] Existing techniques for determining which control settings should be used to control an environment typically employ modeling-based techniques or rely on active control of the system.

[0003] In modeling-based techniques, the system passively observes data, i.e., historical mappings of control settings to environmental responses, and attempts to discover patterns in the data to learn a model that can be used to control the environment. Examples of modeling-based techniques include decision forests, logistic regression, support vector machines, neural networks, kernel machines, and Bayesian classifiers.

[0004] In active control techniques, the system relies on active control of the environment for knowledge generation and application. Examples of active control techniques include randomized controlled experiments, such as the Bandit experiment. Summary of the invention

[0005] This specification describes systems and methods implemented as a computer program on one or more computers in one or more locations for selecting control settings for manufacturing a biopharmaceutical.

[0006] According to a first aspect, a method is provided, the method comprising repeatedly performing the following operations: i) selecting a configuration of input settings for manufacturing a biopharmaceutical batch based on a causal model, the causal model measuring a current causal relationship between the input settings and a measure of quality of the biopharmaceutical batch; ii) determining a measure of quality of the biopharmaceutical batch manufactured using the configuration of input settings; and iii) adjusting the causal model based on the measure of quality of the biopharmaceutical batch.

[0007] In some implementations, the method further includes selecting a configuration of the input settings based on a set of internal control parameters, and adjusting the internal control parameters based on a measure of quality of the batch.

[0008] In some implementations, the measure of the quality of the batch includes one or more of the following: the yield of the batch; the measure of the activity of the batch; the measure of the efficacy of the batch; the measure of impurities in the batch; the measure of the viability of the transfected cells; the measure of the transfection efficiency; the measure of the purity of the biopharmaceutical; the measure of monodispersity; or the measure of cell density. In some implementations, the purity of the biopharmaceutical includes one or more of the following: the measure of host cell protein impurities; the measure of DNA impurities; the measure of RNA impurities; or the measure of host cell metabolite impurities.

[0009] In some implementations, the input settings include one or more of: one or more settings associated with the processing steps; one or more settings associated with the chemical composition of the substance; one or more settings associated with physical control during manufacturing; or one or more settings associated with purifying the biopharmaceutical batch. In some implementations, the settings associated with the processing steps include one or more of: the order of the processing steps; the time spent in each processing step; the time spent in each subprocess of a given processing step; or the selection of parameters for each processing step. In some implementations, the settings associated with the chemical composition include the selection of one or more of: nutrient medium; ion exchange resin; affinity chromatography matrix; polymer; wash buffer; or elution buffer. In some implementations, the settings associated with physical control during manufacturing include one or more of: manufacturing time; purification time; temperature during manufacturing; temperature during purification; humidity during manufacturing; humidity during purification; flow rate used during manufacturing; flow rate used during purification; g-force used during manufacturing; measure of magnetic force used during manufacturing; measure of electrical power used during manufacturing; or measure of agitation used during manufacturing. In some implementations, a configuration associated with purifying a batch of a biopharmaceutical includes purifying the batch using one or more of: electrostatic interactions; hydrophobic interactions; hydrophilic interactions; affinity chromatography; or size exclusion chromatography.

[0010] In some implementations, the method further includes selecting a configuration based on the causal model and a predetermined set of external variables, and the method further includes adjusting a control parameter that parameterizes the effects of the predetermined set of external variables. In some implementations, the predetermined set of external variables includes one or more external variables related to ambient climate during manufacturing. In some such implementations, the external variables related to ambient climate include one or more of: humidity during manufacturing; external temperature during manufacturing; or light levels during manufacturing.

[0011] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0012] Use of the methods described herein allows for rapid improvements in the manufacture of biopharmaceuticals. By repeatedly selecting different control settings and measuring the effects of the control settings on the quality of the biopharmaceutical, the control system is able to generate a causal model that models the causal relationship between the control settings and the quality of the biopharmaceutical more quickly and more accurately than other prior art control systems.

[0013] The control system can also take into account characteristics of the environment that are uncontrollable but affect the quality of the biopharmaceutical. Thus, the causal model can independently model the relationship between the control settings and the quality of the biopharmaceutical for various configurations of the environmental characteristics, so that the quality of the biopharmaceutical may be less susceptible to changes in those characteristics.

[0014] In some implementations, the control system can continue to operate and use the causal model to select control settings for manufacturing the biopharmaceutical. Thus, the system can continuously update the causal model while also utilizing the causal model to optimize the quality of the biopharmaceutical. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A A control system for selecting control settings for application to a biopharmaceutical site is shown.

[0016] Figure 1B Data from an exemplary causal model are shown.

[0017] Figure 2 is a flow chart of an exemplary process for controlling an environment.

[0018] Figure 3 is a flow chart of an exemplary process for performing iterations of environmental control.

[0019] Figure 4A is a flow chart of an exemplary process for determining a program instance.

[0020] Figure 4B An example of an environment including a plurality of physical entities each associated with a spatial extent is shown.

[0021] Figure 5 is a flow chart of an exemplary process for selecting control settings for a set of current instances.

[0022] Figure 6 is a flow chart of an exemplary process for updating a causal model for a given controllable element and a given type of environmental response.

[0023] Figure 7 is a flow chart of an exemplary process for clustering a set of program instances for a given controllable element.

[0024] Figure 8 is a flow chart of an exemplary process for updating a set of internal parameters using random variations.

[0025] Fig. 9 is a flow chart of an exemplary process for updating the value of a data inclusion value for a given controllable element based on a heuristic.

[0026] Fig.10is a flow chart of an exemplary process for responding to changes in one or more characteristics of an environment.

[0027] Fig.11 A representation of the data inclusion window for a given controllable element of an environment is shown when a set of internal parameters defining the data inclusion are randomly varied.

[0028] Fig.12 The performance of the system in controlling an environment is shown relative to the performance of a system controlling the same environment using an existing control scheme.

[0029] Fig.13 The performance of the system is shown relative to the performance of multiple other systems in controlling multiple different environments.

[0030] Fig.14 The performance of the system is shown relative to the performance of multiple other systems in controlling multiple different environments with different temporal effects.

[0031] Fig.15 The performance of the system with and without clustering is shown.

[0032] Fig.16 The performance of the system having the ability to vary data inclusion is shown relative to the performance of the system controlling the same environment while keeping the data inclusion window parameters fixed.

[0033] Fig.17 The performance of the system is shown with and without temporal analysis (ie, with and without the ability to change the time range).

[0034] Fig.18 The performance of the system in controlling an environment is shown relative to the performance of a system controlling the same environment using an existing control scheme ("ucb_lin").

[0035] Throughout the various drawings, like reference numbers and designations refer to like elements. DETAILED DESCRIPTION

[0036] This specification generally describes a control system that controls an environment as the environment changes state. In particular, the system controls the environment to determine the cause-and-effect relationship between the control settings of the environment and the environmental response to the control settings. Specifically, the environment is one or more biopharmaceutical sites. The biopharmaceutical site is a location where a biopharmaceutical is manufactured, such as a manufacturing plant or purification plant for a biopharmaceutical. Here, a biopharmaceutical is defined as any drug product manufactured in a biological source (e.g., a vaccine, a blood component, a gene therapy, or a living cell for cell therapy), extracted from a biological source, or semi-synthesized from a biological source. The system selects control settings for manufacturing (including purification) a batch of biopharmaceuticals at the site. The environmental response is a measure of the quality of a biopharmaceutical manufactured using the input settings selected by the system.

[0037] For example, a measure of the quality of a biopharmaceutical for which a causal relationship is being determined may include: (i) sensor readings or other environmental measurements that reflect the state of the manufactured biopharmaceutical, (ii) a performance metric that measures the performance of a control system based on the environmental measurements, such as a figure of merit or an objective function, or (iii) both.

[0038] In particular, the control system repeatedly selects control settings, each control setting comprising a respective setting for each controllable element in a set of controllable elements of the biopharmaceutical site. In general, selecting different control settings results in differences in system performance, ie, different values ​​of environmental response.

[0039] More specifically, by repeatedly selecting control settings and measuring the effects of the control settings on measures of the quality of the biopharmaceutical, the control system updates a causal model that models the causal relationship between the control settings and the environmental response, i.e., updates the maintained data that identifies the causal relationship between the control settings and system performance.

[0040] Although the causal model is referred to as a “causal model”, in some implementations the model may consist of multiple causal models, each causal model corresponding to a different segment of the environment, ie, to segments of the environment that share certain characteristics.

[0041] In some implementations, the control system may continue to operate and use the causal model to select control settings for the biodrug site. In other implementations, once certain criteria are met, the control system may provide the causal model to an external system, or may provide data showing the causal relationships identified in the causal model to a user for use in controlling the biodrug site. For example, the criteria may be met after the system has controlled the biodrug site for a certain amount of time or has selected settings a certain number of times. For another example, the criteria may be met when the causal relationships identified in the retained data meet certain criteria, such as having non-overlapping confidence intervals.

[0042] While updating the causal model, the system repeatedly selects different control settings based on internal parameters of the control system and characteristics of the biopharmaceutical site and measures the effect of each possible control setting on a measure of quality of the biopharmaceutical manufactured using the control setting.

[0043] In other words, the internal parameters of the control system define both: (i) how the system updates the causal model, and (ii) how the system determines which control settings to select given the current causal model. While updating the causal model, the control system also repeatedly adjusts at least some of the internal parameters as more environmental responses become available to help identify causal relationships.

[0044] Figure 1A The control system 100 is shown selecting control settings 104 to be applied to a biopharmaceutical site 102. Each control setting 104 defines a setting for each of a plurality of controllable elements of the biopharmaceutical site 102. Generally speaking, the controllable elements of the biopharmaceutical site 102 are those elements that can be controlled by the system 100 and that can adopt a plurality of different possible settings.

[0045] The control settings 104 may include settings related to the steps for processing the biopharmaceutical. For example, the control settings 104 may include the order of the processing steps, the time spent in a given processing step, and / or the selection of parameters for a given processing step. The control system 100 may also determine the time spent in one or more sub-processes of a given processing step.

[0046] The control settings 104 may also include settings related to the chemical composition of a given substance. The substance may be the biopharmaceutical itself, or it may be any other substance used during the manufacture of the biopharmaceutical. For example, the control settings 104 may include the selection of a nutrient medium, an ion exchange resin, an affinity chromatography matrix, a polymer, a wash buffer, and / or an elution buffer.

[0047] The control settings 104 may also include settings related to physical controls during manufacturing, i.e., controlling the environment and parameters of the manufacturing process. For example, the control settings 104 may include the manufacturing time and / or purification time of the biopharmaceutical. The control system 100 may also determine the temperature during manufacturing and / or purification, the humidity during manufacturing and / or purification, or the flow rate used during manufacturing and / or purification. The control settings 104 may also include the g-force used during manufacturing, the measure of magnetic force used during manufacturing, the measure of electrical power used during manufacturing, and / or the measure of agitation used during manufacturing.

[0048] The control settings 104 may also include settings related to purifying the biopharmaceutical. In these cases, the purification process may be performed after a separate manufacturing process, during a manufacturing process, or solely, i.e., a given biopharmaceutical site uniquely purifies the biopharmaceutical. The control system 100 may select a method for purifying the biopharmaceutical, for example, the control settings may include selection of electrostatic interactions, hydrophobic interactions, hydrophilic interactions, affinity chromatography, and / or size exclusion chromatography.

[0049] During operation, the control system 100 repeatedly selects the control setting 104 and monitors the environmental response 130 to the control setting 104. The environmental response 130 can be measured by one or more measures of the quality of the batch of biopharmaceuticals manufactured using the control setting 104. For example, the measure of quality can include the yield of the batch, the measure of the activity of the batch, the measure of the efficacy of the batch, and / or the measure of impurities in the batch. If the transfection is part of the manufacturing process of the biopharmaceutical, the response can also include a measure of the viability of the transfected cells or a measure of the transfection efficiency. The measure of quality can also include a measure of monodispersity (e.g., the dispersibility of non-aggregated proteins) and / or a measure of cell density.

[0050] As a specific example, a measure of the quality of a batch of a biopharmaceutical may include a measure of the purity of the biopharmaceutical. Here, the purity of the biopharmaceutical may include a measure of host cell protein impurities, a measure of DNA impurities, a measure of RNA impurities, and / or a measure of host cell metabolite impurities.

[0051] The system may calculate a performance metric for the environmental response 130, ie, may calculate a single value representing the performance of the system in controlling the environment to maximize the quality of the biopharmaceutical. An example performance metric combining all quality metrics used by the system is a weighted sum of the values ​​of the selected quality metrics.

[0052] As another example, for each success metric in the quality metrics, the performance metric may be a weighted sum of the differences between the quality metric and a baseline or expected value of the success metric, i.e., such that the system attempts to minimize deviations outside of acceptable values ​​for each of the quality metrics. Another example of such a performance metric is a weighted sum of the following functions: for each success metric in the quality metrics, if the quality metric is within an acceptable range, the function is zero, and if the quality metric is outside the acceptable range, the function is equal to the distance from the quality metric to the nearest endpoint of the acceptable range.

[0053] The system 100 also monitors characteristics 140 of the environment 102. Generally speaking, the characteristics 140 may include any data characterizing the environment that may modify the effect of the control setting 104 on the environmental response 130, but which is not considered in the control setting, i.e., is not controllable by the control system 100. For example, the environmental characteristics 140 of the biopharmaceutical site 102 may include humidity during manufacturing, external temperature during manufacturing, and / or light level during manufacturing when the corresponding measurements are not adjustable by the biopharmaceutical site 102. Note that in some cases, these measurements may be controllable by the biopharmaceutical site 102, for example, if the site 102 can adjust the lighting within the site. In these cases, the adjustable measurements will be included as control settings instead of the environmental characteristics 140.

[0054] The system 100 uses the environmental response 130 to update the causal model 110 , which models the causal relationship between the control settings and the environmental response, ie, how different settings of different elements affect the value of the environmental response.

[0055] In particular, for each controllable element of the biopharmaceutical site 102 and for each different type of environmental response, the causal model 110 measures the causal effects of different possible settings of the controllable element on the environmental response and the current level of uncertainty of the system regarding the causal effects of the possible settings.

[0056] As a specific example, for each different possible setting of a given controllable element and for each different type of environmental response, the causal model 110 may include: an impact measurement that represents the effect of the possible setting on the environmental response relative to other possible settings of the controllable element, e.g., an average estimate of the true average effect of the possible settings; and a confidence interval for the impact measurement, e.g., a 95% confidence interval that represents the current level of system uncertainty about the causal effect.

[0057] Thus, the system calculates confidence intervals that specify, for example, 95% upper and lower limits on the effect of a control setting on system performance. Specifically, this allows the system to identify when the selection of different control settings results in a (clinically) significant or insignificant difference. The system can avoid testing controllable elements that do not result in significant differences. For example, where the upper and lower limits of the confidence interval show that even the maximum effect would not result in a clinically meaningful difference, and where there is a cost to continue testing / exploring the controllable element, the system can seek authorization to remove the control setting. For a clinical example, imagine a confidence interval for some control setting on systolic blood pressure that shows the effect of the setting to be ±0.02 points. In the presence of any costs (e.g., health-related costs or other costs), the system can seek authorization to stop the experiment for continued experimentation with that setting because the cost of the experiment would outweigh any benefit that the 0.02 pressure reduction would have on the probability of a cardiac event.

[0058] Prior to commencing control of the biopharmaceutical site 102, the control system 100 receives an external input 106. The external input 106 may include data received by the control system 100 from any of a variety of sources. For example, the external input 106 may include data received from a user of the system, data generated by another control system that previously controlled the biopharmaceutical site 102, data generated by a machine learning model, or some combination of these data.

[0059] Generally speaking, external inputs 106 specify at least (i) initial possible values ​​for settings of controllable elements of biopharmaceutical site 102 and (ii) environmental responses that control system 100 tracks during operation.

[0060] For example, external input 106 may specify that control system 100 needs to track: measurements from certain sensors at a biopharmaceutical site, a performance metric derived from certain sensor measurements that system 100 is to optimize in controlling the biopharmaceutical site (i.e., a figure of merit or other objective function), or both.

[0061] The control system 100 uses the external inputs 106 to generate initial probability distributions over initial possible setting values ​​for the controllable elements ("baseline probability distributions"). By initializing these baseline probability distributions using the external inputs 106, the system 100 ensures that settings are selected that do not violate any constraints imposed by the external data 106 and that do not deviate from historical ranges of control settings that have been used to control the biopharmaceutical site 102, if desired by a user of the system 100. For example, if there are certain ranges of control settings that are known to be unsafe for the biopharmaceutical site 102, the external data 106 can define those ranges so that the system 100 never selects control settings within the unsafe range.

[0062] The control system 100 also uses the external inputs 106 to initialize, i.e., assign baseline values ​​to, a set of internal parameters 120. In general, the internal parameters 120 define how the system 100 selects control settings given the current causal model 110 (i.e., given the current causal relationships that have been determined by the system 100 and the system uncertainty about the current causal relationships). The internal parameters 120 also define how the system 100 updates the causal model 110 using the received environmental responses 130.

[0063] As will be described in more detail below, while updating the causal model 110, the system 100 updates at least some of the internal parameters 120. That is, while some of the internal parameters 120 may be fixed to an initialized baseline value during operation of the system 100, the system 100 repeatedly adjusts other of the internal parameters 120 during operation to allow the system to more effectively measure and, in some cases, exploit causal relationships.

[0064] In particular, to control a biopharmaceutical site, during operation, the system 100 repeatedly identifies program instances within the biopharmaceutical site based on the internal parameters 120 .

[0065] Each program instance is a collection of one or more entities within a biodrug site that are associated with a time window. An entity within a biodrug site is a subset of the biodrug site, i.e., a suitable subset or an inappropriate subset. In particular, an entity is a subset of the biodrug site for which an environmental response can be obtained and on which an applied control setting can have an impact.

[0066] For example, when a biopharmaceutical site includes multiple physical entities from which sensor measurements may be obtained, a given program instance may include an appropriate subset of the physical entities to which a set of control settings will be applied. The number of subsets into which entities within the biopharmaceutical site may be divided is defined by internal parameters 120.

[0067] In particular, how the system 100 divides entities into subsets at any given time during system operation is defined by internal parameters that define the spatial extent of control settings applied by the system to an instance. The spatial extent of an instance identifies the subset of the environment assigned to the instance, i.e., such that an environmental response obtained from that subset will be associated with the instance.

[0068] For example, a program instance may include one or more machines that operate using control settings 104; in this case, the spatial scope may define the number and type of machines in the program instance. The system 100 will obtain an environmental response to the selected control settings for a given group of machines. For example, as described above, the system 100 may select a control setting related to a measure of agitation applied by a machine, and then the system 100 may track a selected measure of the quality of a biopharmaceutical batch processed by the machine using the selected measure of agitation.

[0069] The length of the time window associated with an entity in any given program instance is also defined by internal parameters 120. In particular, the time window that the system assigns to any given program instance is defined by internal parameters that define the time range of the control settings applied by the system. This time window (i.e., the time range of the instance) defines which future environmental responses the system 100 will determine are caused by the control settings selected for the program instance.

[0070] As internal parameters 120 change during operation of system 100, instances generated by system 100 may also change. That is, as the system changes internal parameters 120, the system may modify how program instances are identified.

[0071] The system 100 then selects settings for each instance based on internal parameters 120 and, in some cases, environmental characteristics 140 .

[0072] In some cases, ie, when system 100 is exploring the space of possible settings, system 100 selects all instances based on a baseline probability distribution.

[0073] In other cases, i.e., when the system 100 is optimizing an objective function using the determined causal relationships, the system 100 selects settings for some instances ("hybrid instances") using the current causal model 110 while continuing to select settings for other instances ("baseline instances") based on the baseline probability distribution. More specifically, at any given time during operation of the system 100, the internal parameters 120 define the proportion of hybrid instances relative to the total number of instances.

[0074] For each instance, the system 100 also determines, based on the internal parameters 120 , which environmental responses 130 are to be associated with the instance, ie, used to update the causal model 110 .

[0075] The system 100 then sets the settings 104 for each of the instances and monitors the environmental responses 130 to the settings selected for the instances. The system 100 maps the environmental responses 130 to the impact measurements for each instance and uses the impact measurements to determine causal model updates 150 for updating the current causal model 110.

[0076] In particular, the system determines which historical program instances (and environmental responses 130 associated with the instances) should be considered by the causal model 110 based on the internal parameters 120, and determines causal model updates 150 based only on these determined historical program instances. Which historical program instances are considered by the causal model 110 is determined by a set of internal parameters 120 that define a data inclusion window. The data inclusion window specifies, at any given time, one or more historical time windows during which a program instance must have occurred in order for the causal model 110 to consider the outcome for that program instance, i.e., the environmental response 130 associated with that program instance.

[0077] For those internal parameters that the system 100 is changing, the system 100 also periodically updates 160 the data maintained by the system 100 for those internal parameters based on the causal model 110. In other words, when the causal model 110 changes during operation of the system 100, the system 100 also updates the internal parameters 120 to reflect the changes in the causal model 110. In the case where the system 100 assigns some control settings to utilize the current causal model 110, the system 100 may also use the difference between the system performance of the "hybrid" instance and the system performance of the "baseline" instance to determine the internal parameter updates 160.

[0078] Figure 1B Data from an exemplary causal model are shown. In particular, Figure 1B In the example of , the causal model is represented as a graph 180 that shows control settings (i.e., different possible settings of different controllable elements) on the x-axis and the causal effects of the control settings on the y-axis. In particular, for each possible setting of each controllable element, the causal model depicts an impact measure and a confidence interval around the impact measure.

[0079] These causal relationships are shown in more detail for a particular controllable element 192 in the element-specific graph 190. The element-specific graph 190 shows that there are five possible settings for the controllable element 192, where the possible settings are referred to as levels in the graph 190. For each of the five settings, the graph includes a bar representing the impact measurement and a representation of the confidence interval around the bar (as error bars around the impact measurement). Therefore, the information in the causal model for any given setting of the controllable element includes the impact measurement and the confidence interval around the impact measurement. For example, for the second setting 192 (represented as IV-LV2 in the figure), the graph 190 shows: a column 194, which indicates the impact measurement of the second setting; an upper bar 196 above the top of the column 194, which shows the upper limit of the confidence interval for the second setting; and a lower bar 198 below the top of the column 194, which shows the lower limit of the confidence interval for the second setting.

[0080] Although Figure 1B A single causal model is shown, but it will be appreciated from the description below that the system may maintain and update multiple different causal models for any given controllable element - one causal model per program instance cluster.

[0081] Figure 2 2 is a flow chart of an exemplary process 200 for controlling an environment. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., control system 100 of FIG. 1 ) can perform process 200.

[0082] The system assigns baseline values ​​to a set of internal parameters and assigns a baseline probability distribution to each of the controllable elements of the environment (step 202).

[0083] In particular, the system receives external data, such as from a user of the system or data derived from a previous control of the system environment by another system, and then uses the external data to assign baseline values ​​and generate probability distributions. Generally speaking, the external data specifies the initial constraints within which the system operates when controlling the environment.

[0084] In particular, the external data identifies possible control settings for each of the controllable elements in the environment. That is, for each of the controllable elements in the environment, the external data identifies which possible settings the system can select for the controllable element when controlling the system.

[0085] In some cases, the external data may specify additional constraints on the possible control settings, for example, the settings of certain controllable elements depend on the settings of other controllable elements, or certain entities may be associated only with a subset of the possible control settings for a given controllable element.

[0086] Thus, the external data defines a search space of possible combinations of control settings that the system can explore when controlling the environment.

[0087] In some implementations, these constraints can change during operation of the system.

[0088] For example, the system may receive additional external inputs that modify the range of possible control settings or spatial and temporal values ​​for one or more of the controllable elements.

[0089] As another example, if the system determines that an optimal setting for one of the controllable elements or one of the internal parameters is close to a boundary of a search space defined by external constraints, for example, if impact measurements in a causal model indicate that the optimal settings are those settings that are closest to one of the boundaries of the search space, the system may, for example, seek authorization from a system administrator or other user of the system to expand the space of possible values ​​for the controllable element or internal parameter.

[0090] As another example, if external data specifies that the possible settings of some controllable element can be any value within a continuous range, the system can initially discretize that range in one way, and then once the confidence interval indicates strongly enough that the optimal value is in one segment of the continuous range, modify the discretization to favor that segment.

[0091] As another example, if the system determines that a controllable element has no causal effect on the environmental response, for example, if the impact measurements of all possible settings of the controllable element are all likely to be zero, the system may seek authorization to remove the controllable element from the system's control.

[0092] The system then generates a baseline (or "prior") probability distribution over possible control settings for each of the controllable elements of the environment. For example, when the external data specifies only possible values ​​for a given controllable element and does not assign a priority to any of the possible values, the system can generate a uniform probability distribution over the possible values ​​that assigns an equal probability to each possible value. For another example, when the external data prioritizes certain settings for a given controllable element over other settings, such as based on historical results of controlling the environment, the system can generate a probability distribution that assigns a higher probability to the prioritized settings.

[0093] The system also assigns a baseline value to each of the system's internal parameters. In particular, the system's internal parameters include (i) a set of internal parameters that define the spatial extent of program instances generated by the system (referred to as "spatial extent parameters") and (ii) a set of internal parameters that define the temporal extent of program instances generated by the system (referred to as "spatial extent parameters").

[0094] In some cases where the system includes multiple entities, the system may maintain multiple separate sets of spatial extent parameters and temporal extent parameters for each of the multiple entities. In other cases where the system includes multiple entities, the system maintains only a single set of spatial and temporal extent parameters that applies to all of the multiple entities. In other cases where the system includes multiple entities, the system initially maintains a single set of spatial and temporal extent parameters, and during operation of the system, may switch to maintaining a separate set of spatial extent or temporal extent parameters if doing so results in improved system performance, i.e., if different entities respond to control settings differently than other entities.

[0095] Additionally, in some implementations, the system maintains multiple separate sets of time range parameters for different controllable elements.

[0096] The system also maintains (iii) a set of internal parameters (referred to as "data inclusion window parameters") that define the data inclusion window used by the system. In some embodiments, the system maintains a single set of data inclusion window parameters that apply to all controllable elements. In some other embodiments, the system maintains a separate set of data inclusion window parameters for each controllable element of the environment, i.e., to allow the system to use different data inclusion windows for different controllable elements when updating the causal model. As will be described in more detail below, in the case where the system has clustered program instances into more than one cluster, the system may (a) maintain a separate set of data inclusion window parameters per cluster, or (b) maintain a separate set of data inclusion window parameters per cluster and per controllable element, i.e., so that different clusters can use different data inclusion windows for the same controllable element.

[0097] In an implementation where the system utilizes a causal model, the internal parameters also include (iv) a set of internal parameters (referred to as "ratio parameters") that define the ratio of a hybrid instance to a baseline instance. In some implementations, the system maintains a single set of ratio parameters that apply to all controllable elements. In some other implementations, the system maintains a separate set of ratio parameters for each controllable element of the environment, i.e., to allow the system to use different ratios for different controllable elements when selecting control settings. As will be described in more detail below, in the case where the system has clustered program instances into more than one cluster, the system may (a) continue to maintain a single set of ratio parameters across all clusters, (b) maintain a separate set of ratio parameters per cluster, or (c) maintain a separate set of ratio parameters per cluster and per controllable element, i.e., so that different clusters can use different ratios when selecting control settings for the same controllable element.

[0098] In a specific implementation where the system clusters instances into multiple clusters, as will be described below, the internal parameters also include (v) a set of internal parameters that define the current clustering strategy (referred to as "clustering parameters").

[0099] Generally speaking, clustering parameters are or define hyperparameters of the clustering technique used by the system. Examples of such hyperparameters include the cluster size of each cluster, i.e., the number of program instances in each cluster, and environmental characteristics used to cluster program instances.

[0100] The system maintains a set of clustering parameters for each controllable element. That is, for each controllable element, the system uses different hyperparameters when applying the clustering technique to generate a cluster of program instances for the controllable element.

[0101] The internal parameters may also optionally include any of a variety of other internal parameters that affect the operation of the control system. For example, the internal parameters may also include a set of internal parameters that define how to update the causal model (e.g., a set of weights, each weight representing the relative importance of each environmental characteristic during propensity matching between program instances, which may be used to calculate a d score, as described below).

[0102] As described above, the system changes at least some of these internal parameters during operation.

[0103] For each internal parameter in a set of internal parameters that the system changes while controlling the environment, the system may change the value using (i) a heuristic-based approach, (ii) by randomly sampling the values ​​to optimize the quality factor of the internal parameter, or (iii) both.

[0104] For any sets of internal parameters that change only based on heuristics, the system maintains a single value for the internal parameters and repeatedly adjusts that single value based on the heuristics.

[0105] For any multiple sets of internal parameters that are varied by random sampling, the system maintains parameters defining a range of possible values ​​for the internal parameters, and maintains a causal model identifying a causal relationship between the possible values ​​for the internal parameters and a quality factor for the internal parameters. The quality factor for the internal parameters may be different from the performance metric used in the causal model for the control setting. For at least some of the instances at any given time, the system then selects a value from the range of possible values ​​based on the current causal model.

[0106] When the system uses heuristics in addition to random sampling to update a set of internal parameters, the system can use the heuristics to update the range of possible values. That is, the range of possible values ​​is updated by a heuristic-based method, while the causal model for values ​​within the range at any given time is updated by random sampling.

[0107] For any internal parameters that the system does not change while controlling the environment, the system may maintain a fixed range of values ​​and a fixed probability distribution over the fixed range of values, or may maintain a fixed single value that is always used during operation of the system.

[0108] Depending on what is included in the external data, the system assigns a baseline value to each internal parameter, which is either derived from the external data or a default value.

[0109] For example, the external data typically identifies a range of values ​​for the spatial extent and the temporal extent. For example, when the spatial extent is not fixed and is an internal parameter that can be changed by the system, the external data can specify a minimum and maximum value for the spatial extent. Similarly, when the temporal extent is not fixed and is an internal parameter that can be changed by the system, the external data can specify a minimum and maximum value for the temporal extent.

[0110] The system then uses the external data to: assign initial values ​​to the spatial range parameters so that these parameters are bounded by the range of values ​​specified in the external data; and assign initial values ​​to the temporal range parameters so that these parameters are bounded by the range of values ​​specified in the external data.

[0111] For other internal parameters, the system assigns default values. For example, the system may initialize the cluster parameter to indicate that the number of clusters is 1, i.e., so that there are no clusters at the beginning of the control environment, and may initialize the ratio parameter to indicate that there are no mixed instances, i.e., so that the system only explores at the beginning of the control environment. The system may also initialize the data inclusion window parameter to indicate that the data inclusion window includes all historical program instances that have been completed.

[0112] The system performs an initiation phase (step 204). During the initiation phase, the system selects control settings for the program instance based on the baseline probability distribution of the controllable elements and uses the environmental response to update the causal model. That is, as long as no historical causal model is provided as part of the external data, the system does not consider the current causal model when determining which control settings to assign to the program instance.

[0113] Instead, the system uses a baseline probability distribution to select control settings according to an allocation scheme that allows for efficient calculation of impact measurements, i.e., d scores, later. In other words, the allocation scheme selects control settings in a manner that takes into account the blocking scheme that the system uses to calculate impact measurements, i.e., control settings are assigned to different program instances that allow for identification of block groups later, so as to calculate impact measurements between block groups. The blocking scheme (and, accordingly, the allocation scheme) adopted by the system can be any of a variety of schemes that reduce the unexplained variability between different control settings. Examples of blocking schemes that the system can adopt include one or more of double-blind allocation, paired allocation, Latin square allocation, propensity matching, etc. In general, the system can use any appropriate blocking scheme that allocates program instances to block groups based on the current environmental characteristics of the entities in the program instances.

[0114] When one or both of the spatial range and the temporal range are changeable by the system, the system changes the spatial range parameter, the temporal range parameter, or both during the initialization phase so that values ​​of the spatial range and the temporal range that are more likely to result in sufficiently orthogonal program instances are more likely to be selected. A group of instances is considered orthogonal if a control setting applied to one of the instances in the group does not affect an environmental response associated with any of the other instances in the group.

[0115] References below Figure 3 The following describes in more detail the selection of control settings and the updating of the causal model during the initialization phase. Fig.11 Describes changing the spatial or temporal extent parameters in more detail.

[0116] In some implementations, the system continues in this initialization phase throughout the operation of the system. That is, the system continues to explore the space of possible control settings and compiles the results of the exploration in a causal model.

[0117] For example, the system may continue in this initialization phase when the system is updating the causal model with respect to multiple different environmental responses rather than with respect to a single figure of merit or objective function, i.e., when the system does not have a figure of merit or objective function to use when utilizing the causal model.

[0118] In some of these specific implementations, the system continues to explore the space of possible control settings while also adjusting certain initial parameters in the multiple sets of initial parameters based on the causal model, such as spatial range parameters, temporal range parameters, data inclusion window parameters, clustering parameters, etc.

[0119] In some other implementations, once the system determines that certain criteria are met, the system begins executing different phases. In these implementations, during the initialization phase, the system keeps certain of the internal parameters fixed. For example, the system may keep the data inclusion window parameter fixed to indicate that all historical instances should be incorporated into the causal model. For another example, the system may keep the clustering internal parameter fixed to indicate that clustering should not be performed.

[0120] In particular, in these other implementations, once the system determines that the criteria are met, the system can begin executing the utilization phase (step 206).

[0121] For example, once the amount of program instances for which environmental responses have been collected exceeds a threshold, the system can begin executing the exploitation phase. As a specific example, the system can determine that the threshold is met when the total number of such program instances exceeds the threshold. As another specific example, the system can determine that the threshold is met when the minimum number of environmental responses associated with any one possible setting of any controllable element exceeds the threshold.

[0122] Additionally, in some cases, the system does not employ the initialization phase and proceeds immediately to the utilization phase, ie, step 204 is not performed.

[0123] When a threshold is used, the system can determine the threshold in any of a variety of ways.

[0124] For example, the system can determine that a threshold is satisfied when environmental responses have been collected for enough instances such that assigning settings to the instances based on the causal model results in different settings having different probabilities of being selected. Figure 5Describe in more detail how to assign likelihoods based on a causal model.

[0125] As another example, the system may determine that the threshold is the number of program instances required for a statistical test performed by the system to determine the confidence interval to produce an accurate confidence interval, ie, the number of program instances that satisfy the statistical assumptions of the confidence calculation.

[0126] As another example, the system can determine that the threshold is equal to the number of program instances required for the causal model to produce a desired statistical power (ie, as determined by a power analysis).

[0127] During the utilization phase, the system selects control settings for some of the program instances based on the current causal model while continuing to select control settings for other program instances based on the baseline values ​​of the internal parameters.

[0128] In particular, the system changes a ratio internal parameter so that the ratio between how many program instances should be hybrid instances (i.e., instances to which control settings are assigned based on the causal model) and how many program instances should be baseline instances (i.e., instances to which control settings are assigned based on the baseline probability distribution) is greater than zero.

[0129] Because the system begins designating certain instances as hybrid instances during the exploitation phase, the system can begin using system performance differences between hybrid instances and exploratory instances to adjust values ​​of internal parameters (e.g., ratio internal parameters, data inclusion window parameters, etc.).

[0130] References below Figure 3 Selecting control settings, updating causal models, and updating internal parameters while in the exploitation phase are described in more detail.

[0131] In some implementations, once the system determines that certain criteria are met, the system begins the clustering phase (step 208). That is, if the system is configured to cluster program instances, then once the criteria for clustering are met, the system begins the clustering phase. If the system is not configured to cluster instances, then at no time during the operation of the system does the system cluster program instances.

[0132] Generally speaking, the system considers clustering to create subgroups of similar program instances. In the case of the real world, different program instances across groups may respond to different control settings in different ways. The optimal control setting for a program instance may be suboptimal for another program instance. These differences may affect the distribution of performance metrics seen across instances. If a control setting is selected for the entire group, an adverse effect on the overall utility (that is, the overall performance of the system) may be caused. In order to maximize the overall utility across the entire group, the system can cluster instances into subgroups, thereby considering their individual characteristics (modeled in their environmental characteristics) and their feedback characteristics (modeled in the performance metrics received for the control setting). The system selects control settings at the level of these subgroups.

[0133] Depending on the specific implementation and standards, the system may start the clustering phase during the initialization phase or during the utilization phase. Figure 2 Indicating clustering is step 208, while the initialization phase and the utilization phase are steps 204 and 206, respectively, but the clustering phase overlaps with the initialization phase, the utilization phase, or both.

[0134] During the clustering phase, the system clusters program instances into clusters based on the current values ​​of the clustering internal parameters and based on the characteristics of the program instances before assigning control settings to the program instances. As described above, the clustering internal parameters for any given controllable element define the hyperparameters of the clustering technique that will be used to cluster that controllable element.

[0135] Once the system has started the clustering phase at any given time, the system maintains a separate causal model for each cluster. That is, the system identifies separate causal relationships within each cluster. As described above, the system may also maintain multiple separate sets of internal parameters for at least some of the internal parameters of each cluster.

[0136] The following description will generally describe maintaining multiple separate sets of ratio parameters and data inclusion window parameters per cluster and per controllable element. However, it should be understood that when the system maintains only a single set of parameters of a certain type per cluster, the calculations need only be performed once for each cluster, and the results of the single calculation can be used for each controllable element in the cluster. Similarly, when the system maintains only a single set of parameters of a certain type for all clusters, the calculations need only be performed once, and the results of the single calculation can be used for all controllable elements in all clusters.

[0137] During the utilization phase, once clustering has been initiated, within a given cluster the system selects control settings for some of the program instances in the cluster based on the current causal model while continuing to select control settings for other program instances based on baseline values ​​of internal parameters.

[0138] The system may employ any of a variety of criteria to determine when to begin clustering, ie, to determine when cluster internal parameters may begin to change from baseline values ​​indicating that the total number of clusters must be set to one.

[0139] For example, one criterion may include that sufficient environmental responses have been collected, e.g., once the amount of environmental responses collected exceeds a threshold. As a specific example, the system may determine that the threshold is met when the total number of environmental responses exceeds the threshold. As another specific example, the system may determine that the threshold is met when the minimum number of environmental responses associated with any one possible setting of any controllable element exceeds the threshold.

[0140] As another example, another criterion may specify that once the system has determined that, for any of the controllable elements, different environmental characteristics differently affect the causal effects of different control settings for the controllable element, the system may begin clustering. As a specific example, this criterion may specify that the system may begin clustering when the distribution of d scores for any controllable element is statistically different between any two program instances, i.e., the distribution of d scores in a causal model based only on the environmental response of one program instance is statistically different (i.e., at a threshold level of statistical significance) from the distribution of d scores in a causal model based only on the environmental response of another program instance.

[0141] References below Figure 3 Selecting control settings, updating the causal model, and updating internal parameters during the clustering phase are described in more detail.

[0142] Figure 3 1 is a flow chart of an exemplary process 300 for performing iterations of environmental control. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., control system 100 of FIG. 1 ) can perform process 300.

[0143] The system may repeatedly perform process 300 to update a causal model that measures the cause-effect relationship between the control settings and the environmental responses.

[0144] The system determines a set of current program instances based on current internal parameters (step 302). Figure 4A Described in more detail, the system determines the spatial scope and the temporal scope based on current internal parameters, such as based on the likelihood that different spatial scopes and temporal scopes lead to orthogonal instances, and then generates the current program instance based on the spatial scope and the temporal scope.

[0145] As described above, each program instance is a collection of one or more entities within an environment and is associated with a time window. As described in more detail below, the time window associated with a given program instance defines which environmental responses the system attributes to or associates with the program instance.

[0146] In some cases, for each controllable element, the system also determines how long the selected setting for the controllable element will be applied as a proportion of the time window associated with the controllable element (e.g., the entire time window, the first quarter of the time window, or the first half of the time window). Generally speaking, the duration for which the setting is applied can be fixed to a value independent of the time window, can be a fixed proportion of the time window, or the proportion of the time window can be an internal parameter that is changed by the system.

[0147] References below Figure 4A Determining a set of current program instances is described in more detail.

[0148] In the case where the environment includes only a single physical entity, the set of current instances may contain only one instance. Alternatively, the system may identify multiple current instances, each of which includes a single physical entity, but is separated in time, ie, separated by at least the time range of the entity.

[0149] The system assigns control settings for each current instance (step 304). The manner in which the system assigns control settings for any given instance depends on which control phase the system is currently executing.

[0150] As described above, at the beginning of the control environment, i.e., before sufficient information is available to determine causal relationships with any confidence, the system operates in an initialization phase. In the initialization phase, the system selects control settings for an instance without considering the current causal model, i.e., the system explores the space of possible control settings. That is, the system selects control settings for an instance based on a baseline probability distribution over the possible control settings for each controllable element.

[0151] In some implementations, during an initialization phase, the system varies internal parameters that determine the spatial extent, temporal extent, or both of a program instance in order to identify the likelihood that each possible value of the spatial extent and temporal extent results in instances that are orthogonal to one another.

[0152] As described above, in some implementations, the set of control phases includes only an initialization phase, and the system continues to operate throughout this initiation phase, i.e., continues to explore the space of possible control settings while compiling environmental responses in order to update the causal model.

[0153] In some other implementations, once certain criteria are met, the system transitions to an exploitation phase, in which the system selects control settings for some of the current instances based on the current causal model, i.e., to exploit the causal relationships currently reflected in the causal model, while continuing to select control settings for other of the current instances based on the baseline values ​​of the internal parameters.

[0154] Additionally, in some implementations, the system begins performing clustering during an initialization phase or a utilization phase.

[0155] When performing clustering, the system clusters program instances into clusters. Within each cluster, the system proceeds independently as described above.

[0156] That is, during the initialization phase, the system independently selects settings within each cluster using the baseline distribution, and during the utilization phase, the system independently assigns control settings for some of the current instances based on the current causal model within each cluster, while continuing to independently select control settings for other instances in the current instance based on the baseline values ​​of the internal parameters within each cluster.

[0157] By performing clustering, the system is able to conditionally assign control settings based on (i) factor interactions between the settings' effects on environmental responses and environmental characteristics of the instance (e.g., properties of the instance that cannot be manipulated by the control system), (ii) factor interactions of different independent variables, or (iii) both.

[0158] References below Figure 5 The selection of control settings in the exploitation phase with and without clustering is described in more detail.

[0159] The system obtains an environment response for each of the program instances (step 306).

[0160] In particular, the system monitors environmental responses and determines which environmental responses are attributed to which current instance based on a time window associated with each program instance.

[0161] More specifically, for each program instance, the system associates with the program instance each environmental response that (i) corresponds to an entity in the program instance and (ii) is received during some portion of a time window associated with the program instance. As a specific example, to limit residual effects from previous control setting assignments, the system may associate with the program instance each environmental response that corresponds to an entity in the instance and is received for a time period exceeding a threshold duration after the start of the time window (e.g., during the second half of the time window, the last third of the time window, or the last quarter of the time window). In some implementations, the threshold duration is fixed. In other implementations, the system maintains a set of internal parameters that define the threshold duration and changes the duration during operation of the system.

[0162] The system updates the causal model based on the obtained environmental response (step 308). Figure 6 Describe the updated causal model in more detail.

[0163] The system updates at least some of the internal parameters based on the current performance of the system (ie, as reflected in the updated causal model), relative to a baseline performance of the system, or both (step 310).

[0164] In particular, the system can update any of the multiple sets of internal parameters based on a heuristic-based approach, through random changes, or both. The heuristic-based approach can include heuristics derived from one or more of: an updated causal model, the current performance of the system relative to a baseline performance of the system, or criteria determined using a priori statistical analysis.

[0165] In other words, for each set of internal parameters that the system is capable of changing, the system may use one or more of the above techniques to update the set of internal parameters to allow the system to more accurately measure cause-effect relationships.

[0166] In some cases, even if the system is able to change internal parameters, the system constrains certain sets of internal parameters to be fixed. For example, the system may fix data inclusion window parameters and clustering parameters during an initialization phase. As another example, the system may fix clustering parameters until certain criteria are met, and then begin changing all internal parameters under the control of the system during an utilization phase after the criteria have been met.

[0167] References below Figures 8 to 12 Describes updating a set of internal parameters in more detail.

[0168] In general, the system can perform steps 302 to 306 at a different frequency than step 308, and perform step 310 at a different frequency than both steps 302 to 306 and step 310. For example, the system can perform multiple iterations of steps 302 to 306 for each iteration of step 308 performed, i.e., to collect environmental responses for multiple sets of different instances before updating the causal model. Similarly, the system can perform multiple different instances of step 308 before performing step 310, i.e., can perform multiple different causal model updates before updating the internal parameters.

[0169] Figure 4A 4 is a flow chart of an exemplary process 400 for determining a program instance. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., the control system 100 of FIG. 1 ) can perform the process 400.

[0170] The system selects a spatial range for each of the entities in the environment (step 402). The spatial range of a given entity defines a segment of the environment that affects the environmental response obtained from the given entity when controlled by a given set of control settings. The spatial range of a given entity is defined by a set of spatial range parameters (e.g., a set of spatial range parameters specific to a given entity, or a set of spatial range parameters shared between all entities). In some embodiments, the spatial range internal parameters are fixed, that is, maintained constant at the same value, or randomly sampled from a fixed range throughout the environmental control process. For example, if the environment includes only a single entity, each program instance will include the same single entity. For another example, if the environment includes multiple entities, but there is no uncertainty about which entities are affected by the control settings, the spatial range parameters can be fixed to values ​​that ensure that the generated instances will be orthogonal.

[0171] When the spatial range is not fixed and a single value is maintained for the spatial range parameter (i.e., the spatial range parameter is updated only based on heuristics), the system selects the current value of the spatial range parameter for each entity as the spatial range of the entity. When the spatial range is not fixed and a range of values ​​is defined by the spatial range parameter, the system samples the value of the spatial range from the range currently defined by the spatial range parameter based on the current causal model for the spatial range parameter of the entity.

[0172] By selecting spatial ranges for entities, the system limits how many entities are in each program instance and which entities are included in each program instance. In particular, the system generates program instances so that no program instance covers an environment segment within the spatial range of an entity in another program instance, even partially.

[0173] Figure 4BAn example of a map 420 of an environment including a plurality of physical entities each associated with a spatial range is shown. In particular, Figure 4B An environment including multiple physical entities (represented as points in the figure) within a portion of the United States is shown. The spatial range selected by the system for each entity is represented by the shaded circle. For example, the system can maintain a range of possible radii for each entity and can select the radius of the shaded circle for each entity from the range. Figure 4B As can be seen from the example of , different entities can have different spatial extents. For example, entity 412 has a shaded circle of a different size than entity 414.

[0174] As from Figure 4B As can also be seen from the example of, the system can also optionally apply additional criteria to reduce the possibility that the program instances are not orthogonal. In particular, the system has also selected a buffer zone (represented as a dashed circle) extending beyond the spatial extent of the entity for each entity, and has required that no entity in a different instance can have a spatial extent within the buffer zone.

[0175] Due to spatial extent and due to buffers, some entities within the environment Figure 4B 414. In the iteration shown, entities 414 are not selected as part of the program instance. These unselected entities (e.g., entity 416) are represented as points without shaded circles or dotted circles. In particular, the system has not yet selected these entities because their spatial ranges intersect with the spatial range or buffer of another entity selected as part of the program instance. For entity 416, this entity is not selected because the spatial range of entity 416 will have intersected with the spatial range or buffer of entity 414. For example, given the spatial range and buffer of the sampled, the system may have selected the spatial range that will maximize the number of program instances that can be included in a set of current instances without violating any of the standards.

[0176] The system selects a time range for each program instance, or if different controllable elements have different time ranges, then selects a time range for each controllable element of each program instance (step 404). As described above, the time range defines a time window associated with each program instance in the program instance or a time window associated with a controllable element within a program instance.

[0177] In some cases, the time horizon may be fixed, i.e., prior to controlling the operation of the system, the user of the system knows which environmental responses observed for a given entity in the environment should be attributed to the program instance that includes that entity. In other cases, the time horizon may be unknown or associated with a level of uncertainty, i.e., the user of the system does not know or specify exactly how long after a set of settings are applied that the effects of the settings will be observed.

[0178] In the case where the time range is not fixed, the system samples the value of the time range from the range currently defined by the time range parameter based on the current causal model for the time range parameter. As described above, different entities (and therefore different program instances) may have multiple different sets of time range parameters, or all entities may share the same set of time range parameters.

[0179] The system generates a program instance based on the selected spatial range and the selected time range (step 406). In other words, the system divides the entities in the environment based on spatial range, that is, so that any entity in a program instance has a spatial range (or buffer, if used) that does not intersect the spatial range of another entity in a different program instance, and associates each program instance with a time window defined by the spatial range of the program instance.

[0180] Figure 5 1 is a flow chart of an exemplary process 500 for selecting control settings for a set of current instances. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed control system (e.g., the control system 100 of FIG. 1 ) can perform the process 500.

[0181] The system determines the current program instance (step 502), for example, as described above with reference to Figure 4A described.

[0182] The system then performs steps 504 to 514 for each of the controllable elements to select settings for the controllable elements for all current program instances.

[0183] Optionally, the system clusters the current program instance based on the environmental characteristics, that is, generates multiple clusters for the controllable elements (step 504). Because clustering is performed for each controllable element, the system can cluster the current program instance in different ways for different controllable elements. Figure 7 Description Clustering of program instances.

[0184] That is, when the system is currently executing the clustering phase, the system first determines the current cluster assignment for the current program instance. After the system determines the current cluster assignment, the system performs iterations of steps 506 to 514 independently for each cluster.

[0185] When the system is not currently executing the clustering phase, the system does not cluster the current program instances, and performs a single iteration of steps 506 to 514 for all current program instances.

[0186] The system determines a current mixed-to-baseline ratio (step 506). In particular, when the set of ratio parameters for the controllable element includes only a single value, the system selects the current value of the ratio parameter as the current mixed-to-baseline ratio. When the system defines a range of possible values ​​for the ratio parameter of the controllable element, the system samples a value of the mixed-to-baseline ratio from the current range of possible values ​​defined by the ratio parameter based on a causal model for the set of ratio parameters.

[0187] The system identifies each instance as a hybrid instance of a controllable element or a baseline instance of a controllable element based on the current hybrid to baseline ratio (step 508). For example, the system can assign each instance as a hybrid instance with a probability based on the ratio, or can randomly divide the total number of instances to be as close to the ratio as possible. Alternatively, when the system randomly changes at least one of the internal parameters based on the difference between the hybrid instance performance and the baseline instance performance, the system can apply an assignment scheme that assigns instances based on the current ratio and takes into account the blocking scheme used when calculating the causal model of the difference between the measured performances, i.e., as described above.

[0188] The system selects control settings for the controllable elements for the baseline instance based on the baseline values ​​of the internal parameters and according to the allocation scheme (step 512). In other words, the system selects control settings for the baseline instance based on the baseline probability distribution over possible values ​​of the controllable elements determined at the beginning of the initialization phase.

[0189] The system selects control settings for the hybrid instance based on the current causal model and according to the allocation scheme (step 514).

[0190] In particular, the system maps the current causal model to a probability distribution over possible settings of the controllable elements. For example, the system can apply probability matching to map the impact measurements and confidence intervals for the controllable elements in the causal model to probabilities.

[0191] In general, the system assigns control settings based on these probabilities, and such that the system will later identify a sufficient number of chunk groups when calculating the d-score. As a specific example, the system may then partition the mixed instances into chunk groups (based on the same chunking scheme that will later be used to calculate the d-score), and then select control settings within each chunk group according to a probability distribution over the possible settings, i.e., such that each instance in the chunk group is assigned any given possible setting with a probability specified in the probability distribution.

[0192] Figure 6600 is a flow chart of an exemplary process 600 for updating a causal model for a given controllable element and a given type of environmental response. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed control system (e.g., the control system 100 of FIG. 1 ) can perform the process 600.

[0193] The system can perform process 600 for each controllable element and for each type of environmental response for which the system maintains a causal model. For example, when the system maintains a causal model that models causal effects for only a single performance metric, the system performs process 600 only for that performance metric. Alternatively, when the system maintains a causal model that models causal effects for multiple different types of environmental responses, the system performs process 600 for each type of environmental response (e.g., each different type of sensor reading or measurement result).

[0194] When the system is currently clustering program instances into clusters, the system can independently perform process 600 for each cluster. That is, the system can independently maintain and update the causal model for each cluster.

[0195] The system determines the current data inclusion window of the controllable element (step 602), i.e., based on the current data inclusion window parameters of the controllable element. In particular, when the set of data inclusion window parameters of the controllable element includes only a single value, the system selects the current value of the data inclusion window parameter as the current data inclusion window. When the set of data inclusion window parameters of the controllable element defines a range of possible values, the system samples the values ​​of the data inclusion window from the range of values ​​currently defined by the set of data inclusion window parameters. When the system does not change the data inclusion window parameters, the system sets the value to a fixed initial data inclusion window or samples the value from a fixed range of possible values.

[0196] The system obtains, for each possible value of the controllable element, environmental responses of a given type that were recorded for the instances for which the possible value of the controllable element was selected (step 604). In particular, the system only obtains environmental responses for instances that occurred during the current data inclusion window.

[0197] The system updates the impact measurements in the causal model based on the environmental responses to the possible settings of the controllable elements (step 606).

[0198] That is, the system determines a set of block groups based on a blocking scheme (eg, one of the blocking schemes described above).

[0199] For each block group, the system then determines a corresponding d-score for each possible setting selected in any of the instances in the block group. In general, the system calculates an impact measure, i.e., a d-score, for a given controllable element based on a block scheme, i.e., calculates a d-score between the environmental responses for instances assigned to the same block group.

[0200] As a specific example, in a blocking scheme that allocates blocking groups to include at least one instance of each possible setting, the impact measurement di for a possible setting i of the controllable element may satisfy:

[0201]

[0202] where x_i is the ambient response of a given type for the instance within the block group where setting i has been selected, the sum is the sum over all possible settings except i, and N is the total number of possible settings.

[0203] As another specific example, in a blocking scheme that assigns pairs of instances to blocking groups, the impact measure di for a possible setting i of the controllable element may satisfy:

[0204] d_i=x_i-x_(i+1).

[0205] where x_i is the given type of environmental response for the instance of the block group in which setting i has been selected, and x_(i+1) is the given type of environmental response for the instance within the block group in which setting i+1 has been selected, where setting i+1 is the next higher possible setting of the controllable element. For the highest setting of the controllable element, setting i+1 may be the lowest setting of the controllable element.

[0206] As yet another specific example, in a blocking scheme that assigns pairs of instances to blocking groups, the impact measurements di for a possible setting i of the controllable element may satisfy:

[0207] d_i=x_i-x_1,

[0208] Where x_1 is an environmental response of a given type for an instance in which a predetermined one of the possible settings of the controllable element has been selected.

[0209] The system then computes the updated overall impact measure for a given setting i as the average of the d scores computed for setting i.

[0210] In some cases, the d-score calculation may be a proportional calculation rather than an additive calculation, ie, the subtraction operation in any of the above definitions may be replaced by a division operation.

[0211] For each possible value of the controllable element, the system determines a confidence interval for the updated impact measure (step 608). For example, the system can perform a t-test or other statistical hypothesis test to construct a p% confidence interval around the updated impact measure (i.e., around the mean of the d-scores), where p is a fixed value, such as 95% or 97.5% or 99%.

[0212] In some implementations, for example, when external data specifies that different controllable elements have different costs or risk levels associated with deviating from a baseline probability distribution for the different controllable elements, the system applies different p-values ​​for the different controllable elements.

[0213] In some implementations, where certain settings of a controllable element are associated with different implementation costs or higher risks, the system applies a correction (e.g., a Bonferroni correction) to the confidence intervals. In particular, in a Bonferroni correction, a correction is applied such that if N confidence intervals are calculated for N possible settings of a controllable element and the overall desired confidence level for the element is 95% (i.e., α=0.05), then the α value used for each individual test to calculate the confidence interval is α / N. If certain settings are associated with higher risks or implementation costs, then a "corrected" α value associated with a higher confidence level can be specified for those settings. This forces the system to accumulate more data before utilizing those settings.

[0214] Figure 7 1 is a flow chart of an exemplary process 700 for clustering a set of program instances for a given controllable element. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., control system 100 of FIG. 1 ) can perform process 700.

[0215] The system selects the current hyperparameters of the clustering technique being used by the system from the clustering parameters of the controllable elements (step 702). In particular, each hyperparameter that can be changed by the system is defined by a distinct set of internal parameters. That is, the clustering parameters include a separate set of internal parameters for each hyperparameter that is under the control of the system during operation.

[0216] The system may perform clustering using any of a variety of clustering techniques. However, the hyperparameters varied by the system will typically include hyperparameters of the size of the clusters generated by the clustering technique, and in some cases, the environmental characteristics of the instances that the clustering technique considers when generating the clusters.

[0217] For example, the system can use statistical analysis, such as factor analysis of variance (ANOVA), to generate cluster assignments. In particular, factor ANOVA is used to find factors that explain the largest amount of variance between clusters, i.e., environmental characteristics. That is, when calculating the D score for each possible control setting, factor ANOVA can monitor the interaction terms between these treatment effects and external factors. As data accumulates and interactions begin to emerge, factor ANOVA creates different clusters of instances across space and time, where each cluster represents a distinct external factor state or attribute.

[0218] For another example, the system can use machine learning techniques to generate clustering assignments. As a specific example of machine learning techniques, the system can use decision trees. Decision trees are classic machine learning algorithms for classification and regression problems. Decision trees use recursive segmentation schemes by successively identifying the best variables (i.e., the best environmental characteristics) to analyze using information theory functions such as the Gini coefficient. As another specific example of machine learning techniques, the system can use conditional inference trees. Similar to decision trees, conditional inference trees are recursive binary segmentation schemes. The algorithm continues by selecting a series of variables to be analyzed based on a significance test program to segment based on the strongest environmental characteristic factors. As another specific example, the system can use a machine learning model (e.g., a deep neural network) to process data representing each of the environmental characteristics associated with program instances to generate embeddings, and then cluster the program instances into specified clusters based on the similarity between the embeddings, for example, using k-means clustering or another clustering technique. As a specific example, embedding can be the output of an intermediate layer of a neural network that has been trained to receive data representing program instances and predict the value of a performance metric of a program instance.

[0219] In some cases, as the system operation progresses, i.e., as more data becomes available, the system can switch clustering techniques. For example, once more than a threshold amount of program instances are available, the system can switch from using statistical techniques or decision trees to using deep neural networks.

[0220] The system clusters the instances in the current data inclusion window using clustering techniques according to the selected hyperparameters (step 704).

[0221] The system calculates a causal model for each cluster (step 706), i.e., as described above with reference to Figure 6 described above, but only uses instances that have been assigned to the cluster.

[0222] The system then independently assigns control settings for the controllable elements within each of the clusters based on the causal model computed for the cluster (step 708), i.e., as described above with reference to Figure 5Specifically, the system clusters each current instance using a clustering technique and then assigns a control setting to a given current instance based on the cluster to which the current instance is assigned and using a causal model computed for the cluster if the given current instance is not designated as a baseline instance.

[0223] The system may then determine whether the clustering parameters need to be adjusted (step 710), i.e., determine whether the current values ​​of the clustering parameters are not optimal, and if not, update the clustering parameters of the controllable elements. In particular, during operation, the system updates the clustering parameters to balance two competing objectives: (1) pooling instances into clusters so that the effects of the controllable elements on the performance metric have the greatest intra-cluster similarity and the effects of the controllable elements on the performance metric have the greatest inter-cluster difference, and (2) maximizing the size of the clusters so as to have the largest possible intra-cluster sample size to improve the accuracy of the causal model. The system may achieve this by adjusting these values ​​using heuristics, using random sampling, or using both heuristics and random sampling.

[0224] The system may determine whether to change the number of clusters, ie, change the value of the clustering parameter of the controllable element, in any of a variety of ways (ie, based on any of a variety of heuristics).

[0225] More generally, as described above, for any given set of internal parameters changed by the system, the system is able to adjust the set of internal parameters in one of three ways: (i) using a heuristic-based approach to adjust a single value, (ii) using random variation to adjust the probabilities of different values ​​assigned to a range of values, or (iii) using a heuristic-based approach to adjust the range of values ​​while using random variation to adjust the probabilities within the current range.

[0226] Heuristic-based methods may include heuristics based on characteristics of the current causal model, heuristics based on a priori statistical analysis, or both.

[0227] In the random variation method, the system maintains a causal model that measures the causal effects between different values ​​within the current range and the quality factor of the set of internal parameters. The system then maps the causal model to probabilities of different values ​​and, when necessary, selects values ​​for the internal parameters based on the probabilities. As will be described in more detail below, the quality factor of any given set of internal parameters is typically different from the performance metric measured in the causal model that models the causal relationship between the control setting and the performance metric.

[0228] Figure 8800 is a flow chart of an exemplary process 800 for updating a set of internal parameters using random changes. For convenience, process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed control system (e.g., control system 100 of FIG. 1 ) can perform process 800.

[0229] Process 800 may be performed for any set of internal parameters that are being updated using random variation. Examples of such internal parameters may include any or all of multiple sets of data inclusion window parameters, multiple sets of clustering parameters, multiple sets of ratio parameters, multiple sets of spatial extent parameters, multiple sets of temporal extent parameters, etc.

[0230] As described above, during the clustering phase and for any set of internal parameters other than clustering parameters, the system may perform process 800 independently for each cluster or for each controllable element and for each cluster.

[0231] Additionally, the system may also independently perform process 800 for each controllable element, where random variation is used to change the clustering parameters.

[0232] The system maintains a causal model for the set of internal parameters, the causal model measuring causal relationships between different possible values ​​of the internal parameters and a figure of merit for the set of internal parameters (step 802).

[0233] For example, the quality factor of the set of internal parameters can be the difference between the performance of the hybrid instance and the performance of the baseline instance. In this example, the quality factor measures the relative performance of the hybrid instance relative to the baseline instance, and the system calculates an impact measure of the quality factor, i.e., the d-score, for different values ​​within the range defined by the internal parameters.

[0234] Therefore, when calculating the causal model for this set of internal parameters, the system is as described above with reference to Figure 6 The above continues, except that: (i) the possible settings are possible values ​​of the intrinsic parameter, and (ii) each xi in the score computation is the difference between: (1) the performance metric of the hybrid instance for which the control setting is assigned, wherein the control setting has the possible value of the selected intrinsic parameter, and (2) the performance metric of the corresponding baseline instance.

[0235] As another example, the quality factor of the set of internal parameters may be a measure of the accuracy of the causal model for the controllable element, eg, a measure of the width of the confidence intervals for different settings of the controllable element.

[0236] The maintained causal model may be determined based on a data inclusion window of the set of internal parameters. When the set of internal parameters is actually a data inclusion window parameter, the data inclusion window is different for different possible values ​​in the current range. When the set of internal parameters is a set of different internal parameters, the data inclusion window may be a separate set of internal parameters that is fixed or varies based on a heuristic as described below or also based on random variation as described in this figure.

[0237] The system maps the causal model to a probability distribution over possible values ​​in the value range, for example using probability matching (step 804). That is, the system maps the impact measurements and confidence intervals to the probability of each possible value in the value range using probability matching or another suitable technique.

[0238] When it is necessary to sample a value from the range, the system samples the value from the range of possible values ​​according to the probability distribution (step 806). That is, when the system needs a value from the range defined by the internal parameters to operate (e.g., assigning a time range to a program instance, assigning a data inclusion window to a given controllable element, determining hyperparameters of a clustering technique, or assigning a current hybrid to baseline ratio for a set of current instances), the system samples from the range of possible values ​​according to the probability distribution. By sampling values ​​in this manner, the system ensures that the values ​​that are most likely to optimize the quality factor of the set of internal parameters (e.g., maximize the difference between the hybrid instance and the baseline instance) are sampled more frequently, while still ensuring that the space of possible values ​​is explored.

[0239] The system computes updates to the causal model (step 808). That is, when a new environmental response is received for a new program instance, the system recomputes the causal model by computing an overall impact measure (i.e., the average of the d scores) and a confidence interval around the overall impact measure. The system can perform the same calculations as described above with reference to Figure 6 The causal model update is performed in the same manner as described, i.e., by selecting block groups, computing d scores within those block groups (a quality factor based on the set of parameters described above), and then generating a causal model from those d scores.

[0240] By repeatedly performing process 800 , the system can repeatedly adjust the probabilities assigned to values ​​within the range to tend to produce values ​​that are more favorable to the figure of merit.

[0241] For example, when the set of internal parameters is data inclusion window parameters, maintaining a causal model that models the impact of different data inclusion window values ​​on hybrid and baseline performance allows the system to select a data inclusion window that produces a more accurate and robust causal model calculated for the controllable element.

[0242] As another example, when the set of internal parameters are spatial or temporal range parameters, maintaining a causal model that models the effects of different spatial or temporal range values ​​on hybrid and baseline performance allows the system to select spatial or temporal ranges that produce orthogonal program instances that maximize hybrid instance performance relative to baseline instance performance.

[0243] As another example, when the set of internal parameters defines clustering hyperparameters, maintaining a causal model that models the effects of different hyperparameter values ​​on hybrid and baseline performance allows the system to select a cluster assignment that maximizes the performance of the system, i.e., more efficiently identifying clusters that satisfy the above reference Figure 7 The cluster assignment of the target.

[0244] In some implementations, the system determines whether to adjust the current range of possible values ​​for the internal parameter (step 810). As described above, the range of possible values ​​for any given internal parameter can be fixed or can be adjusted using heuristics to ensure that the space of possible values ​​being explored remains reasonable throughout the operation of the system.

[0245] One example of a heuristic that can be used to adjust the current range of possible values ​​is a heuristic that depends on the shape of the current causal model. In particular, as the magnitude of the impact measurement in the causal model increases as it approaches the current upper limit of the range, the system can increase the upper limit of the range (or increase both the upper and lower limits of the range), and as the magnitude of the impact measurement increases as it approaches the current lower limit of the range, the system can decrease the lower limit (or decrease both the upper and lower limits).

[0246] Another example of a current heuristic that can be used to adjust possible values ​​is a heuristic that relies on statistical power analysis.

[0247] For example, when the set of internal parameters is a set of clustering parameters that define the cluster size used by the clustering technique, the system can calculate a statistical power curve that represents the effect of changes in sample size (i.e., cluster size) on the width of the confidence interval reflected by the current causal model for the controllable element. Given the nature of the statistical power curve, the confidence interval quickly becomes more accurate at the small end of the sample size, but as the sample size increases, each additional increase in the sample size results in a disproportionately smaller increase in the accuracy of the confidence interval (i.e., a disproportionately smaller decrease in the width of the confidence interval). Therefore, exploring larger cluster sizes can result in very small gains in statistical power, and with it a high risk of inaccurately representing the current decision space. To account for this, the system can then constrain the range of possible cluster sizes to fall between a lower threshold and an upper threshold on the statistical power curve. By constraining the cluster size in this way, the system will not explore clusters that are so small that the statistical power is too small to calculate a significant confidence interval. The system will also not experiment with unnecessarily large cluster sizes (i.e., cluster sizes that result in a small gain in statistical power in exchange for the risk of failing to capture all possible variations between instances).

[0248] For another example, when the set of internal parameters is a set of ratio parameters, the system can perform a statistical power analysis to calculate the minimum number of baseline instances required to determine that the hybrid instance is better than the baseline instance with a threshold statistical power given the current causal model for the ratio parameters. The system can then adjust the lower limit of the range of possible ratio values ​​so that the ratio does not cause the number of baseline instances to fall below the minimum value.

[0249] As another example of adjusting ranges based on heuristics, when updating the ranges of time range parameters for entities in the environment based on heuristics, the system can maintain a causal model for each entity that measures the causal relationship between: (i) the control setting selected at a given control iteration, and (ii) the environmental response obtained from the entity at a subsequent control iteration (i.e., the control iteration immediately following the given control iteration). Because the system attempts to select time ranges for entities that ensure orthogonality of program instances, if the time ranges have been appropriately selected, the causal model should indicate that the causal effect between the current control setting and the environmental response to the subsequent control setting is likely to be zero. Therefore, if the causal model indicates that the overlap of the confidence interval of the impact measurement result for any of the control settings with zero exceeds a threshold, the system can determine that the lower limit of the range of possible time ranges is to be increased.

[0250] As another example of adjusting the range based on a heuristic, when updating the range of the spatial range parameters of entities in the environment based on a heuristic, the system may maintain a causal model for each given entity that measures the causal relationship between: (i) the control settings selected for the program instance that includes the given entity at a given control iteration, and (ii) the environmental response obtained from entities adjacent to the given entity at the current control iteration. The adjacent entity can be the entity that is closest to the given entity among the entities included in the current set of instances of the current control iteration. Since the system attempts to select a spatial range for the entity that ensures orthogonality of the program instances, if the spatial range has been appropriately selected, the causal model should indicate that the causal effect between the current control setting for the given entity and the environmental response to the adjacent entity is likely to be zero. Therefore, if the causal model indicates that the overlap of the confidence interval of the impact measurement result for any of the control settings with zero exceeds a threshold, the system can determine that the lower limit of the range of possible spatial ranges is to be increased.

[0251] References below Fig.12 Additional examples of heuristics that may be used to adjust the range of possible values ​​for the data inclusion window and ratio parameters are described in more detail.

[0252] Fig. 9 1 is a flow chart of an exemplary process 900 for updating the value of a given controllable element's data inclusion value based on a heuristic. For convenience, process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed control system (e.g., control system 100 of FIG. 1 ) may perform process 900.

[0253] Generally speaking, the system performs process 900 for a data inclusion window when the data inclusion window is a parameter that varies based on a heuristic without using random variation.

[0254] When the system maintains multiple clusters for a given controllable element, the system may perform process 900 independently for each cluster, i.e., such that the data inclusion window for a given controllable element within one cluster can be updated differently than the set of internal parameters for a given controllable element within another cluster.

[0255] The system accesses the current causal model for a given controllable element (step 902).

[0256] The system analyzes one or more characteristics of the current causal model (step 904). For example, the system may perform a normality test to determine whether the d scores for various possible control settings of a given controllable element are normally distributed (step 904). In particular, the system may perform a normality test, such as a Shapiro-Wilk test, on the distribution of d scores for a given controllable element in the current causal model. Generally speaking, the system scales and pools the distribution of d scores between different possible settings to generate a single distribution, and then performs a normality test on the single distribution. The system may perform this test for different data inclusion windows (e.g., for the current causal model calculated using the current data inclusion window and one or more alternative causal models calculated using one or more alternative data inclusion windows) to find the longest data inclusion window that satisfies the normality test at a specified p-value.

[0257] As another specific example, the system can measure the overlap of confidence intervals between different impact measurements in a given controllable element in the current causal model. The system can perform this test for different data inclusion windows (e.g., for the current causal model calculated using the current data inclusion window and one or more alternative causal models calculated using one or more alternative data inclusion windows) to find the data inclusion window that is closest to the desired degree of overlap.

[0258] As another specific example, the system can calculate a statistical power analysis to identify a sample size that will result in a current causal model having a desired statistical power. The system can then adjust the data inclusion window so that the number of instances included in the adjusted window is equal to the identified sample size.

[0259] The system determines whether to adjust the data inclusion window parameters based on the results of the analysis (step 906). For example, the system can adjust the data inclusion window parameters to specify the longest data inclusion window that satisfies the normality test as described above, or the data inclusion window that is closest to the expected degree of overlap, or the data inclusion window that includes the number of instances equal to the identified sample size.

[0260] Fig. 9 The example of is an example of adjusting the data inclusion window based on heuristics. However, in general, any of the internal parameters can be adjusted based on heuristics (rather than being kept fixed or adjusted using random changes). The following are several examples of setting internal parameters based on heuristics.

[0261] For example, the system can use a statistical power analysis to set the value of the ratio parameter. In particular, the system can perform a statistical power analysis to calculate the minimum number of baseline instances required to determine that the hybrid instance is better than the baseline instance at a threshold statistical power. The system can then adjust the value of the ratio parameter to be equal to the minimum value.

[0262] As another example, to set the value of the cluster size hyperparameter, the system may perform an a priori statistical power analysis to determine a sufficient number of environmental responses required for the causal model to have the desired statistical power (i.e., rather than a range as described above), and set the value of the cluster size to that range.

[0263] The above description describes how the system may modify internal parameters during operation of the system. Such adjustment of internal parameters may allow the system to effectively account for changes in environmental characteristics, i.e., environments where the mapping from control settings to environmental responses is not static and may change at different times during operation of the system. Unless properly accounted for, changes in environmental characteristics that do not have the same effect on all possible control settings for all controllable elements may result in inaccurate causal models based on stale data that is no longer relevant, and may therefore reduce the effectiveness of the system in controlling the environment.

[0264] Fig.10 1 is a flow chart of an exemplary process 1000 for responding to changes in one or more characteristics of an environment. For convenience, process 1000 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed control system (e.g., control system 100 of FIG. 1 ) can perform process 1000.

[0265] The system monitors the environmental response to the control settings selected by the system (step 1002). That is, as described above, the system repeatedly selects control settings and monitors the response to those selected control settings.

[0266] The system determines an indication that one or more characteristics of the environment have changed (step 1004). In particular, the change in the characteristic of the environment is a change in the relative effects that different settings of modifying at least one of the controllable elements have on the environmental response monitored by the system. That is, by determining an indication that one or more characteristics have changed, the system determines that the relative causal effects of the different settings on the environmental response are likely to have changed, i.e., different from a global change that affects all possible control settings in different ways. Although the system does not have access to direct information specifying that a change has occurred, the system can determine an indication that a change is likely to have occurred based on the monitored environmental response.

[0267] For example, when the difference between the current system performance and the baseline system performance decreases, the system may determine that an indication that a change has occurred. In particular, as described in more detail below, the system may determine this based on a performance metric that increases for smaller possible values ​​of the data inclusion window (i.e., as reflected by a causal model for the above data inclusion window).

[0268] As another example, as described above, when a normality test determines that the d-scores for the possible settings of a controllable element are no longer normally distributed, the system can determine an indication that a change has occurred.

[0269] In response to determining an indication that one or more characteristics of the environment have changed, the system adjusts internal parameters of the system (step 1006).

[0270] Generally speaking, the system adjusts the value of the internal parameter to indicate an increase in the level of uncertainty regarding whether the causal model maintained by the system accurately captures the causal relationship between the control setting and the environmental response.

[0271] For example, the system may adjust the data inclusion window parameters to narrow the data inclusion window, i.e., so that only more recent historical environmental responses will be included in determining the causal model. That is, the system may adjust the data inclusion window parameters so that the range of possible data inclusion windows tends to be shorter data inclusion windows.

[0272] As another example, the system can adjust the ratio parameter to reduce the mixing and exploration ratio, that is, so that there are fewer mixing instances relative to the exploration instances. By reducing the ratio, the system relies less on the current causal model when selecting control settings, and explores the space of possible control settings more frequently. That is, the system can adjust the ratio parameter so that the range of possible ratios tends to be smaller ratios.

[0273] As another example, the system may adjust clustering parameters to reduce the number of clusters that instances are clustered into. By reducing the number of clusters, the system prevents the causal model from clustering based on features that may no longer be relevant in explaining differences in system performance between clusters.

[0274] Fig.11 A representation 1100 of a data inclusion window for a given controllable element of an environment is shown when a set of internal parameters defining the data inclusion are randomly varied. Fig.11 As can be seen in the example of , while the data inclusion window can range from zero (i.e., excluding data) to infinity (i.e., including all program instances), the current random variation range 110 from which the data inclusion window for a given controllable element is sampled is between a lower limit A 1102 and an upper limit B 1104. In some cases, the lower limit A 1102 and the upper limit B 1104 are fixed, and the system adjusts the probabilities assigned to different values ​​between the lower limit A 1102 and the upper limit B 1104 by updating the causal model as described above. In other cases, the system can change the lower limit A 1102 and the upper limit B 1104 while also updating the causal model. In particular, the system can adjust the range 1110 based on the likelihood that the relative causal effects of different possible values ​​of the controllable element are changing.

[0275] In particular, if Fig.11As shown, the system maintains a range of possible values ​​for the data inclusion window. That is, the data inclusion window parameters include the lower limit of the range, the upper limit of the range, and the possible values ​​that the data inclusion window can take within the range. The data inclusion window parameters also include the probabilities of the possible values ​​used when randomly sampling the values. As described above with reference to Figure 8 As described, these probabilities are adjusted by the system.

[0276] In some cases, the range of possible values ​​is fixed. However, in other cases, the system changes the lower and upper limits of the range based on one or more heuristics to adjust the possible data inclusion windows explored by the system and prevent the system from exploring data inclusion windows that are too short or too long.

[0277] For example, the system may calculate a statistical power curve that represents the effect that changes in sample size (via changes in the data inclusion window) will have on the width of the confidence interval for the controllable element of the current causal model. Given the nature of the statistical power curve, the confidence interval quickly becomes more accurate at the small end of the sample size, but as the sample size increases, each additional increase in the sample size results in a disproportionately smaller increase in the precision of the confidence interval (i.e., a disproportionately smaller decrease in the width of the confidence interval). Therefore, exploring longer data inclusion windows may result in very small gains in statistical power, and with it a high risk of inaccurately representing the current decision space. To account for this, the system may then constrain the range of the data inclusion window to produce the number of samples that fall between a lower threshold and an upper threshold on the statistical power curve. By constraining the data inclusion window in this way, the system will not explore data inclusion windows that are so short that the statistical power is too small to calculate a significant confidence interval, that is, it will not explore data inclusion windows that result in insufficient data to calculate a statistically significant confidence interval. The system also does not explore unnecessarily long data inclusion windows, i.e., data inclusion windows that result in small gains in statistical power at the risk of failing to account for recent changes in environmental characteristics.

[0278] As another example, the system may calculate a stability measure of the interaction between time and the relative impact measurements of possible control settings of the controllable elements, such as factor analysis. That is, the system may determine the stability of the cause-effect relationship over time. When the stability measure indicates that the cause-effect relationship is stable, the system may increase the upper limit or both the upper limit and the lower limit of the data inclusion window range, and when the stability measure indicates that the cause-effect relationship is unstable (i.e., dynamically changing), the system may decrease the upper limit or both the upper limit and the lower limit. This allows the system to explore smaller data inclusion windows and ignore older data when there is a higher probability that the environmental characteristic is changing, and explore larger data inclusion windows when there is a higher probability that the environmental characteristic is stable.

[0279] As another example, the system may adjust the range based on the shape of the causal model as described above. In particular, the system may explore the range of longer data inclusion windows when the magnitude of the impact on the measurement increases as the data inclusion window becomes higher, and explore the range of smaller data inclusion windows when the magnitude of the impact on the measurement increases as the data inclusion window becomes shorter. In other words, the system may move the range downward when the difference decreases, and move the range upward when the difference increases. This allows the system to explore smaller data inclusion windows and ignore older data when the probability that the environmental characteristics are changing is high.

[0280] In some cases, the system may apply some combination of these heuristics, for example, by allowing the upper limit to increase based on either or both of the latter two examples, as long as the upper limit does not exceed the size corresponding to the upper threshold on the statistical power curve, and allowing the lower limit to decrease based on either or both of the latter two examples, as long as the lower limit does not fall below the size corresponding to the lower threshold on the statistical power curve.

[0281] Although these examples are described with respect to data inclusion windows, similar heuristics can also be used to adjust the ratio of mixed instances to baseline instances, i.e., increasing the number of baseline instances when there is a high probability that the environmental characteristics are changing or have recently changed, and decreasing the number of baseline instances when there is a high probability that the environmental characteristics are stable.

[0282] Fig.12 The system (in Figures 12 to 18 The performance of a system (denoted as “DCL” in FIG. 1 ) in controlling an environment is relative to the performance of a system controlling the same environment using an existing control scheme. In particular, Fig.12 The performance of the system is shown compared to three different types of existing control schemes: (i) a "none" scheme, in which the system does not select any settings and only receives a baseline environmental response; (ii) a "random" scheme, in which the system randomly assigns control settings without replacement; and (iii) various prior art reinforcement learning algorithms.

[0283] exist Fig.12 In the example of , the environment being controlled has 3 controllable elements, each with 5 possible control settings, and the value of the performance metric at each iteration is drawn from a Gaussian distribution that is always fixed. The application of a particular control setting changes the parameters of the Gaussian distribution from which the value of the performance metric is drawn. These characteristics are similar to those that exist in simple or highly controlled real-world environments (such as certain production lines), but without the additional complexity that may be encountered in more complex real-world environments.

[0284] Fig.12The upper set of graphs in shows the performance of each system in terms of Mean Cumulative FOM ("MeanCumFOM"). Mean Cumulative FOM at any given iteration is the average of the performance metrics (i.e., FOM) received starting from the first iteration until the given iteration, i.e., the cumulative average performance metric value across time.

[0285] Fig.12 The lower set of graphs in shows the performance of each system with a mean FOM per instance ("MeanFOM"). The mean FOM per instance at any given iteration is the average of the performance metrics received for that instance at the given iteration, i.e., without considering previous iterations.

[0286] In general, the first column ("DCL") shows the results for the system, while the remaining columns show the results for existing control schemes.

[0287] As indicated above, Fig.12 The environment for which the results are shown is less complex than many real-world environments, e.g., since the causal effects are fixed, there are no external uncontrollable characteristics that affect the performance measures, and there is no uncertainty about the spatial or temporal extent. However, even in this relatively simple environment, the performance of the system meets or exceeds that of prior art systems, whether or not advanced features are enabled.

[0288] A description of a prior art system for benchmarking system performance is as follows:

[0289] BGE–Boltzmann-Gumbel Exploration [Cesa-Bianchi et al., Boltzmann Exploration Done Right, Conference on Neural Information Processing Systems (NeurIPS), 2017] is a multi-arm Bandit algorithm that uses an exponentially weighted approach for control setting assignment selection. It maintains a distribution over the FOM for each control setting assignment. At each step, samples are generated from each of these distributions, and the algorithm selects the control setting assignment that corresponds to the maximum sample. The internal parameters of the distribution are then updated using the received feedback.

[0290] Ep Greedy – Epsilon Greedy is a generalized multi-arm Bandit algorithm that selects a random control setting assignment with probability ε and selects the control setting assignment that gave the highest average FOM in the past with probability 1-ε. In practice, it explores ε% of the time and exploits 1-ε% of the time.

[0291] UCB – Upper Confidence Bar (UCB) [Auer et al., Finite-time Analysis of the Multiarmed Bandit Problem, Machine Learning, 2002] The Multiarmed Bandit algorithm is one of the two basic approaches to solving the Multiarmed Bandit problem. It works by computing the average FOM and confidence intervals from historical data. It selects a control setting allocation by computing the control setting allocation with the highest average FOM plus the confidence interval. In this way, it is optimistic about the potential FOM of the control setting allocation and learns over time which control setting allocation has the highest FOM.

[0292] Lin UCB - LinUCB [Li et al., A Contextual-Bandit Approach to Personalized News Article Recommendation, International World Wide Web Conference (WWW), 2010] constructs UCB by maintaining the mean FOM and confidence interval, and makes a key assumption: the expected FOM is a linear function of the characteristics of the program instances in the experiment and the control setting assignments. The algorithm is then able to select the best control setting assignment for any individual program instance. Lin UCB is expected to perform best when the ideal control setting assignment is different for different groups of program instances.

[0293] Monitored UCB – Monitored UCB [Cao et al., Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Bandit, International Conference on Artificial Intelligence and Statistics (AISTATS), 2019] constructs UCB by computing mean FOM and confidence intervals, but is designed for environments where sudden changes in FOM can occur. Therefore, it incorporates a change point detection algorithm that identifies when the FOM changes and resets internal parameters (effectively resetting the mean FOM and confidence intervals) to start learning a new FOM. Monitored UCB is expected to perform well (outperforming UCB and variants) in environments where sudden changes in FOM occur.

[0294] ODAAF – Optimization with Delayed, Aggregated Anonymous Feedback [Pike-Burke et al., Bandits with Delayed, Aggregated Anonymous Feedback, International Conference on Machine Learning (ICML), 2018] is a multi-armed Bandit algorithm designed to work in settings where feedback is subject to random bounded delays. Feedback is additively aggregated and anonymized before being sent to the algorithm, which makes this setting significantly more challenging. The algorithm proceeds in stages, maintaining a set of candidates for possible optimal control setting assignments. In each stage, it iteratively cycles through these candidates and updates the performance metric estimates for these candidates as it receives feedback. At the end of each stage, the algorithm eliminates candidates for which the estimated performance metric is significantly suboptimal.

[0295] Thompson Sampling – Thompson sampling [Agrawal and Goyal. Analysis of Thompson Sampling for the Multi-armed Bandit Problem, Conference on Learning Theory (COLT), 2012] is a probabilistic matching algorithm and is another fundamental approach to solving the multi-armed bandit problem (the other approach is optimization-based methods such as UCB). It works by maintaining a distribution over the estimated FOM for each control setting assignment option, sampling from each distribution, and then selecting the control setting assignment option with the highest sampled (estimated) FOM. Once the true FOM is observed, the (posterior) distribution is updated using a Bayesian approach. The algorithm selects control setting assignments proportionally to the probability that each control setting assignment is the optimal control setting assignment.

[0296] Fig.13 The performance of the system is shown relative to the performance of multiple other systems in controlling multiple different environments.

[0297] In particular, each of the other systems uses a corresponding one of the above-mentioned existing control schemes to control a plurality of different environments.

[0298] The environments being controlled each have 3 controllable elements, each controllable element has 5 possible settings, and the value of the performance metric being optimized at each iteration is drawn from a Gaussian distribution.

[0299] The environments have varying degrees of complexity by adding various factors that cause variability between different program instances.

[0300] In particular, the base environment shown in the set of top figures varies the mean and variance of the Gaussian distribution according to the program instance, ie, so that different program instances may receive different performance metric values ​​even if the same control settings are selected.

[0301] Other environments also introduce time-based variations in the effects of applying different possible settings for controllable elements, the underlying sinusoidal behavior in the performance metrics, and the effects of different settings for different groups of instances (i.e., which represent interactions between environmental characteristics and controllable elements).

[0302] As from Fig.13 It can be seen that many existing control schemes generally perform well in simple baseline situations, and that a given control scheme may perform well with an additional complexity factor, but no existing control scheme performs well in all situations. On the other hand, the system performs equivalent to or better than the best existing control scheme in all environments. Therefore, Fig.13 The examples show that the system is able to perform similarly or better than possible settings of other control schemes for each of the different environments, due to the system's ability to automatically adapt to changing complex environments without the need for manual model selection (i.e., by continually changing the system's internal parameters to account for the different characteristics of different environments), even when no prior knowledge of the environmental characteristics is available.

[0303] A detailed explanation of each of the environments being controlled is given below.

[0304] 00_base – 100 program instances; 3 controllable elements with 5 possible settings, each with a performance metric value drawn from a Gaussian distribution. Selecting different possible settings for IV changes the mean and / or standard deviation of the distribution. This environment is relatively simple, but does have many combinations of possible control settings as commonly found in real-world environments.

[0305] 01_add_subject_var – Starting from 00_base, splits the program instances into 3 groups that have different base rate means and standard deviations for their performance metric value distributions. This introduces additional variance in the data without changing the impact of the control setting assignments. This type of program instance / EU variance is very typical for the real world. For example, this particular configuration mimics the sales behavior of various products, where a small group of products accounts for the majority of the overall sales (80 / 20 rule), a larger group of products has medium sales, and most products have low sales.

[0306] 02_add_dynamic – Starting from 00_base, the effects of the possible settings of the IV undergo multiple transitions at predetermined times (which are unknown to the algorithm), causing the effects of the possible settings of the IV to reverse. This changing behavior is very typical for the real world. For example, the effectiveness of different advertising campaigns and technologies often changes over space and time (what worked before may not work now). Similarly, the optimal control setting distribution selection on a production line will vary due to factors such as temperature, humidity, and nuances of specific equipment (e.g., wear and tear).

[0307] 03_add_subject_var_dynamic – A combination of 01_add_subject_var and 02_add_dynamic. The combination of these two behaviors (above) makes this environment even more similar to many dynamic real-world environments.

[0308] 04_add_sine – Starting from 00_base, an overall sinusoidal pattern is added to the performance metric values. This simulates periodic trends in the FOM (e.g., seasonal, weekly) that are independent of the effects that the IV may be set to. Some algorithms have difficulty handling the additional data variance. This type of periodic behavior is very typical of the real world. For example, retail sales, supply chains, etc. often follow weekly, monthly, and seasonal cycles that introduce significant differences in performance metrics. As another example, manufacturing and other processes that are affected by seasonal changes in weather may also experience similar effects. The key challenge in these cases (which the system addresses) is to be able to distinguish the effects of marketing activities (for example) from these base behaviors.

[0309] 05_add_subject_var_sine – A combination of 01_add_subject_var and 04_add_sine. The combination of these two behaviors (above) makes the environment even more similar to complex and dynamic real-world environments.

[0310] 06_add_ev_effect – The optimal combination of IV possible settings is different for some program instances. Such variations in the allocation of control settings are very typical for real world situations. For example, different advertising or promotion methods will work better than others depending on the product involved, the recipient of the content, space, time, etc.

[0311] 10_complex - a combination of 01_add_subject_var, 02_add_dynamic, 04_add_sine, and 06_add_ev_effects. This environment does the most to capture real world behavior, as it takes all of the above real world behaviors and combines them into one environment.

[0312] Fig.14 The performance of the system is shown relative to the performance of multiple other systems in controlling multiple different environments with different temporal effects.

[0313] In particular, each of the other systems uses corresponding existing control schemes to control multiple different environments.

[0314] The environment being controlled has 4 controllable elements, each with 2 possible settings, and the performance metric at each iteration is drawn from a Gaussian distribution. The environments are imposed with different time delays and durations that affect the time at which the performance metric is generated relative to the initial application of the control settings for a given instance. For example, in the top environment, the environmental response for all effects is delayed by 2 time iterations and lasts for 3 time iterations. In the following environments, all 4 controllable elements have different time delays and durations. The third and fourth environments add additional complexity and variability.

[0315] As from Fig.14 As can be seen from the examples, the system is able to perform similarly or better than other control schemes for each of the different environments. This illustrates the ability of the system to dynamically adapt to the temporal behavior of the effects of applied control settings (ie, by changing the temporal horizon parameters during operation).

[0316] Furthermore, two of these environments include underlying periodic behaviors that are independent of the IV control setting dispensing effects. Such behavior is typical for situations encountered in the real world (e.g., advertising, pharmaceuticals) where actions taken have delayed rather than immediate effects. At the same time, such scenarios often have residual effects that persist after the control setting dispensing has ceased. Furthermore, these temporal behaviors are rarely found in isolation. Instead, they will most often occur simultaneously with the underlying behavior, similar to the sinusoidal pattern shown. As can be seen from Fig.14 It can be seen that the described system outperforms conventional systems, namely due to being able to better account for different temporal behaviors by adjusting the time range parameter and other internal parameters for adjusting the underlying behavior variations.

[0317] Fig.14 Details of the environment shown are as follows.

[0318] 00_temporal – 500 program instances; 4 controllable elements with 2 possible settings, each with a performance measure drawn from a Gaussian distribution. Selecting different possible settings for the IV changes the mean and / or standard deviation of the distribution. Performance measures for all effects are delayed by 2 temporal iterations and persist for 3 iterations.

[0319] 01_temporal_multi – Same as 00_temporal, except that the 4 controllable elements have different time delays and durations.

[0320] 02_temporal_sine – starts from 00_base, adds sine behavior

[0321] 03_temporal_delay_only – Same as 00_temporal, but with duration behavior removed

[0322] 04_temporal_multi_delay_only – Same as 01_time_multi, but with duration behavior removed

[0323] 05_temporal_sine_delay_only – Same as 02_temporal_sine, but with duration behavior removed

[0324] Fig.15 The performance of the system with and without clustering is shown. The environment being controlled has 3 controllable elements, each with 5 possible settings, and the performance metric at each iteration is derived from a Gaussian distribution that is fixed throughout the experiment. The environment being controlled has different optimal control setting allocations (controllable elements) depending on the characteristics of the program instance / EU described by the environment characteristics. One set of control setting allocations will produce overall good results, but these results are actually negative for subpopulations. If the subpopulation is given its specific ideal control setting allocation, the overall utility is improved. This is typical for real-world situations where the optimal control setting allocation can vary greatly based on external characteristics. The left figure shows the performance of the system when a clustering component is included. In this case, the system allocates specific control setting allocations to the program instance / EU, which results in an overall higher FOM. The right figure shows the performance of the system without using a clustering component (i.e., without ever entering the clustering stage). In this case, the algorithm utilizes a single overall control setting allocation method for all program instances, which causes the algorithm to use a non-optimal control setting allocation for a subpopulation. As can be seen from Fig.15 It can be seen that the system performs better when clustering is used.

[0325] Fig.16 The performance of the system capable of varying the data inclusion is shown relative to the performance of the system controlling the same environment while keeping the data inclusion window parameters fixed. Fig.16 In the example of , the environment being controlled exhibits two gradual changes in the relative effects of control settings on performance measures. This is typical for the real world in two ways: 1) the effects of actions (e.g., advertising, manufacturing parameters) are rare and, if any, static; and 2) when such changes occur, they are typically gradual in nature rather than abrupt. The left figure shows the performance of the system with the DIW component included. In this case, the system is able to quickly detect that the effect has changed, for example, by hybrid baseline comparison, and the system can immediately relearn the optimal control setting allocation by narrowing the data inclusion window. The right figure shows the performance of the system without the DIW component. In this case, the algorithm adapts very gradually to handle the change in the effect. By the time it does so, the effect has changed again.

[0326] Fig.17 The performance of the system is shown with and without temporal analysis (i.e., with and without the ability to change the time horizon). The environment being controlled has 4 controllable elements, each with 2 possible settings, and the performance metric at each iteration is drawn from a Gaussian distribution that is fixed throughout the experiment. The environments are imposed with different time delays and residual behaviors that affect the time when the performance metric is generated relative to the initial application of the IV possible settings. In addition, two of these environments include underlying periodic behavior that is independent of the effect. This behavior is typical for situations encountered in the real world (e.g., advertising, pharmaceuticals) because actions often taken do not have immediate effects, and they often have residual effects even after the control setting assignment has ceased. In addition, such temporal variations often exist in the context of other underlying behaviors. The figure shows the values ​​of temporal optimization within the system. The left column shows the performance of the system using the temporal component. The right column shows the performance of the system without the temporal component. As can be seen from Fig.17 As can be seen from the examples, when the environment has these temporal characteristics, the system performs significantly better when using temporal analysis.

[0327] Fig.18 The performance of the system in controlling an environment is shown relative to the performance of a system controlling the same environment using an existing control scheme ("Lin UCB"). Fig.18In the example of , the environment being controlled has a cyclical underlying behavior that is independent of the effects of the IV possible settings and changes in those effects, such that the optimal control setting assignments change over time. These characteristics are similar to those that exist in many real-world environments where there are regular underlying dynamics (e.g., weekly, monthly, or quarterly patterns) and the effects of control setting assignments / actions change over time. Fig.18 A subset of the time periods is shown during which the effect of the possible settings of the IV is changed in the base environment (during iterations 200 to 250). Fig.18 It can be seen that the performance of the existing control scheme remains in the utilization stage based on the previous control setting allocation effect and cannot adapt to changes quickly. On the other hand, the performance of the system quickly adapts to the effect of the change and finds incremental improvement under the environmental effect after the change (upper graph). This leads to an increased incremental benefit from the system (lower graph). It should be noted that the cumulative benefit of utilizing the system will continue to increase over time.

[0328] Although the above description uses certain terms to refer to the features of the system or the actions performed by the system, it should be understood that these terms are not the only terms that can be used to describe the operation of the system. Some examples of alternative terms are as follows. For example, a controllable element may alternatively be referred to as an independent variable (IV). As another example, an environmental characteristic may alternatively be referred to as an external variable (EV). As another example, an environmental response may alternatively be referred to as a dependent variable (DV). As another example, a program instance may alternatively be referred to as an experimental unit or a self-organizing experimental unit (SOEU). As another example, the possible settings of a controllable element may alternatively be referred to as the level of an element (or IV). As another example, a control setting may alternatively be referred to as a process decision, and the allocation of a control setting to a program instance may be referred to as a process allocation.

[0329] In this specification, the term "repeatedly", i.e., in the context of repeatedly performing an operation, is generally used to mean that the operation occurs multiple times, with or without a particular sequence. For example, a process may constantly or iteratively follow a set of steps in a specified order, or may follow these steps randomly or non-sequentially. In addition, the steps may not all be performed with the same frequency, for example, the frequency of performing processing allocations may be higher than the frequency of updating causal learning, and the frequency of the latter may change over time, for example, as the utilization phase becomes dominant and / or as computing power / speed requirements change over time.

[0330] This specification uses the term "configured to" in conjunction with systems and computer program components. For a system of one or more computers to be configured to perform specific operations or actions, it means that software, firmware, hardware, or a combination thereof has been installed on the system, which, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform specific operations or actions, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0331] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs. One or more computer programs may include one or more modules of computer program instructions, which are encoded on a tangible non-transitory storage medium for data processing equipment to perform or control the operation of data processing equipment. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, an optical signal, or an electromagnetic signal) to be transmitted to a suitable receiver device for execution by a data processing device, and the propagation signal is generated to encode information.

[0332] The term "data processing apparatus" refers to data processing hardware, and encompasses all types of equipment, devices, and machines for processing data, including (by way of example) a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or also include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0333] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may (but need not) correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program under consideration, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0334] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed in a different way.

[0335] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally speaking, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0336] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit and one or more programmed computers.

[0337] Computers suitable for executing computer programs may be based on general or special microprocessors or both, or any other type of central processing unit. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by a dedicated logic circuit, or may be incorporated into a dedicated logic circuit. Generally speaking, the computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or will be operably connected to receive data from a mass storage device, or to transfer data to a mass storage device, or both. However, the computer does not have to have these devices. In addition, the computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few.

[0338] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including (by way of example) semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0339] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) for displaying information to a user), a keyboard, and a pointing device (e.g., a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide for interaction with a user; for example, feedback provided to a user may be any form of sensory feedback, such as visual feedback, audio feedback, or tactile feedback; and input from a user may be received in any form, including audio, speech, or tactile input. In addition, a computer may interact with a user by sending files to or receiving files from a device used by a user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in return.

[0340] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with a specific implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0341] A computing system may include a client and a server. The client and the server are usually far apart from each other and generally interact through a communication network. The reason for the client and server relationship is that the computer programs that run on the respective computers and have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML web page) to a user device, for example, for the purpose of displaying data to a user interacting with a device used as a client or receiving user input from the user. Data generated at the device, for example, due to user interaction, can be received from the user device at the server.

[0342] Although this specification includes many specific implementation details, these specific implementation details should not be understood as limitations on the scope of any invention or the scope of the claimed content, but should be understood as descriptions of features that may be specific to specific embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as functioning in certain combinations and thus even initially protected by claims, one or more features derived from a claimed combination may be separated from the combination in some cases, and a claimed combination may involve sub-combinations or variations of sub-combinations.

[0343] Similarly, although the operations shown in the drawings and described in the claims have a specific order, this should not be understood as requiring that these operations must be performed in the specific order shown or in a sequential order, or that all of the operations shown must be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and should be understood that the program components and systems can generally be integrated together in a single software product or can be packaged into multiple software products.

[0344] Specific embodiments of the subject matter of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired results. For example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for automatically updating a causal model, the method comprising: receiving one or more external inputs via a control system configured to automatically optimize a measure of quality of a batch of a biopharmaceutical; generating, by the control system and based on the one or more external inputs, one or more baseline values ​​for a plurality of internal parameters, wherein the plurality of internal parameters define how the control system updates a causal model and how the control system determines which control settings to select given a current causal model; accessing, by the control system, a current causal model that measures one or more causal relationships between one or more control settings and a measure of quality of the batch of biopharmaceutical; selecting, by the control system and based on the accessed current causal model and the plurality of internal parameters, one or more control settings specifying one or more controllable aspects of the batch of the biopharmaceutical; Use the control system to repeatedly perform the following operations: selecting a configuration of input settings for manufacturing the batch of the biopharmaceutical based at least in part on the one or more control settings and the current causal model; automatically determining a measure of quality of the manufactured batch of the biopharmaceutical based at least in part on the one or more control settings; as well as The current causal model is automatically adjusted based on a measure of quality of the biopharmaceutical batch. 2 . The method according to claim 1 , further comprising adjusting the internal parameters based on the adjusted causal model.

3. The method of any one of claims 1 or 2, wherein the measure of the quality of the batch comprises one or more of: the yield of the batch; a measure of the activity of the batch; a measure of the efficacy of the batch; the measure of impurities in the batch; A measure of the viability of transfected cells; A measure of transfection efficiency; a measure of the purity of the biopharmaceutical; a measure of monodispersity; or A measure of cell density.

4. The method of claim 3, wherein the purity of the biopharmaceutical comprises one or more of the following: measurement of host cell protein impurities; A measure of DNA impurities; A measure of RNA impurities; or A measure of host cell metabolite impurities.

5. The method of claim 1, wherein the control settings include one or more of the following: one or more settings associated with a processing step; One or more settings related to the chemical composition of a substance; One or more settings related to physical controls during manufacturing; or One or more settings associated with purifying the biopharmaceutical batch.

6. The method of claim 5, wherein the settings associated with a processing step include one or more of the following: the sequence of the processing steps; The time spent in each processing step; the time spent in each subprocess of a given processing step; or Selection of parameters for each processing step.

7. The method according to any one of claims 5 or 6, wherein the settings related to chemical composition include selection of one or more of the following: Nutrient medium; Ion exchange resins; Affinity chromatography matrices; polymer; Wash buffer; or Elution buffer.

8. The method of claim 5, wherein the settings related to physical control during manufacturing include one or more of the following: Manufacturing time; Purification time; Temperature during manufacturing; Temperature during purification; humidity during manufacturing; Humidity during purification; flow rates used during manufacturing; the flow rate used during purification; The g-forces used during manufacturing; The measure of magnetic force used during manufacturing; The measurement of electricity used during manufacturing; or A measure of agitation used during manufacturing.

9. The method of claim 5, wherein the settings associated with purifying the biopharmaceutical batch include purifying the batch using one or more of: Electrostatic interactions; hydrophobic interactions; Hydrophilic interactions; Affinity chromatography; or Size Exclusion Chromatography.

10. The method of claim 1, wherein: Selecting the configuration further comprises selecting the configuration based on the causal model and a predetermined set of external variables; and The method also includes adjusting a control parameter that parameterizes the effect of the predetermined set of external variables.

11. The method of claim 10, wherein the predetermined set of external variables includes one or more external variables related to ambient climate during manufacturing.

12. The method of claim 11, wherein the external variables related to ambient climate include one or more of the following: humidity during manufacturing; the external temperature during manufacture; or Light levels during manufacturing.

13. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of claim 1.

14. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of claim 1.

15. The method of claim 1, wherein the plurality of internal parameters comprises at least one of: one or more spatial ranges, and one or more temporal ranges.

16. The method of claim 1, wherein the external input defines a search space of possible combinations of control settings.

17. The method of claim 1, wherein the internal parameters include one or more clustering parameters that define hyperparameters of a clustering technique used by the control system.

Images