Online constrained reinforcement learning method and device for single-constraint system to be controlled

Abstract

The present invention provides an online constrained reinforcement learning method and device for a single-constraint system to be controlled, comprising: sampling simulation results of actions on the basis of a preset sampling quantity, and determining an observed optimal action, feasible actions and infeasible actions on the basis of the sampled simulation results; calculating Q factors and sample variances of the feasible actions and the optimal action, and constraint function QH factors and sample variances of the infeasible actions; using a sampling data allocation algorithm to obtain a target sample size of each action; determining a supplementary sampling quantity of each action on the basis of the target sample size of each action; using supplementarily sampled simulation results to redetermine Q factors and sample variances of the feasible actions and the optimal action, and constraint function QH factors and sample variances of the infeasible actions; and adjusting a total consumed sampling quantity, determining whether a total consumed sample size satisfies an action sampling quantity constraint condition, and if yes, outputting the optimal action to a system to be controlled. The present invention can enhance the performance of optimization strategies.

Description

An online constraint reinforcement learning method and apparatus for single-constraint controlled systems Technical Field

[0001] This invention relates to the field of reinforcement learning, and more particularly to an online constraint reinforcement learning method and apparatus for single-constraint controlled systems. Background Technology

[0002] In existing technologies, the operation and scheduling of multi-energy microgrids in complex systems can be achieved using intelligent online learning (e.g., reinforcement learning). Specifically, existing online learning algorithms mainly include the following two categories:

[0003] The first type of learning process includes: first, establishing the objective function and constraints; then, optimizing the performance of the objective function to obtain a solution that satisfies the constraints; and finally, judging the feasibility of the solution (i.e., performing strategy feasibility processing) to obtain all the final feasible strategies.

[0004] The second type of learning process includes: first, pre-establishing the objective function and constraints; then, establishing an alternative objective function that is compatible with the performance and constraints of the objective function; and finally, optimizing the alternative objective function.

[0005] The first type of method addresses feasibility first, then optimization. However, since there are often many feasible solutions, in practical applications, optimization is performed based on a subset of feasible solutions, limiting the search scope of feasible strategies to a proper subset of the feasible region. Therefore, it is difficult to guarantee the performance of the final feasible strategy. The second type of method replaces the constrained problem with an unconstrained problem, making it difficult to guarantee the feasibility of the strategy during the learning process and thus unsuitable for online use.

[0006] Summary of the Invention

[0007] This invention addresses the problems of low efficiency in determining control strategies and poor control performance in the online optimization control of single-constraint controlled systems in the prior art.

[0008] To address the aforementioned technical problems, the present invention provides a method comprising:

[0009] Simulate each action and sample the simulation results of each action according to a preset sampling amount. Determine the optimal action, feasible action, and infeasible action based on the sampled simulation results.

[0010] Calculate the Q-factor and sample variance of each feasible action and optimal action based on the simulation results; calculate the constraint function Q of each infeasible action based on the simulation results. H Factors and sample variance; record the sample size consumed for each action and the total sample size consumed;

[0011] Using a sampling data allocation algorithm, the Q-factor and sample variance of each possible action and the optimal action, and the constraint function Q of each incapable action are determined. H Factors, sample variance, and total consumed samples are analyzed to obtain the target sample size for each action; the sampling data allocation algorithm is determined based on the objective function and constraint function of the system under control, and is used to asymptotically maximize the probability of correctly selecting the optimal action in each state;

[0012] The additional sampling amount for each action is determined based on the target sample size and the consumed sample size for each action.

[0013] Supplementary sampling is performed based on the supplementary sampling amount for each action. The simulation results of the supplementary sampling are used to redetermine the Q-factor and sample variance of the feasible actions and the optimal action, as well as the constraint function Q of each infeasible action. H Factors and sample variance;

[0014] Adjust the total amount of samples consumed, and determine whether the total amount of samples consumed is less than the preset total amount of samples. If so, recalculate the target amount of samples and subsequent steps. If not, output the final optimal action to the system under control.

[0015] A second aspect of the present invention provides an online constraint reinforcement learning device for a single-constraint controlled system, the device comprising:

[0016] The simulation unit is used to simulate each action and sample the simulation results of each action according to a preset sampling amount. Based on the sampled simulation results, the optimal action, feasible action, and infeasible action are determined.

[0017] The first calculation unit is used to calculate the Q-factor and sample variance of each feasible action and optimal action based on the simulation results of each feasible action and optimal action; and to calculate the constraint function Q of each infeasible action based on the simulation results of each infeasible action. H Factors and sample variance; record the sample size consumed for each action and the total sample size consumed;

[0018] The second calculation unit is used to allocate sampling data using an algorithm to determine the Q-factor and sample variance of each possible action and the optimal action, as well as the constraint function Q of each incapable action. H Factors, sample variance, and total consumed samples are analyzed to obtain the target sample size for each action; the sampling data allocation algorithm is determined based on the objective function and constraint function of the system under control, and is used to asymptotically maximize the probability of correctly selecting the optimal action in each state;

[0019] The third calculation unit is used to determine the supplementary sampling amount for each action based on the target sample size and the sample size already consumed for each action.

[0020] The supplementary sampling unit is used to perform supplementary sampling based on the supplementary sampling amount of each action, and to redetermine the Q-factor and sample variance of the feasible actions and the optimal action, as well as the constraint function Q of each infeasible action, using the simulation results of the supplementary sampling. H Factors and sample variance;

[0021] The control unit is used to adjust the total amount of samples consumed, determine whether the total amount of samples consumed is less than the preset total amount of samples, if so, recalculate the target amount of samples and subsequent steps, if not, output the final optimal action to the system under control.

[0022] A third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the foregoing embodiments.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor of a computer device, implements the method described in any of the foregoing embodiments.

[0024] The present invention provides an online constraint reinforcement learning method and apparatus for single-constraint control systems. Applicable to control systems with a single constraint (i.e., one constraint condition), the method pre-determines a sampling data allocation algorithm based on the objective function and constraint function of the control system. This algorithm asymptotically maximizes the probability of correctly selecting the optimal action in each state, concentrating simulation resources on actions that truly need to be distinguished. This leads to increasingly accurate observations of each action, ultimately yielding the optimal action. Compared to existing technologies, this method considers the entire set of feasible strategies and ensures the estimability of the optimal action selection, thus improving the performance of the final action determination. Furthermore, by controlling the sampling amount, this invention also improves the data efficiency of online reinforcement learning during the operation and scheduling of the control system, significantly reducing the time required for online decision-making and decreasing data usage.

[0025] To make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 shows a flowchart of the process for determining the criteria of the sampling data allocation algorithm according to an embodiment of the present invention;

[0028] Figure 2 shows a flowchart of the process for determining the objective function of the approximate optimal action selection probability function and the action sampling amount constraint conditions in an embodiment of the present invention.

[0029] Figure 3 shows a flowchart of the online constraint reinforcement learning method for a single-constraint controlled system according to an embodiment of the present invention;

[0030] Figure 4 shows a flowchart of the process for determining the sampling amount of each action target in an embodiment of the present invention;

[0031] Figure 5 shows a structural diagram of the online constraint reinforcement learning device for a single-constraint controlled system according to an embodiment of the present invention.

[0032] Figure 6 shows a structural diagram of a computer device according to an embodiment of the present invention;

[0033] Figure 7 shows the flowchart of the online constraint reinforcement learning method for single-constraint controlled systems according to an embodiment of the present invention.

[0034] Explanation of symbols in the attached drawings:

[0035] 501. Simulation Unit;

[0036] 502. First Calculation Unit;

[0037] 503. Second Calculation Unit;

[0038] 504. Third Calculation Unit;

[0039] 505. Supplementary sampling unit;

[0040] 506. Control unit;

[0041] 602. Computer equipment;

[0042] 604, Processor;

[0043] 606. Memory;

[0044] 608. Drive mechanism;

[0045] 610. Input / output module;

[0046] 612. Input devices;

[0047] 614. Output devices;

[0048] 616. Presentation equipment;

[0049] 618. Graphical User Interface;

[0050] 620. Network interface;

[0051] 622. Communication link;

[0052] 624. Communication bus. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0055] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0056] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0057] It should be noted that the online constraint reinforcement learning method and apparatus for single-constraint controlled systems of the present invention can be applied to the field of multi-energy microgrids, such as scenarios where multi-energy microgrids charge electric vehicles via charging piles. It can also be applied to any field other than the power sector, such as the field of autonomous driving scheduling. The present invention does not limit the application field of the online constraint reinforcement learning method and apparatus for single-constraint controlled systems. Any complex system with single-constraint and online control requirements can be controlled online using the method of the present invention.

[0058] For a single-constraint control system, before implementing the online constraint reinforcement learning method, a sampling data allocation algorithm needs to be established in advance, as shown in Figure 1, including:

[0059] Step 101: Based on the equipment information and load information, system status, control actions, optimization target information and constraint requirements of the system to be controlled, establish the objective function and constraint conditions of the control strategy.

[0060] Step 102: Transform the objective function to obtain the action Q-factor for reinforcement learning, and transform the constraint conditions of the constraint function to obtain the constraint function Q for reinforcement learning. H Constraints on the factors.

[0061] Step 103, based on the action Q factor and constraint function Q H The constraints of the factors are used to establish the objective function of the approximate optimal action selection probability function and the constraints of the action sampling amount.

[0062] Step 104: Determine the criteria for the sampling data allocation algorithm based on the objective function of the near-optimal action selection probability function and the action sampling amount constraint.

[0063] In detail, the equipment information, load information, system status, control actions, optimization target information, and constraint requirement information in step 101 can be determined according to the specific situation of the system to be controlled, and the present invention does not limit the specific parameters included therein. An objective function is established based on the equipment information, load information, system status, control actions, and optimization target; and constraint conditions for the constraint function are established based on the equipment information, load information, system status, control actions, and constraint requirement information.

[0064] In one embodiment, the objective function and the constraint conditions of the constraint functions can be expressed using the following formulas:

[0065] Where J is the objective function, s t Let d be the state of the system to be controlled at time t, and d be the strategy. t () represents policy d in state s tThe following actions are defined as follows: c is the target object function, T is the simulation time period length, h is the constraint function, and H0 is a given constant.

[0066] Taking a multi-energy microgrid as an example, a multi-energy microgrid is a network with multiple energy sources (such as wind power, solar power, and urban power grid) and multiple devices (such as electrical appliances, energy storage devices, and power generation devices). To reduce costs, wind power, solar power, fuel cells, and other power generation devices and energy storage devices are built in-house. By rationally scheduling the devices, especially the charging and discharging of energy storage devices, this multi-energy microgrid can charge electric vehicles at a lower electricity price than the urban power grid through charging stations and charge a charging service fee. While the charging completion time of electric vehicles has some flexibility, it cannot be delayed indefinitely; unmet charging and electricity demands will incur penalties. Therefore, in the scenario of charging electric vehicles through a multi-energy microgrid, the optimization objective is to schedule the charging power of each device (especially energy storage devices and charging stations) to minimize operating costs, with the constraint that the penalty for breach of contract is less than a predetermined value.

[0067] Correspondingly, J above represents the target cost of operating a multi-energy microgrid, that is, the average cost of operating a multi-energy microgrid should be minimized; s t d represents the state of the multi-energy microgrid at time t, including but not limited to solar power generation, wind power generation, number of people in each room of the building, IT load, electric vehicle charging demand, energy storage level of energy storage devices, and real-time electricity price of the main grid; d is the strategy; d(s t ) represents the control actions of the system under control, including charging, discharging, or not charging or discharging the energy storage device, the charging power of each charging pile, and the electricity purchased from the city power grid; c represents the cost of purchasing electricity from the city power grid minus the charging revenue collected from electric vehicle users; h represents the penalty for not meeting the electricity demand at time t; H(d) represents that the average penalty should not exceed the given value H0.

[0068] When implementing step 102 above, the action Q factor and action constraint function Q can be established according to the above formulas (1) and (2). H :

[0069] Where Q(s,a) is the action Q-value function in state s, Q H (s,a) is the action constraint function Q under state s. H c is the target object function, h is the constraint function, s' is the future state, and P a (s,s') represents the probability of transitioning to the next state s' when taking action a in state s, and Q(s',d(s')) is the action Q factor in state s'. H (s',d(s')) is the action constraint function Q under state s'.H factor.

[0070] Furthermore, due to P a (s,s') is usually not obtainable. The action value function of formula (3) and the action constraint function of formula (4) can be obtained using the following observation formula:

[0071] in, Let Q(s,a) be an estimate. For Q H The estimated value of (s,a), where k is the iteration number and N is the number of iterations. a The number of samples for action a.

[0072] Based on the problem structure and formula (5) of online reinforcement learning, the objective function and constraints represented by formulas (1) and (2) are transformed to obtain the optimization model represented by the following formula: stH k+1 (d k+1 )≤H0 (8)

[0073] Where 'a' represents an action, taking multi-energy microgrid control as an example, actions include charging and discharging energy storage devices, or neither charging nor discharging, the charging power of each charging pile, or whether charging is performed; A is the action space, which is a set consisting of all possible actions. is the observed value of the Q factor, and k is the iteration number.

[0074] Compared to the problems in formulas (1) and (2), the problem in formulas (7) and (8) decomposes the performance requirements for the entire strategy d into the requirements for action optimization under the current state s during online operation.

[0075] Convert formula (8) into constraint function Q. H The constraints on the factors, specifically the transformations, include:

[0076] When the strategy d for the (k+1)th iteration is obtained k+1 At that time, the (k+1)th policy d k+1 With the kth strategy d k The difference lies in state s, from which we can derive:

[0077] Where, d k+1 (s)≠d k (s);

[0078] Furthermore, we can obtain:

[0079] make D = π k (s), thus we can obtain the following formula:

[0080] Combining the above equation with formula (8), we can deduce the following constraint function Q. H Factor constraints:

[0081] Where D is π k Q(s) represents the steady-state probability of state s in the k-th iteration; H k+1 (s,a) represents the value of the value function defined by the constraint function in the (k+1)th iteration; a is the action, and H0 is the preset value;

[0082] π k (s') represents the steady-state probability of state s' in the k-th iteration. In the kth iteration, action d is taken in state s'. k The constraint function Q of (s′) H Factor value.

[0083] The corresponding objective function is expressed as:

[0084] This step explicitly transforms the feasibility determination problem of strategy d into the feasibility determination problem of different actions a. This decoupling decisively improves the data efficiency of the entire algorithm.

[0085] When the above step 103 is implemented, as shown in Figure 2, it includes:

[0086] Step 201, based on the action Q factor and constraint function Q H Factor constraints are used to establish the probability function for selecting the true optimal action and the constraints on the action sampling amount.

[0087] Specifically, the probability function for selecting the true optimal action and the constraint on the action sampling amount are expressed as follows:

[0088] MaxN1,...N|A| PCS, st N1+N2+...+N |A| = N (10)

[0089] Where, N i Let |A| be the amount of sampled data obtained for action i, |A| be the size of the discrete action space A, and N represent the total amount of sampled data in each state during online reinforcement learning. PCS (Probability of Correct Selection) is defined as follows:

[0090] PCS means the probability that the observed optimal action is indeed the true optimal action. For example, at a certain moment, the solar and wind power generation capacity of a multi-energy microgrid is high, while the building's electricity demand and the total charging demand of electric vehicles are relatively low. In this case, the true optimal action might be to charge the energy storage device and charge each charging pile. If the optimal action estimated based on simulation observation data is also like this, then the true optimal action under the current state has been correctly selected.

[0091] PCS characterizes the probability of this random event occurring.

[0092] Step 202: Transform the true optimal action selection probability function to obtain the approximate optimal action selection probability function.

[0093] The objective function of the approximate optimal action selection probability function and the constraint condition for the action sampling amount are expressed by the following formula:

[0094] Where APCS is the approximate optimal action selection probability function; N i Q is the number of samples for action i; N is the total number of samples; k+1 (s, a) h ) represents the action that can be taken in state s. b The Q-factor of the next iteration; Q k+1 (s, a) i ) represents the action that can be taken in state s. i The Q-factor of the next iteration; Indicates that no action can be taken under state s. i The constraint function Q for the next iteration H factor; The boundary conditions represent the constraint functions; D is π. k (s), representing the steady-state probability of state s in the k-th iteration; H0 is a preset value; Indicates taking any action a in state s. b The constraint function Q for the next iteration H factor.

[0095] The implementation process of step 104 above includes:

[0096] When the total number of samplings is set to N, the objective function of the approximate optimal action selection probability function and the action sampling amount constraint are expressed by formula (12) as follows:

[0097] According to formulas (13) and (14), the Lagrange function is defined as follows:

[0098] Where λ is a Lagrange multiplier.

[0099] For an observed action i, its sampling quantity N is... i Treating it as a variable, the derivative of formula (15) yields:

[0100] in:

[0101] Therefore, we can obtain:

[0102] For an observation that cannot be acted upon, its sampling quantity N is [number]. i If we consider them as variables, then based on formula (15), we can obtain:

[0103] in,

[0104] Therefore, we can obtain:

[0105] For observing the optimal action Introduce two sampling variables N b1 and N b2 To prove its feasibility and optimality, we can obtain the following based on formula (15):

[0106] Transforming formula (18) yields the following formula:

[0107] Substituting formula (23) into formula (22), we get:

[0108] Therefore, we can obtain:

[0109] Next, we will examine the sampling number N for possible actions i and j. i With N j The relationship between them can be obtained from formula (18):

[0110] Assume N b1 N i and N j To be good, that is, N b1 >>N i N b1 >>N j Then we can get:

[0111] Therefore, formula (25) can be rewritten as:

[0112] Further results were obtained:

[0113] Taking the natural logarithm of both sides of the above formula yields:

[0114] Let N i =α i N,N j =α j N, then ∑ i α i =1, and as N→∞, the logarithmic term is significantly smaller than the other terms, so we can obtain:

[0115] Similarly, for the observations of inactive actions i and j, we can obtain a conclusion similar to that of formula (26):

[0116] Similarly, for the observed optimal action b2 and the observed inoperable action i, we can obtain a conclusion similar to that of formula (26):

[0117] In summary, based on the above analysis, the criteria for the sampling data allocation algorithm can be derived as follows:

[0118] Criterion 1: When i and j represent observed actionable actions,

[0119] Where, N i and N j σ represents the sampling amount of the observed actions i and j. i and σ j δ is the sample standard deviation of the Q factor for actions i and j. b，i =Q k+1 (s, a) b )-Q k+1 (s, a) i ), δ b，j =Q k+1 (s, a) b )-Q k+1 (s, a) j );

[0120] b represents the optimal action;

[0121] Criterion 2: When i and j represent observational actions that cannot be performed,

[0122] Where, N i and Nj σ represents the sampling quantity of the observations of the immobile actions i and j. h，i and σ h，i The constraint function Q represents the observation of the inability to perform actions i and j. H The sample variance of the factor;

[0123] Criterion 3: When i represents an observed action,

[0124] Where, N b N represents the number of samples for the optimal action. i σ represents the sampling quantity of the observed action i. b σ represents the sample standard deviation of the Q-factor of the optimal action b. i The sample standard deviation of the Q factor representing the action i. Indicates the optimal action to observe;

[0125] Criterion 4: When i represents an unobservable, non-action,

[0126] Where, N b N represents the number of samples for the optimal action. i σ represents the sampling quantity for the observation of the immovable action i. h，b The constraint function Q represents the optimal action b. H The sample variance of the factor, σ h，i To observe the inaction of i, construct a constraint function Q. H The sample variance of the factor;

[0127] k represents the number of iterations, Q k+1 (s, a) represents the Q-factor value of taking action a in state s during the k-th iteration. Q represents the constraint function Q that determines the action a to be taken in state s during the k-th iteration. H Factor value, H0 represents the preset value, D is π k (s) represents the steady-state probability of state s in the k-th iteration.

[0128] In detail, when the single letter 'x' represents an action, such as 'i', 'j', or 'b', it is related to 'a'. x The meaning is the same, that is, a x It also indicates an action.

[0129] Criterion 1 states that the sampling size for feasible actions i and j should be proportional to the square of their respective "noise-to-signal ratios." Here, noise figuratively describes the sample standard deviation of a single observation of the action's Q-factor, while signal figuratively describes the difference in Q-factor performance between the action and the optimal action. The greater the observational "noise" of the action's performance, the larger the sampling size should be for that action; when the action's performance is significantly worse than the currently observed optimal action, it should receive fewer samples.

[0130] Criterion 2 states that the sampling rate for infeasible actions i and j should be proportional to the square of their respective "constraint noise-to-signal ratio". Here, "constraint noise" figuratively describes the constraint function Q of the action. H The standard deviation of a single observation of a factor is a signal that visually describes the constraint function Q between the action and the optimal action. H Differences in factor performance. Similar to criterion 1, criterion 2 allocates the sampling size among actions where observation is not feasible in a manner proportional to the square of the "constraint-to-noise ratio".

[0131] Guideline 3 states that the best-performing action should be sampled more extensively to ensure its performance is indeed optimal. This guideline, along with Guideline 1, defines the sampling amount among possible actions.

[0132] Criterion 4 states that the sample size between observing infeasible actions and observing the best action should be proportional to the square of their "constraint-to-noise ratio". This criterion, together with Criterion 2, defines the sample size between observing infeasible actions and establishes a proportional relationship between the sample size of observing infeasible actions and observing the best action.

[0133] The above four criteria calculate the proportional relationship of the sampling amounts that all actions should obtain under the current state, based on the current samples. In multi-energy microgrid operation and scheduling applications, sampling computing resources are allocated iteratively.

[0134] In one embodiment of the present invention, based on the established sampling data allocation algorithm, online constraint reinforcement learning can be performed, specifically as shown in Figure 3, including:

[0135] Step 301: Simulate each action and sample the simulation results of each action according to the preset sampling amount. Determine the optimal action, feasible action and infeasible action based on the sampled simulation results.

[0136] The actions in this step can be determined based on the action space. For example, in a scenario of charging ten cars, each car has two actions: charging and not charging. Therefore, there are 1024 possible actions when the ten cars are combined.

[0137] The preset sampling quantity (N0) can be set based on the preset total sampling quantity N, the preset number of loops, the motion space, and the newly added sampling quantity △. Generally, the preset number of loops is greater than 10, and the preset total sampling quantity N is much larger than the preset sampling quantity N0 multiplied by the number of motions in the motion space.

[0138] In this step, the actions can be directly applied to the simulation model of the system under control (e.g., a digital twin model), and the simulation results of each action can be determined through the simulation model. In practice, the execution results of a preset sample size can also be obtained from historical data based on the state and actions, and the obtained execution results can be used as the simulation results.

[0139] Step 302: Calculate the Q-factor and sample variance of each feasible action and optimal action based on the simulation results; calculate the constraint function Q of each infeasible action based on the simulation results. H Factors and sample variance; record the sample size consumed and the total sample size consumed for each action.

[0140] In detail, the sample standard deviation of the Q-factor for feasible / optimal actions in this step is calculated based on the Q-factor of each sampling number for feasible / optimal actions. The constraint function Q for infeasible actions... H The sample variance of the factor is constrained by the constraint function Q for each sampling number based on the inability to perform the sampling. H The variance was calculated using factors. For specific calculation methods, please refer to existing variance calculation formulas.

[0141] In this step, the total number of samples consumed is the product of the preset sampling amount and the action space, plus the newly added sampling amount △.

[0142] Step 303: Using the sampling data allocation algorithm, assign the Q-factor and sample variance of each possible action and the optimal action, and the constraint function Q of each incapable action. H Factors, sample variance, and total sample size consumed are analyzed to obtain the target sample size for each action.

[0143] The sample size of each action target obtained in this step can make the near-optimal action selection probability move in the direction of maximization.

[0144] Step 304: Determine the supplementary sampling amount for each action based on the target sample size and the consumed sample size for each action.

[0145] During this step, the supplementary sampling quantity for each action is obtained by subtracting the sample quantity consumed by each action from the target sample quantity for each action.

[0146] Step 305: Perform supplementary sampling based on the supplementary sampling amount for each action, and use the simulation results of the supplementary sampling to redetermine the Q-factor and sample variance of the feasible actions and the optimal action, as well as the constraint function Q of each infeasible action. HFactors and sample variance.

[0147] Step 306: Adjust the total amount of samples consumed. Determine whether the total amount of samples consumed is less than the preset total amount of samples. If yes, return to re-execute step 303. If no, output the final optimal action to the system under control.

[0148] During this step, the consumed sample quantity is increased by the newly added sample quantity Δ to obtain the adjusted total consumed sample quantity. The constraint condition for the action sample quantity is the preset total sample quantity N, which can be set jointly according to the preset initial value of the sample quantity in step 301 and the action space, as long as it can satisfy the preset number of repeated cycles from step 301 to step 306.

[0149] In this step, if the total number of samples consumed is less than or equal to the action sampling amount constraint, it means that the number of samples consumed after the next iteration has not reached or is exactly equal to the preset total number of samples, and the feasibility of the action can continue to be observed. If the total number of samples consumed is greater than the action sampling amount constraint, it means that the number of samples consumed after the next iteration has exceeded the preset total number of samples, and the feasibility of the action cannot continue to be observed.

[0150] In one specific implementation, the algorithm flow of this embodiment is shown in Figure 7. In Figure 7, n0 is the preset sampling amount, and a is the action. For the action space, These are the sample standard deviation of the Q-factor for each action and the constraint function Q. H The sample variance of the factor Δ represents the number of samples already consumed, △ represents the number of new samples, N represents the preset total number of samples, and α represents the number of samples already consumed. i The ratio of the sample size that should be obtained for action i. N represents the number of samples that action i should obtain. i N represents the number of samples consumed for action i. i 'N' represents the target sample size for action i. i '-N i Supplement the sampling amount for action i.

[0151] This embodiment predetermines the sampling data allocation algorithm based on the objective function and constraint function of the system under control. This algorithm asymptotically maximizes the probability of correctly selecting the optimal action in each state. Compared to existing technologies, it considers the entire set of feasible strategies and ensures the estimability of the optimal action selection. It concentrates simulation resources on actions that truly need to be distinguished, making the observation results for each action increasingly accurate, ultimately leading to the optimal action and improving the performance of the final action determination. Furthermore, by controlling the sampling amount, this embodiment also improves the data efficiency of online reinforcement learning during the scheduling of the system under control, significantly reducing the time required for online decision-making and decreasing data usage.

[0152] In one embodiment of the present invention, as shown in FIG4, step 303 above assigns the Q factor and sample variance of each possible action and the optimal action, and the constraint function Q of each incapable action, according to the sampling data allocation algorithm. H Factor analysis, sample variance, and total consumed sample size were used to obtain the target sample size for each action, including:

[0153] Step 401: Based on the sampling data allocation algorithm and action sampling constraints, determine the Q-factor and sample variance of each possible action and the optimal action, and the constraint function Q of each incapable action. H Factors and sample variances are used to calculate the proportion of target samples for each action.

[0154] Step 402: The number of samples that should be obtained for each action is obtained by multiplying the target sample quantity ratio of each action by the total number of samples consumed.

[0155] Step 403: Select the larger of the number of samples that should be obtained for each action and the number of samples that have actually been used for each action as the target number of samples for each action.

[0156] Based on the same inventive concept, this invention also provides an online constraint reinforcement learning device for single-constraint controlled systems, as described in the following embodiments. Since the principle of the online constraint reinforcement learning device for single-constraint controlled systems is similar to that of the online constraint reinforcement learning method for single-constraint controlled systems, the implementation of the online constraint reinforcement learning device for single-constraint controlled systems can refer to the online constraint reinforcement learning method for single-constraint controlled systems; repeated details will not be elaborated further.

[0157] Specifically, as shown in Figure 5, the online constraint reinforcement learning device for single-constraint controlled systems includes:

[0158] The simulation unit 501 is used to simulate each action and sample the simulation results of each action according to a preset sampling amount. Based on the sampled simulation results, the optimal action, the actionable action, and the actionless action are determined.

[0159] The first calculation unit 502 is used to calculate the Q-factor and sample variance of each feasible action and optimal action based on the simulation results of each feasible action and optimal action; and to calculate the constraint function Q of each infeasible action based on the simulation results of each infeasible action. H Factors and sample variance; record the sample size consumed and the total sample size consumed for each action.

[0160] The second calculation unit 503 is used to apply the sampling data allocation algorithm to the constraints of the action sampling quantity, the Q factor and sample variance of each action and the optimal action, and the constraint function Q of each inactive action. H Factors and sample variance, total number of samples consumed The analysis yields the target sample size for each action; the sampling data allocation algorithm is determined based on the objective function and constraint function of the system under control, and is used to asymptotically maximize the probability of correctly selecting the optimal action in each state.

[0161] The third calculation unit 504 is used to determine the supplementary sampling amount for each action based on the target sample amount for each action and the sample amount consumed for each action.

[0162] The supplementary sampling unit 505 is used to perform supplementary sampling based on the supplementary sampling amount of each action, and to redetermine the Q-factor and sample variance of the feasible actions and the optimal action, as well as the constraint function Q of each infeasible action, using the simulation results of the supplementary sampling. H Factors and sample variance.

[0163] Control unit 506 is used to adjust the total amount of samples consumed, determine whether the total amount of samples consumed is less than the preset total amount of samples, if so, recalculate the target amount of samples and subsequent steps, if not, output the final optimal action to the system under control.

[0164] This embodiment predetermines the sampling data allocation algorithm based on the objective function and constraint function of the system under control. This algorithm asymptotically maximizes the probability of correctly selecting the optimal action in each state. Compared to existing technologies, it considers the entire set of feasible strategies and ensures the estimability of the optimal action selection. It concentrates simulation resources on actions that truly need to be distinguished, making the observation results for each action increasingly accurate, ultimately leading to the optimal action and improving the performance of the final action determination. Furthermore, by controlling the sampling amount, this embodiment also improves the data efficiency of online reinforcement learning during the scheduling of the system under control, significantly reducing the time required for online decision-making and decreasing data usage.

[0165] In one embodiment of the present invention, a computer device 602 is also provided, as shown in FIG6. The computer device 602 may include one or more processors 604, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 602 may also include any memory 606 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 606 may include any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 602. In one case, when the processor 604 executes associated instructions stored in any memory or combination of memories, the computer device 602 may perform any operation of the associated instructions. The computer device 602 also includes one or more drive mechanisms 608 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0166] Computer device 602 may also include an input / output module 610 (I / O) for receiving various inputs (via input device 612) and providing various outputs (via output device 614). A specific output mechanism may include a presentation device 616 and an associated graphical user interface 618 (GUI). In other embodiments, the input / output module 610 (I / O), input device 612, and output device 614 may be omitted, and the device may function solely as a computer device within a network. Computer device 602 may also include one or more network interfaces 620 for exchanging data with other devices via one or more communication links 622. One or more communication buses 624 couple the components described above together.

[0167] Communication link 622 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 622 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0168] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the methods described in any of the foregoing embodiments.

[0169] This invention also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the method described in any of the foregoing embodiments.

[0170] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0171] It should also be understood that, in the embodiments of the present invention, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the present invention, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0172] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or may be electrical, mechanical, or other forms of connection.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0176] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. An online constraint reinforcement learning method for single-constraint controlled systems, characterized in that, The method comprises: simulating each action, sampling the simulation results of each action according to a preset sampling amount, and determining the observed optimal action, actionable action and inactionable action according to the sampled simulation results; According to the simulation results of each actionable action and the optimal action, the Q factor and the sample variance of each actionable action and the optimal action are calculated; according to the simulation results of each non-actionable action, the constraint function Q of each non-actionable action is calculated H factor and the sample variance; the consumed sample amount of each action and the total consumed sample amount are recorded; The Q factor and sample variance of each actionable and optimal action, and the constraint function Q of each non-actionable are analyzed by using a sampling data distribution algorithm H The target sample amount of each action is obtained by analyzing the Q factor and sample variance of each actionable and optimal action, and the constraint function Q of each non-actionable, and the total amount of consumed samples; the sampling data distribution algorithm is determined according to the target function and constraint function of the system to be controlled, and is used to gradually maximize the probability of correctly selecting the optimal actionable in each state. determining the supplemental sampling amount of each action according to the target sample amount of each action and the consumed sample amount of each action; According to each action supplement sampling quantity, supplementary sampling is carried out, and the Q factor and sample variance of the executable action and optimal action and the constraint function Q of each non-executable action are re-determined by using the simulation result of supplementary sampling H factor and sample variance; adjusting the total consumed sample amount, determining whether the total consumed sample amount is less than a preset total sampling amount, if yes, recalculating the target sample amount and the subsequent steps, and if no, outputting the final optimal action to the controlled system.

2. The method of claim 1, wherein, The sampling data allocation algorithm determination process comprises: establishing the constraint conditions of the objective function and constraint function of the control strategy according to the device information and load information, system state, control action, optimization target information and constraint requirement information of the controlled system; The target function of the control strategy is converted to obtain an action Q factor of reinforcement learning, and the constraint condition of the constraint function is converted to obtain a constraint function Q of reinforcement learning H factor of the constraint condition; According to the action Q factor and the constraint function Q H The action sampling quantity constraint condition is established according to the action Q factor and the constraint function Q determining the criteria of the sampling data allocation algorithm according to the objective function and action sampling amount constraint conditions of the approximate optimal action selection probability function.

3. The method of claim 2, wherein, The constraint condition of the constraint function is converted to obtain a constraint function Q of reinforcement learning H The constraint condition of the factor includes obtaining a constraint function Q expressed by the following formula H The constraint condition of the factor: where Q H k+1 V (s, a) is the value function defined for the constraint function in the k+1 iteration; a is the action; D is π k (s), which represents the steady-state probability of state s in the k iteration; H0 is a preset value; π k (s') represents the steady-state probability of state s' in the kth iteration, represents taking action d in state s' in the kth iteration k constraint function Q of state s' H factor value.

4. The method of claim 2, wherein, According to the action Q factor and the constraint function Q H The action sampling quantity constraint condition is established according to the action Q factor and the constraint function Q According to the action Q factor and the constraint function Q H The real optimal action selection probability function and the action sampling quantity constraint condition are established according to the action Q factor and the constraint function Q The approximate optimal action selection probability function is obtained by converting the real optimal action selection probability function.

5. The method of claim 4, wherein, The objective function and action sampling amount constraint conditions of the approximate optimal action selection probability function are represented by the following formula: max APCS s.t.∑ i N i = N where APCS is the approximate optimal action selection probability function; N i is the number of samples for action i; N is the total number of samples; Q k+1 (s, a) represents the Q-factor for the next iteration of the step where the action a h is taken in state s; Q b (s, a) represents the Q-factor for the next iteration of the step where the action a k+1 is taken in state s; Q i (s, a) represents the Q-factor for the next iteration of the step where the action a i is taken in state s; and representing that the inaction a is taken at state s i constraint function Q for the next iteration of steps H factors; boundary conditions of the constraint function; D is π k (s), representing the steady-state probability of state s in the kth iteration; H0is a preset value; π k (s') represents the steady-state probability of state s' in the kth iteration, represents taking action d in state s' in the kth iteration k constraint function Q of state s' H factor value; represents taking any action a in state s b the constraint function Q for the next iteration of steps H factors.

6. The method of claim 1, wherein, The sampling data allocation algorithm comprises the criteria satisfying the following conditions: the greater the first noise-to-signal ratio of the actionable action, the greater the actionable action sampling amount, wherein the first noise-to-signal ratio is the ratio of the sample standard deviation of the action Q factor to the difference between the optimal action Q factor and the action Q factor; The greater the second noise-to-signal ratio, the greater the number of non-actuable samples, wherein the second noise-to-signal ratio is the action constraint function Q H The sample variance of the factor and the optimal action constraint function Q H The factor and the action constraint function Q H The ratio of the difference in the factors; the sampling amount of the optimal action is greater than the actionable action sampling amount; the sampling amount of the optimal action and the inactionable action is proportional to the square of the second noise-to-signal ratio of the optimal action and the inactionable action.

7. The method of claim 6, wherein, The sampling data allocation algorithm comprises the following criteria: Criterion 1 : when i and j represent observed actionable, where N i and N j denote the number of samples observing actions i and j, σ i and σ j denote the sample standard deviation of the Q factors for actions i and j, δ b，i = Q k+1 (s, a b ) - Q k+1 (s, a i ), and δ b，j = Q k+1 (s, a b ) - Q k+1 (s, a j ). Criterion 2: when i and j represent observations for which no action is possible, where N i and N j denote the number of samples observing the inactionable acts i and j, σ h，i and σ h，i denote the constraint functions Q H of the sample variances of the factors; Criterion 3: When i represents an observed actionable, where N b represents the sample size of the optimal action, N i represents the sample size of the observed action i, σ b represents the sample standard deviation of the Q-factor of the optimal action b, σ i represents the sample standard deviation of the Q-factor of the action i, indicating the observed optimal action; Criterion 4: when i represents no action is observable, where N b represents the number of samples of the optimal action, N i represents the number of samples of the observed inaction i, σ h，b represents the constraint function Q H of the sample variance of the factor, σ h，i is the constraint function Q H of the sample variance of the factor; k denotes the iteration round number, Q k+1 (s, a) denotes the Q-factor value for taking action a in state s in the kth iteration round, Qk(s, a) represents a constraint function for taking action a in state s in the kth iteration H factor value, H0 represents a preset value, D is π k (s), denotes the steady state probability of state s in the kth iteration of the loop, π k (s') denotes the steady state probability of state s' in the kth iteration of the loop, represents taking action d in state s' in the kth iteration k constraint function Q of state s' H factor value.

8. The method of claim 7, wherein, The Q factor and sample variance of each actionable and optimal action, and the constraint function Q of each non-actionable are analyzed by using a sampling data distribution algorithm H The target sample amount of each action is obtained by analyzing the Q factor and sample variance of each actionable and optimal action, and the constraint function Q of each non-actionable, and the total amount of consumed samples, including: Based on the sampling data allocation algorithm, the Q-factor and sample variance of each observation's possible actions and optimal actions, and the constraint function Q of each observation's incapable actions are determined. H Factor and sample variance analysis were performed to calculate the proportion of target samples for each action. obtaining the sample amount that each action should obtain by multiplying the target sample amount proportion of each action by the total consumed sample amount; selecting the greater of the sample amount that each action should obtain and the actual used sample amount of each action as the target sample amount of each action.

9. An online constraint reinforcement learning device for a single-constrained controlled system, characterized in that, The device comprises: a simulation unit configured to simulate each action, sample the simulation results of each action according to a preset sampling amount, and determine the observed optimal action, actionable action and inactionable action according to the sampled simulation results; The first calculation unit is configured to calculate the Q factor and sample variance of each actionable action and the optimal action according to the simulation results of each actionable action and the optimal action, and calculate the constraint function Q of each non-actionable action according to the simulation results of each non-actionable action. H The first calculation unit is configured to calculate the Q factor and sample variance of each actionable action and the optimal action according to the simulation results of each actionable action and the optimal action, and calculate the constraint function Q of each non-actionable action according to the simulation results of each non-actionable action. a second calculation unit configured to analyze the Q factor and sample variance of each actionable action and optimal action, the constraint function Q H factor and sample variance, and the total amount of consumed samples to obtain a target sample amount of each action; the sampling data allocation algorithm is determined according to a target function and a constraint function of the system to be controlled, and is used to progressively maximize the probability of correctly selecting the optimal actionable action in each state. a third calculation unit configured to determine the supplemental sampling amount of each action according to the target sample amount of each action and the consumed sample amount of each action; The supplementary sampling unit is used to perform supplementary sampling based on the supplementary sampling amount of each action, and to redetermine the Q-factor and sample variance of the feasible actions and the optimal action, as well as the constraint function Q of each infeasible action, using the simulation results of the supplementary sampling. H Factors and sample variance; a control unit configured to adjust the total consumed sample amount, determine whether the total consumed sample amount is less than a preset total sampling amount, if yes, recalculate the target sample amount and the subsequent steps, and if no, output the final optimal action to the controlled system.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 8 when executing the computer program.

Images