High-efficiency reinforcement learning method and apparatus for operation scheduling of controlled system

Abstract

A high-efficiency reinforcement learning method and apparatus for the operation scheduling of a controlled system. The method comprises: sampling state-action pairs in a controlled system, and determining, on the basis of sampling results, the action type of each action in each state, a Q-factor sample variance and the total number of consumed samples (S201); using a sampling data allocation algorithm to determine a target sample size of each action in each state (S202); on the basis of the target sample size of each action and the number of consumed samples of each action, determining a supplementary sampling quantity for each action (S203); performing supplementary sampling on the basis of the supplementary sampling quantity for each action, and using simulation results of supplementary sampling to re-determine the action type of each action and the Q-factor sample variance of each state-action pair (S204); and adjusting the total number of consumed samples, determining whether the total number of consumed samples meets an action sampling quantity constraint condition, and if so, outputting an optimal action observed in each state (S205). The high-efficiency reinforcement learning method can distinguish the importance of state-action pairs, thereby improving the sampling efficiency and optimizing the policy performance.

Description

A High-Efficiency Reinforcement Learning Method and Device for Controlled System Operation Scheduling Technical Field

[0001] This invention relates to the field of reinforcement learning, and in particular to a highly efficient reinforcement learning method and apparatus for scheduling the operation of a controlled system. Background Technology

[0002] For complex controlled systems (such as smart grids), their economic operation has significant engineering requirements. Reinforcement learning, with its potential to learn from data and gradually improve policy performance, has thus gained considerable attention in engineering applications. Taking smart grids as an example, the state space (including the generation side, demand side, and energy storage side) and action space (such as generation scheduling, charging power of electric vehicle charging piles, and charging or discharging power of energy storage stations) in real-world problems are generally very large. Existing reinforcement learning methods for the operation and scheduling of complex controlled systems have the following limitations:

[0003] (1) In online learning, given the current state, sampling efficiency can be improved by adjusting the number of samples allocated to different actions. However, the importance of decisions varies under different states. Online learning makes it difficult to systematically distinguish the differences in the importance of different decisions. Therefore, existing methods struggle to quickly obtain the optimal strategy when computational resources are limited.

[0004] (2) During the offline learning process, although different sampling numbers can be allocated to different (state, action) pairs, the impact of decision importance in different states on sampling allocation is not analyzed, resulting in problems such as long learning time, low efficiency and poor policy determination performance.

[0005] Summary of the Invention

[0006] This invention addresses the problem in the prior art that, during the scheduling of complex controlled systems, the impact of different state-action pairs on the allocation of sampling quantities is not distinguished, resulting in low learning efficiency and poor accuracy of optimal actions.

[0007] To address the aforementioned technical problems, the first aspect of this invention provides a highly efficient reinforcement learning method for scheduling the operation of a controlled system, comprising:

[0008] The simulation results of each state-action pair of the controlled system are sampled according to the preset sampling number. Based on the sampled simulation results, the action type of each action in each state, the Q factor sample variance of each state-action pair, and the total number of samples consumed are determined.

[0009] The sampling data allocation algorithm is used to analyze the Q-factor sample variance and the total number of consumed samples of each state-action pair to obtain the target sample size of each action in each state. The sampling data allocation algorithm includes a sampling quantity determination criterion that distinguishes the importance of each type of action in each state, which is used to asymptotically maximize the probability of correctly selecting the optimal action.

[0010] The number of additional samples for each action is determined based on the target sample size and the sample size already consumed for each action.

[0011] Supplementary sampling is performed based on the number of supplementary samples for each action, and the action type and Q-factor sample variance of each state-action pair are re-determined using the simulation results of the supplementary sampling.

[0012] Adjust the total amount of samples consumed, and determine whether the total amount of samples consumed is less than the preset total amount of samples. If so, recalculate the target amount of samples and subsequent steps. If not, output the optimal action observed in each state.

[0013] A second aspect of the present invention provides a high-efficiency reinforcement learning device for scheduling the operation of a controlled system, comprising:

[0014] The simulation calculation unit is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the action type of each action in each state, the Q-factor sample variance of each state-action pair, and the total number of samples consumed based on the sampled simulation results.

[0015] The target sampling quantity determination unit is used to analyze the Q-factor sample variance and the total amount of consumed samples of each state-action pair using the sampling data allocation algorithm to obtain the target sample quantity of each action in each state. The sampling data allocation algorithm includes a sampling quantity determination criterion that distinguishes the importance of each type of action in each state, which is used to progressively maximize the probability of correctly selecting the optimal action.

[0016] The supplementary sampling quantity determination unit is used to determine the supplementary sampling quantity for each action based on the target sample quantity for each action and the sample quantity consumed by each action.

[0017] The supplementary sampling unit is used to perform supplementary sampling based on the number of supplementary samples for each action, and to redetermine the action type of each action and the Q-factor sample variance of each state-action pair using the simulation results of the supplementary sampling.

[0018] The analysis unit is used to adjust the total amount of samples consumed, determine whether the total amount of samples consumed is less than the preset total amount of samples, and if so, restart the target amount of samples determination unit; otherwise, output the optimal action observed in each state.

[0019] The present invention provides a high-efficiency reinforcement learning method and apparatus for the operation and scheduling of controlled systems. This method is applicable to complex controlled systems, such as smart grids, involving multiple states and actions. Based on the Q-function derived from the objective function transformation of the controlled system, a sampling quantity determination criterion is obtained that can distinguish the importance of different types of actions in each state. This criterion asymptotically maximizes the probability of correctly selecting the optimal action, concentrating simulation resources on the truly important actions that need to be distinguished. This achieves effective sampling, reduces learning time, lowers time costs, and makes the observation results of each action increasingly accurate, ultimately yielding the optimal action. Compared to existing technologies, this method considers the entire set of feasible strategies, can distinguish the importance differences between different state-action pairs, and ensures the estimability of the optimal action selection, thus improving the performance of the final action determination. Furthermore, by controlling the sampling quantity, the present invention can also improve the data efficiency of online reinforcement learning during the operation and scheduling of the controlled system, significantly reducing the time required for online decision-making and decreasing data usage.

[0020] To make the above and other objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 shows a flowchart of the sampling data allocation algorithm determination process according to an embodiment of the present invention;

[0023] Figure 2 shows a flowchart of a high-efficiency reinforcement learning method for controlling system operation scheduling according to an embodiment of the present invention;

[0024] Figure 3 shows a flowchart of the process for determining the sampling amount of each action target under each state in an embodiment of the present invention;

[0025] Figure 4 shows a structural diagram of the high-efficiency reinforcement learning device for controlling system operation scheduling according to an embodiment of the present invention;

[0026] Figure 5 shows a structural diagram of a computer device according to an embodiment of the present invention;

[0027] Figure 6 shows a flowchart of the high-efficiency reinforcement learning method according to an embodiment of the present invention.

[0028] Explanation of symbols in the attached figures: 401, Simulation calculation unit; 402, Target sampling quantity determination unit; 403, Supplementary sampling quantity determination unit; 404, Supplementary sampling unit; 405, Analysis unit; 502, Computer equipment; 504, Processor; 506, Memory; 508, Drive mechanism; 510, Input / output module; 512, Input device; 514, Output device; 516, Presentation device; 518, Graphical user interface; 520, Network interface; 522, Communication link; 524, Communication bus. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0031] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0032] It should be noted that the data involved in this invention (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0033] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of the present invention. However, they do not mean that the applicant has used or necessarily used the solution.

[0034] The controlled system described in this invention is a system with multiple states and multiple actions, such as a smart grid system. The corresponding state space includes, for example, the states of the power generation side, demand side, and energy storage side of the grid. The corresponding action space includes, for example, power generation scheduling, the charging power of electric vehicle charging piles, and the charging or discharging power of energy storage stations. In specific implementations, other multi-state and multi-action systems may also be used; this invention does not specifically limit the controlled system.

[0035] Before implementing the high-efficiency reinforcement learning method for controlling system operation scheduling described in this invention, a sampling data allocation algorithm needs to be established first to indicate:

[0036] (1) The higher the signal-to-noise ratio of the non-observed optimal action under each state s, the greater the number of samples of the non-observed optimal action.

[0037] In the noise-information ratio, the noise refers to the sample standard deviation of the Q factor of the non-observed optimal action under each state s, and the information refers to the performance difference between the Q factor of the non-observed optimal action under each state s and the Q factor of the observed optimal action under each state s.

[0038] (2) The number of samples of the best observed action in each state s should be greater than the number of samples of other non-best observed actions.

[0039] (3) The number of samples of the optimal action observed in each state should be proportional to the product of the steady-state probability of each state and the sample standard deviation of the Q factor of each state.

[0040] Specifically, as shown in Figure 1, the process of determining the sampling data allocation algorithm includes:

[0041] Step 101: Based on the equipment information and load information of the controlled system, system status, control actions, and optimization target information, establish the objective function of the control strategy.

[0042] Step 102: Transform the objective function of the control strategy to obtain the action Q function for reinforcement learning.

[0043] Step 103: Based on the action Q function, establish the objective function of the approximate optimal action selection probability function and the constraint condition of the number of action samples.

[0044] The objective function of this near-optimal action selection probability function contains data items that distinguish the importance of each type of action in each state.

[0045] Step 104: Determine the criteria for the sampling data allocation algorithm based on the objective function of the near-optimal action selection probability function and the constraint on the number of action samples.

[0046] In detail, the equipment information, load information, system status, control actions, and optimization target information in step 101 can be determined according to the specific situation of the controlled system, and this invention does not limit the specific parameters included therein. An objective function is established based on the equipment information, load information, system status, control actions, and optimization target.

[0047] Taking a smart grid as an example, consider a grid with a large number of new energy sources (such as solar and wind power) connected to it. The randomness and volatility of new energy power generation require coordinated control of the grid's supply and demand sides. Optimization control on only one side may not guarantee the stable operation of the entire grid system. This grid contains various devices (such as electrical appliances, energy storage devices, and power generation devices), resulting in a very large state space. The goal is to use offline learning to find relatively important (state, action) pairs from a large number of (state, action) pairs, and to appropriately allocate sampling resources according to the importance of (state, action), thereby enabling the best possible operation scheduling strategy to be found within a limited time.

[0048] In some implementations, the established control objective can be expressed using the following formula:

[0049] Among them, s t The system state vector at time t represents the system state vector. Taking a smart grid where the controlled system is connected to a large number of new energy sources (such as solar energy, wind energy, etc.) as an example, the system state vector includes, for example, the power generation of each power generation device, the power demand of each power consumption device, and the energy storage level of each energy storage device.

[0050] d represents the operation scheduling strategy, which is the action to be taken based on the system state. For example, the scheduling strategy is the charging or discharging power of energy storage devices, the generating power of generator sets, the total charging power of electric vehicle charging stations, etc.

[0051] c t This represents the control objective of the controlled system at time t. For example, it can be a cost function. Taking the controlled system as a smart grid as an example, it can be the total power generation cost of the smart grid, which includes the relatively free or inexpensive power generation from new energy sources and the relatively expensive power generation costs from the main grid, minus the fees charged to users, etc.

[0052] T represents the total simulation duration;

[0053] J(d) represents the average operating cost of strategy d.

[0054] For formula (1), the control objective is to minimize J(d), and the content represented by formula (1) can be described as follows:

[0055] Where, N sa This represents the number of samples for the Q-factor of (s, a) pair (state, action); S represents the state space, s represents the state; A represents the action space, a represents the action; N represents the total number of samples, which is a preset value and can be set by the user according to actual needs; d b This indicates the target control strategy.

[0056] When implementing step 102, the action Q function is established according to the above formula (1):

[0057] Among them, Q d (s,a) represents the Q-function of action a in state s; d represents the given policy; for simplicity, the above Q-function is... d (s,a) can also be represented as Q(s,a);

[0058] c represents the target object value corresponding to state s and action a;

[0059] s' represents the future state; P(s′|s,a) represents the probability of taking action a from state s to transition to the next state s'.

[0060] Q d (s',d(s')) represents the Q-factor of state s' and its policy d'.

[0061] Furthermore, since P(s′|s,a) is usually not obtainable, the Q function of formula (3) can be observed using the following formula:

[0062] Among them, Q k+1 (s,a) represents Q d The estimated value of (s,a); Q k (s',d(s')) represents Q d (s',d(s')) estimates, where k is the number of iterations.

[0063] Step 103 above establishes the objective function of the approximate optimal action selection probability function and the action sampling quantity constraint based on the action Q function, including:

[0064] First, based on the action Q function, establish the true optimal action selection probability function and the action sampling quantity constraint; then, transform the true optimal action selection probability function to obtain the approximate optimal action selection probability function.

[0065] In one specific implementation, the probability function for selecting the true optimal action is expressed as follows:

[0066] Where PCS represents the probability function of the true optimal action selection, which is also known as the probability function of correct selection.

[0067] π(s) represents the steady-state probability distribution of state s, which can be estimated through simulation;

[0068] Q(s, a) b (s) is the action Q function, representing the state s and action a. b The initial objective function value, such as average cost;

[0069] ε is a constant representing the value that the objective function should not exceed, such as the value that the average operating cost of the power grid should not exceed.

[0070] This indicates the constraint on the number of action samples.

[0071] Compared with the objective function represented by formula (1), the true optimal action selection probability function maximizes the probability of finding the running strategy, transforming the performance of the originally difficult-to-evaluate optimal running strategy into a probability that is relatively easy to evaluate.

[0072] In one specific implementation, the reasoning process of (5) includes:

[0073] According to Bayesian theory, the obtained samples are considered to be N sa For a given and fixed condition, the true value Q(s,a) is considered as a random variable following a posterior distribution. In fact, if the prior distribution follows a Gaussian distribution and the observation noise also follows a Gaussian distribution, then the posterior distribution also follows a Gaussian distribution. In this embodiment, the observation noise refers to the mean of the observed samples. The difference between the actual sample Q(s,a) and the actual sample Q(s,a). According to the central limit theorem, assume the sample... If the sample follows an independent distribution, then the sample mean is... The difference between Q(s,a) and Q(s,a) follows a Gaussian distribution, which leads to the conclusion that:

[0074] Assumption: For all states and actions, the prior distribution of Q(s,a) follows a Gaussian distribution.

[0075] Under the above assumptions, the hindrance distribution of Q(s,a) follows a Gaussian distribution. yes variance.

[0076] Since the true Q(s,a) is a random variable, if the optimal action observed in each state is chosen, the performance of the policy can be expressed as:

[0077] in, If the distribution is steady-state, then the probability of making the correct choice is:

[0078] However, the optimal strategy The authenticity is difficult to assess. The replacement is based on the above formula. Maximizing the probability of the correct choice can yield the optimal strategy, which is the above (5).

[0079] In some implementations, the objective function of the approximate optimal action selection probability function and the constraint on the number of action samples are expressed by the following formula:

[0080] Max APCS;

[0081] Wherein, APCS represents the approximate optimal action selection probability function;

[0082] Indicates the optimal action observed in state s. The Q-factor value;

[0083] Q(s,a) represents the Q factor value of the unobserved optimal action a in state s;

[0084] π(s) represents the steady-state probability distribution of state s;

[0085] ε represents a constant;

[0086] N sa This represents the sample size of any action a under state s;

[0087] N represents the total number of samples.

[0088] In the near-optimal action selection probability function, This data item distinguishes the importance of different types of actions in each state. The meaning of the approximate optimal action selection probability function is that the observed optimal action in each state is indeed the true optimal action, and the objective function value of the observed optimal policy does not exceed ε.

[0089] Specifically, the reasoning process for deriving formula (6) from formula (5) includes:

[0090] Calculating PCS in closed-form terms is difficult. Therefore, we first give a lower bound for PCS, representing an approximate PCS (denoted as APCS). Specifically, based on the Bonferroni inequality and PCS, we can obtain the following inequality:

[0091] In formula (7), the right side is APCS.

[0092] To obtain a more accurate expression of APCS, we have:

[0093] in,

[0094] therefore, Follows Gaussian distribution in:

[0095] Then we can get:

[0096] in,

[0097] Furthermore, we can obtain:

[0098] Integrating formulas (7)-(9), we get:

[0099] The right side of the above equation is represented as APCS.

[0100] In step 104 above, in order to solve formula (10), based on the Lagrange concept and APCS, the following Lagrange function can be obtained:

[0101] Where λ is a Lagrange multiplier.

[0102] According to the Karush-Kuhn-Tucker (KKT) conditions, the necessary conditions for local optimal performance can be obtained as follows:

[0103] Based on this, we can conclude that:

[0104] First, for the unobserved optimal action a in state s, we can obtain:

[0105] in,

[0106] Combining formulas (11) to (13), we can obtain:

[0107] Secondly, the optimal action for observing state s. We can obtain:

[0108] Based on the above formula, APCS can be rewritten as:

[0109] APCS = A + B;

[0110] in,

[0111] Then we can get:

[0112] Combining formulas (15) and (16), we can obtain:

[0113] Furthermore, we can obtain

[0114] Next, we introduce N. sb1 and N sb2 Represent A and B respectively The value of N is determined, and the analysis concludes by letting N... sb1 =N sb2 We can obtain:

[0115] Next, we will analyze N. sa Regarding the relationship, firstly, assuming that non-optimal actions i and j are observed in state s, according to formula (14), we can obtain:

[0116] When the total number of samples N approaches infinity, most of the samples should be allocated to the action that is observed to be optimal in state s.

[0117] Assumption Then we get:

[0118] Further findings:

[0119] By rearranging the above equation, we can obtain:

[0120] Taking the logarithm of both sides, we get:

[0121] When N is large enough, the logarithmic term is significantly smaller than the other terms, and we can obtain:

[0122] For observing the optimal action Combining formulas (14) and (17), we can obtain:

[0123] According to formula (18), the optimal action is observed under different states for two actions. and We can obtain:

[0124] Further results were obtained:

[0125] Formulas (19), (20), and (21) can be used to construct the criteria for the sampling data allocation algorithm.

[0126] Compared to the objective function of the true optimal action selection probability function, the objective function of the approximate optimal action selection probability function replaces the difficult-to-calculate joint probability of the true optimal action probability function with the pairing probability between the observed optimal action and other actions in each state s, as well as the probability of the performance of the observed optimal action in all states. Therefore, the objective function of the approximate optimal action selection probability function is easier to compute, and can even be accelerated using parallel computing. Based on the objective function of the approximate optimal action selection probability function, and given a determined sampling data allocation algorithm, the asymptotically optimal solution can be easily computed.

[0127] In step 104 above, based on the objective function of the near-optimal action selection probability function and the constraint condition of the number of action samples, the criteria for determining the sampling data allocation algorithm are as follows:

[0128] Criterion 1: For unobserved optimal actions i and j,

[0129] Where, N si N represents the number of samples for action i in state s. sj σ represents the number of samples for action j in state s. si σ represents the sample standard deviation of the Q factor for action i in state s. sj δ represents the sample standard deviation of the Q-factor of action j in state s. si δ represents the difference in performance between action i in state s and the observed optimal action in state s. sj This represents the difference between the performance of action j in state s and the observed optimal action in state s; specifically, the difference can be represented by the corresponding Q-factor difference.

[0130] Criterion 2: Optimal action for observation in state s ab ,

[0131] in, Indicates action a in state s b Number of samples, Indicates action a in state s b Q-factor sample standard deviation, N sa σ represents the number of samples for the unobserved optimal action a in state s. sa This represents the sample standard deviation of the Q factor for the unobserved optimal action a in state s.

[0132] Criterion 3: The optimal action a for observation in state s1 b (s1) and the optimal action a observed in state s2 b (s2),

[0133] in, Describes the optimal action a under state s1. b The number of samples (s1), Describes the optimal action a under state s2. b The number of samples in (s2), where π(s1) represents the steady-state probability distribution of state s1, and π(s2) represents the steady-state probability distribution of state s2. Describes the optimal action a under state s1. b The sample standard deviation of the Q factor of (s1), Describes the optimal action a under state s2. b The sample standard deviation of the Q factor of (s2).

[0134] It must also meet criterion 4:

[0135] Criterion 1 states that, under state s, the number of samples for two unobserved optimal actions i and j should be proportional to the square of their "noise-to-information ratio" (NRI). In the NRI, noise refers to the sample standard deviation of the single-observation performance of the Q-factor of the (state, action) pair, while information refers to the NRI between the (state, action) pair and the observed optimal action a for that state. b (i.e., (s,a)) b The Q factor of )) has poor performance.

[0136] Criterion 2 states that the optimal action a is observed in state s. b (s) The sample size should be proportional to the sample standard deviation of the Q factor from a single observation, and the sample size should be explicitly given. It should be equal to the sample quantity (N) obtained by other actions in the same state. sa / σ sa ) 2 The sum of these. If the observed standard deviations of the Q factors for each action are comparable, then observing the optimal action will yield significantly more samples than observing other actions.

[0137] Criterion 3 states that the sample size of the Q-factor corresponding to the optimal action observed in each of two different states should be proportional to the product of the steady-state probability of each state and the standard deviation of its Q-factor samples. According to Criterion 3, the optimal action observed in states that frequently occur in steady state will receive a larger sample size than the optimal action observed in other states.

[0138] Guideline 4 states that after calculating the proportion of samples that the Q factor of each (state, action) pair should receive according to Guidelines 1-3, the total number of samples should be equal to the given total number of samples.

[0139] Based on the determined sampling data allocation algorithm, a highly efficient reinforcement learning method for scheduling the controlled system can be implemented. Specifically, as shown in Figure 2, the highly efficient reinforcement learning method for scheduling the controlled system includes:

[0140] Step 201: Sample the simulation results of each state-action pair of the controlled system according to the preset sampling quantity, and determine the action type of each action in each state, the Q-factor sample variance of each state-action pair, and the total number of samples consumed based on the sampled simulation results.

[0141] When this step is implemented, each action and each state-action pair in each state can be recorded as (state, action).

[0142] The preset sampling quantity (N0) can be set based on the preset total sampling quantity N, the preset number of loops, the motion space, and the newly added sampling quantity △. Generally, the preset number of loops is greater than 10, and the preset total sampling quantity N is much larger than the preset sampling quantity N0 multiplied by the number of motions in the motion space.

[0143] The actions in this step can be determined based on the action space. For example, in a scenario of charging ten cars, each car has two actions: charging and not charging. Therefore, there are 1024 possible actions when the ten cars are combined.

[0144] The state in this step can be determined based on the state space.

[0145] In this step, the (state, action) can be directly applied to the simulation model of the controlled system (e.g., a digital twin model), and the simulation results for each (state, action) are determined through the simulation model. In practice, the execution results of a preset sample size can also be obtained from historical data based on the state and action, and the obtained execution results are used as the simulation results.

[0146] The total number of samples consumed in this step is the product of the total number of states, the total number of actions, and the preset number of samples, plus the preset increase. This can be calculated using the following formula:

[0147] Where n0 represents the preset sampling amount; S represents the state quantity in the state space; A represents the action quantity in the action space; and Δ represents the preset increment. The preset increment is a preset value, which should be selected so that the number of iterations of the entire algorithm is neither too many nor too few, for example, at least 10 or 20 times.

[0148] Step 202: Analyze the Q-factor sample variance and the total number of consumed samples for each state-action pair using the sampling data allocation algorithm to obtain the target sample size for each action in each state.

[0149] The sampling data allocation algorithm includes a criterion for determining the number of samples to distinguish the importance of each type of action in each state. The criterion is used to progressively maximize the probability of correctly selecting the optimal action. The process of determining the criterion can be referred to the aforementioned embodiments, and will not be repeated here.

[0150] Step 203: Determine the number of additional samples for each action based on the target sample size and the sample size already consumed for each action.

[0151] Step 204: Perform supplementary sampling based on the number of supplementary samples for each action, and use the simulation results of the supplementary sampling to redetermine the action type of each action and the Q-factor sample variance of each state-action pair.

[0152] Step 205: Adjust the total amount of samples consumed. Determine whether the total amount of samples consumed is less than the preset total amount of samples. If yes, re-execute step 202 and subsequent steps. If no, output the optimal action observed in each state.

[0153] During this step, the consumed sample quantity is increased by the newly added sample quantity Δ to obtain the adjusted total consumed sample quantity. The action sample quantity constraint is the preset total sample quantity N, which can be set jointly according to the preset initial value of the sample quantity in step 201 and the action space, as long as it can satisfy the preset number of repetitions from step 201 to step 205.

[0154] In this step, if the total number of samples consumed is less than or equal to the action sampling amount constraint, it means that the number of samples consumed after the next iteration has not reached or is exactly equal to the preset total number of samples, and the feasibility of the action can continue to be observed. If the total number of samples consumed is greater than the action sampling amount constraint, it means that the number of samples consumed after the next iteration has exceeded the preset total number of samples, and the feasibility of the action cannot continue to be observed.

[0155] In one specific implementation, the algorithm flowchart of this embodiment is shown in Figure 6. In Figure 6, n0 is the preset sampling amount, and a represents the action. Represents the action space, σ sa 2 The sample standard deviation of the Q factor for state s-action a is represented by n0|S||A|, where n0|S||A| represents the actual number of samples consumed. This indicates the number of samples already consumed, △ indicates the number of new samples added, N indicates the preset total number of samples, and α indicates the number of samples already consumed. sa This indicates the ratio of the number of samples that should be obtained for state s to the number of samples that should be obtained for action a. N represents the number of samples that should be obtained for state s-action a. sa The number of samples consumed for state s-action i is N. sa 'N represents the target sample size for state s-action i.' sa '-N sa Supplementary sampling amount for state s-action a; This represents the optimal action for observing the final determined state s.

[0156] Once the optimal action observed in each state is obtained, the controlled system can be controlled based on the optimal action observed in each state.

[0157] This embodiment is applicable to complex controlled systems involving multiple states and actions, such as smart grids. Based on the Q-function derived from the objective function transformation of the controlled system, a sampling quantity determination criterion is obtained that can distinguish the importance of different types of actions in each state. This criterion asymptotically maximizes the probability of correctly selecting the optimal action, concentrating simulation resources on the truly important actions that need to be distinguished. This makes the observation results for each action increasingly accurate, ultimately yielding the optimal action. Compared to existing technologies that consider the entire set of feasible strategies, this invention can distinguish the importance differences between different state-action pairs and ensure the estimability of the optimal action selection, thus improving the performance of the final determined action. Furthermore, by controlling the sampling quantity, this invention can also improve the data efficiency of online reinforcement learning during the operation and scheduling of the controlled system, significantly reducing the time required for online decision-making and decreasing data usage.

[0158] In one embodiment of the present invention, as shown in Figure 3, step 202 above uses a sampling data allocation algorithm to analyze the Q-factor sample variance and the total amount of samples consumed for each state-action pair to obtain the target sample size for each action in each state, including:

[0159] Step 301: Analyze the Q-factor sample variance of each state-action pair according to the sampling data allocation algorithm to obtain the target sample quantity ratio of each action in each state.

[0160] Step 302: The number of samples that should be obtained for each action in each state is obtained by multiplying the target sample quantity ratio of each action in each state by the total number of samples consumed.

[0161] Step 303: Select the larger of the number of samples that should be obtained for each action in each state and the number of samples that have actually been used for each action in each state as the target number of samples for each action in each state.

[0162] In one embodiment of the present invention, step 203, which determines the supplementary sampling quantity for each action based on the target sample quantity and the consumed sample quantity for each action, includes: subtracting the consumed sample quantity for each action from the target sample quantity for each action to obtain the supplementary sampling quantity for each action.

[0163] In one embodiment of the present invention, the supplementary sampling process in step 204 based on the supplementary sampling quantity can refer to step 201, and its specific implementation process will not be repeated here.

[0164] Based on the same inventive concept, this invention also provides a high-efficiency reinforcement learning device for controlled system operation scheduling, as described in the following embodiments. Since the principle of the high-efficiency reinforcement learning device for controlled system operation scheduling is similar to that of the high-efficiency reinforcement learning method for controlled system operation scheduling, the implementation of the high-efficiency reinforcement learning device for controlled system operation scheduling can refer to the high-efficiency reinforcement learning method for controlled system operation scheduling; repeated details will not be elaborated further.

[0165] Specifically, as shown in Figure 4, the high-efficiency reinforcement learning device for scheduling the operation of the controlled system includes:

[0166] The simulation calculation unit 401 is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the action type of each action in each state, the Q-factor sample variance of each state-action pair, and the total number of samples consumed based on the sampled simulation results.

[0167] In detail, the implementation process of obtaining simulation results in the simulation calculation unit 401 can be found in the method embodiment, and will not be repeated here.

[0168] The target sampling quantity determination unit 402 is used to analyze the Q-factor sample variance and the total amount of consumed samples of each state-action pair using a sampling data allocation algorithm to obtain the target sample quantity of each action in each state. The sampling data allocation algorithm includes a sampling quantity determination criterion that distinguishes the importance of each type of action in each state. The criterion is obtained by analyzing the Q function transformed from the objective function of the controlled system and is used to asymptotically maximize the probability of correctly selecting the optimal action.

[0169] The specific criteria included in the sampling data allocation algorithm can be found in criteria 1 to 3 of the previous embodiments.

[0170] The supplementary sampling quantity determination unit 403 is used to determine the supplementary sampling quantity for each action based on the target sample quantity and the consumed sample quantity for each action. In this embodiment, the supplementary sampling quantity for each action is obtained by subtracting the consumed sample quantity for each action from the target sample quantity for each action. This step involves the same states; that is, determining the supplementary sampling quantity for each action based on the target sample quantity and the consumed sample quantity for each action refers to determining the supplementary sampling quantity for action a in state s based on the target sample quantity for action a in state s and the consumed sample quantity for action a in state s.

[0171] The supplementary sampling unit 404 is used to perform supplementary sampling based on the supplementary sampling quantity of each action, and to redetermine the action type of each action and the Q-factor sample variance of each state-action pair using the simulation results of the supplementary sampling. Specifically, the supplementary sampling process can be referred to the aforementioned embodiment, and will not be repeated here.

[0172] Analysis unit 405 is used to adjust the total amount of samples consumed, determine whether the total amount of samples consumed is less than the preset total amount of samples, if so, restart the target amount of samples determination unit, if not, output the optimal action observed in each state.

[0173] This embodiment, by employing a sampling quantity determination criterion, asymptotically maximizes the probability of correctly selecting the optimal action. It concentrates simulation resources on the truly important actions that need to be distinguished, making the observation results for each action increasingly accurate, ultimately leading to the optimal action. Compared to existing technologies that consider the entire set of feasible strategies, this approach can distinguish the differences in importance between different state-action pairs and ensures the predictability of the optimal action selection, thus improving the performance of the final action determination. Furthermore, by controlling the sampling quantity, this invention also improves the data efficiency of online reinforcement learning during the controlled system's operation and scheduling, significantly reducing the time required for online decision-making and decreasing data usage.

[0174] In one embodiment of the present invention, a computer device is also provided. As shown in FIG. 5, the computer device 502 may include one or more processors 504, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 502 may also include any memory 506 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 506 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 502. In one case, when the processor 504 executes associated instructions stored in any memory or combination of memories, the computer device 502 may perform any operation of the associated instructions. The computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0175] Computer device 502 may also include an input / output module 510 (I / O) for receiving various inputs (via input device 512) and providing various outputs (via output device 514). A specific output mechanism may include a presentation device 516 and an associated graphical user interface 518 (GUI). In other embodiments, the input / output module 510 (I / O), input device 512, and output device 514 may be omitted, and the device may function solely as a computer device within a network. Computer device 502 may also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the components described above together.

[0176] Communication link 522 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 522 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0177] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described method.

[0178] This invention also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the steps of the above-described method.

[0179] This invention also provides a computer program product, which includes a computer program that, when executed by a processor of a computer device, implements the methods described in any of the foregoing embodiments.

[0180] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0181] It should also be understood that, in the embodiments of the present invention, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the present invention, the character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0182] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or may be electrical, mechanical, or other forms of connection.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0186] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A high-efficiency reinforcement learning method for scheduling the operation of a controlled system, characterized in that, include: The simulation results of each state-action pair of the controlled system are sampled according to the preset sampling number. Based on the sampled simulation results, the action type of each action in each state, the Q factor sample variance of each state-action pair, and the total number of samples consumed are determined. The sampling data allocation algorithm is used to analyze the Q-factor sample variance and the total number of consumed samples of each state-action pair to obtain the target sample size of each action in each state. The sampling data allocation algorithm includes a sampling quantity determination criterion that distinguishes the importance of each type of action in each state, which is used to asymptotically maximize the probability of correctly selecting the optimal action. The number of additional samples for each action is determined based on the target sample size and the sample size already consumed for each action. Supplementary sampling is performed based on the number of supplementary samples for each action, and the action type and Q-factor sample variance of each state-action pair are re-determined using the simulation results of the supplementary sampling. Adjust the total amount of samples consumed, and determine whether the total amount of samples consumed is less than the preset total amount of samples. If so, recalculate the target amount of samples and subsequent steps. If not, output the optimal action observed in each state.

2. The method as described in claim 1, characterized in that, The process of determining the sampling data allocation algorithm includes: Based on the equipment and load information, system status, control actions, and optimization target information of the controlled system, establish the objective function of the control strategy; The objective function of the control strategy is transformed to obtain the action Q function for reinforcement learning; Based on the action Q function, an objective function for the approximate optimal action selection probability function and a constraint on the number of action samples are established. The objective function of the approximate optimal action selection probability function has data items that distinguish the importance of each type of action in each state. Based on the objective function of the near-optimal action selection probability function and the constraint on the number of action samples, the criteria for the sampling data allocation algorithm are determined.

3. The method as described in claim 2, characterized in that, Based on the action Q-function, establish the objective function for the approximate optimal action selection probability function and the constraints on the number of action samples, including: Based on the action Q function, establish the probability function for selecting the true optimal action and the constraint on the number of action samples. Based on the importance of each state and type of action, the probability function for selecting the true optimal action is transformed to obtain the probability function for selecting the approximate optimal action.

4. The method as described in claim 2, characterized in that, The objective function of the approximate optimal action selection probability function and the constraint on the number of action samples are expressed by the following formula: Max APCS; Here, APCS represents the approximate optimal action selection probability. Indicates the optimal action observed in state s. The Q-factor value, Q(s,a) represents the Q-factor value of the unobserved optimal action a in state s, π(s) represents the steady-state probability distribution of state s, ε represents a constant, and N sa Let N represent the number of samples for any action a under state s, and let N represent the total number of samples.

5. The method as described in claim 1, characterized in that, The sampling data allocation algorithm is used to indicate: The higher the noise-to-signal ratio of the non-observed optimal action in each state s, the larger the number of samples of the non-observed optimal action. In the noise-to-signal ratio, the noise refers to the sample standard deviation of the Q factor of the non-observed optimal action in each state s, and the information refers to the performance difference between the Q factor of the non-observed optimal action in each state s and the Q factor of the observed optimal action in state s. The number of samples for the best observed action in each state s should be greater than the number of samples for other non-best observed actions; The number of samples of the optimal action for each state should be proportional to the product of the steady-state probability of each state and the sample standard deviation of the Q factor of each state.

6. The method as described in claim 5, characterized in that, The sampling data allocation algorithm includes the following criteria: Criterion 1: For unobserved optimal actions i and j, Where, N si N represents the number of samples for action i in state s. sj σ represents the number of samples for action j in state s. si σ represents the sample standard deviation of the Q factor for action i in state s. sj δ represents the sample standard deviation of the Q-factor of action j in state s. si δ represents the difference in performance between action i in state s and the observed optimal action in state s. sj This represents the difference in performance between action j in state s and the observed optimal action in state s. Criterion 2: The optimal action a for observation in state s b , in, Indicates action a in state s b Number of samples, Indicates action a in state s b Q-factor sample standard deviation, N sa σ represents the number of samples for the unobserved optimal action a in state s. sa This represents the sample standard deviation of the Q factor for the unobserved optimal action a in state s. Criterion 3: The optimal action a for observation in state s1 b (s1) and the optimal action a observed in state s2 b (s2), in, Describes the optimal action a under state s1. b The number of samples (s1), Describes the optimal action a under state s2. b The number of samples in (s2), where π(s1) represents the steady-state probability distribution of state s1, and π(s2) represents the steady-state probability distribution of state s2. Describes the optimal action a under state s1. b The sample standard deviation of the Q factor of (s1), Describes the optimal action a under state s2. b The sample standard deviation of the Q factor of (s2).

7. The method as described in claim 1, characterized in that, The target sample size for each action in each state is obtained by analyzing the Q-factor sample variance and the total number of samples consumed for each state-action pair using a sampling data allocation algorithm, including: The Q-factor sample variance of each state-action pair is analyzed based on the sampling data allocation algorithm to obtain the target sample quantity ratio of each action in each state. The number of samples that should be obtained for each action in each state is obtained by multiplying the proportion of the target sample quantity for each action in each state by the total number of samples consumed. The larger of the number of samples that should be obtained for each action in each state and the number of samples that have actually been used for each action in each state is selected as the target number of samples for each action in each state.

8. A high-efficiency reinforcement learning device for scheduling the operation of a controlled system, characterized in that, include: The simulation calculation unit is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the action type of each action in each state, the Q-factor sample variance of each state-action pair, and the total number of samples consumed based on the sampled simulation results. The target sampling quantity determination unit is used to analyze the Q-factor sample variance and the total amount of consumed samples of each state-action pair using the sampling data allocation algorithm to obtain the target sample quantity of each action in each state. The sampling data allocation algorithm includes a sampling quantity determination criterion that distinguishes the importance of each type of action in each state, which is used to progressively maximize the probability of correctly selecting the optimal action. The supplementary sampling quantity determination unit is used to determine the supplementary sampling quantity for each action based on the target sample quantity for each action and the sample quantity consumed by each action. The supplementary sampling unit is used to perform supplementary sampling based on the number of supplementary samples for each action, and to redetermine the action type of each action and the Q-factor sample variance of each state-action pair using the simulation results of the supplementary sampling. The analysis unit is used to adjust the total amount of samples consumed, determine whether the total amount of samples consumed is less than the preset total amount of samples, and if so, restart the target amount of samples determination unit; otherwise, output the optimal action observed in each state.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor of the computer device, it implements the method of any one of claims 1 to 7.

Images