Supervised reinforcement learning method and apparatus for operation scheduling of controlled system

Abstract

A supervised reinforcement learning method and apparatus for operation scheduling of a controlled system. The method comprises: determining a Q factor, a Q factor sample variance and the total number of consumed samples of each state-action pair; calculating the importance of each non-observed optimal action in each state, and sending to a supervisor the top △ M non-observed optimal actions ranked by importance; adjusting the total number of consumed samples, and using a sampling data allocation algorithm to analyze the Q-factor sample variance and the total number of consumed samples of each state-action pair to obtain a target number of samples of each action in each state; determining the number of supplementary samples of each action; performing supplementary sampling on the basis of the number of supplementary samples of each action, and using supplementary sampling results to re-determine the previous steps; and determining whether the total number of consumed samples meets an action sampling quantity constraint condition, and if so, outputting an optimal action observed in each state.

Description

A supervised reinforcement learning method and apparatus for scheduling the operation of a controlled system Technical Field

[0001] This invention relates to the field of reinforcement learning, and more particularly to a supervised reinforcement learning method and apparatus for the operation scheduling of a controlled system. Background Technology

[0002] For complex controlled systems (such as smart grids), their economic operation has significant engineering requirements. Reinforcement learning, with its potential to learn from data and gradually improve policy performance, has thus gained significant attention in engineering applications. Taking new energy power systems as an example, these systems contain a large number of new energy (such as wind, solar, and fuel cells) power generation devices, energy storage devices, and electrical devices (such as building electricity and electric vehicle charging station electricity). Their operating strategies need to determine what actions to take given the current system state.

[0003] In reality, the aforementioned systems are characterized by large state and action spaces. The state space includes, for example, the states of various power generation devices, energy storage devices, and electrical consumption devices. The action space includes, for example, the charging or discharging power of energy storage devices, the charging power of electric vehicle charging stations, and the output of thermal power generating units. For these complex controlled systems, there are two main methods for determining the operation scheduling strategy: one is to directly use general reinforcement learning methods. This type of method requires a large amount of training data to obtain a good operation strategy, thus suffering from low learning efficiency and long processing time. The other method involves letting the system run for a period of time, then having experienced engineers evaluate the merits of two actions under the same state, and determining the scheduling strategy based on the engineer's evaluation. However, having engineers provide the optimal action for each state is too time-consuming and impractical. Furthermore, this method does not directly apply the engineer's experience to the optimization of the controlled system's operation strategy, resulting in the ineffective utilization of the engineer's experience.

[0004] Summary of the Invention

[0005] This invention addresses the problems in the prior art where the feedback information from supervisors is not combined with environmental interaction data during the operation and scheduling of complex controlled systems, resulting in low learning efficiency, poor accuracy of optimal actions, and waste of supervisor experience.

[0006] To address the aforementioned technical problems, the first aspect of this invention provides a supervised reinforcement learning method for the operation scheduling of a controlled system, comprising:

[0007] The simulation results of each state-action pair of the controlled system are sampled according to the preset sampling number. Based on the sampled simulation results, the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed are determined.

[0008] The importance of each non-observed optimal action in each state is calculated based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state. The observed optimal actions and their importance are then ranked in order of importance. M Each unobserved optimal action is sent to the supervisor so that the supervisor can determine the optimal action.

[0009] The total amount of samples consumed is adjusted, and the sample data allocation algorithm is used to analyze the Q-factor sample variance and the total amount of samples consumed for each state-action pair to obtain the target sample size for each action in each state. The sample data allocation algorithm includes the criteria for determining the number of samples for each type of action, which is used to progressively maximize the probability of correctly selecting the optimal action to observe.

[0010] The number of additional samples for each action is determined based on the target sample size and the sample size already consumed for each action.

[0011] Supplementary sampling is performed based on the number of supplementary samples for each action. The simulation results of the supplementary sampling are used to redetermine the action type steps and recalculate the importance of each non-observed optimal action in each state and the feedback supervisor steps.

[0012] Determine whether the total number of samples consumed is less than or equal to the preset total number of samples. If so, re-execute the steps to adjust the total number of samples consumed and the steps thereafter. Otherwise, output the optimal action for each state.

[0013] A second aspect of the present invention provides a supervised reinforcement learning device for the operation scheduling of a controlled system, comprising:

[0014] The simulation calculation unit is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed based on the sampled simulation results.

[0015] The supervisory unit calculates the importance of each non-observed optimal action in each state based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state, and ranks the observed optimal actions and their importance in the top Δ. M Each unobserved optimal action is sent to the supervisor so that the supervisor can determine the optimal action.

[0016] The allocation calculation unit is used to adjust the total amount of samples consumed. It uses the sampling data allocation algorithm to analyze the Q-factor sample variance of each state-action pair and the total amount of samples consumed to obtain the target sample size of each action in each state. The sampling data allocation algorithm includes the criteria for determining the number of samples for each type of action, which is used to progressively maximize the probability of correctly selecting the optimal action to observe.

[0017] The supplementary sampling quantity determination unit is used to determine the supplementary sampling quantity for each action based on the target sample quantity for each action and the sample quantity consumed by each action.

[0018] The supplementary sampling unit is used to perform supplementary sampling based on the number of supplementary samples for each action. The simulation calculation unit uses the simulation results of the supplementary sampling to re-execute the steps to determine the action type, and the supervision unit recalculates the importance of each non-observed optimal action in each state and provides feedback to the supervisor.

[0019] The analysis unit is used to determine whether the total number of consumed samples is less than or equal to the preset total number of samples. If so, the allocation calculation unit will re-execute the steps to adjust the total number of consumed samples and the subsequent steps. Otherwise, the optimal action for each state will be output.

[0020] A third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the foregoing embodiments.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor of a computer device, implements the method described in any of the foregoing embodiments.

[0022] A fifth aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor of a computer device, implements the method described in any of the foregoing embodiments.

[0023] The supervised reinforcement learning method and apparatus for controlled system operation scheduling provided by this invention are applicable to complex controlled systems including multiple states and multiple actions, such as smart grids and new energy power systems, and can achieve the following technical effects:

[0024] (1) By optimizing the use of limited opportunities to communicate with supervisors and limited opportunities to interact with the environment, the supervision of supervisors and sampling data can be combined to improve the efficiency and performance of reinforcement learning.

[0025] (2) By filtering out the non-observed optimal actions and observed optimal actions with the highest importance ranking to the supervisor, the accuracy of determining the optimal action can be improved.

[0026] (3) By using the sampling quantity to determine the criterion, the probability of correctly selecting the optimal action can be gradually maximized. Simulation resources are concentrated on the important actions that truly need to be distinguished, achieving effective sampling, reducing learning time, lowering time costs, and making the observation results of each action increasingly accurate, ultimately obtaining the optimal action. At the same time, by controlling the sampling quantity, the data efficiency of online reinforcement learning during the operation and scheduling of the controlled system can be improved, significantly reducing the time required for online decision-making and reducing data usage.

[0027] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 shows a flowchart of the sampling data allocation algorithm determination process according to an embodiment of the present invention;

[0030] Figure 2 shows a flowchart of the supervised reinforcement learning method for the operation scheduling of the controlled system according to an embodiment of the present invention;

[0031] Figure 3 shows a flowchart of the process for determining the sampling amount of each action target under each state in an embodiment of the present invention;

[0032] Figure 4 shows a structural diagram of the supervised reinforcement learning device for the operation scheduling of the controlled system according to an embodiment of the present invention;

[0033] Figure 5 shows a structural diagram of a computer device according to an embodiment of the present invention;

[0034] Figure 6 shows a flowchart of the supervised reinforcement learning program design for the operation scheduling of the controlled system according to an embodiment of the present invention.

[0035] Explanation of symbols in the attached figures: 401, Simulation calculation unit; 402, Supervision unit; 403, Allocation calculation unit; 404, Supplementary sampling quantity determination unit; 405, Supplementary sampling unit; 406, Analysis unit; 502, Computer equipment; 504, Processor; 506, Memory; 508, Drive mechanism; 510, Input / output module; 512, Input device; 514, Output device; 516, Presentation device; 518, Graphical user interface; 520, Network interface; 522, Communication link; 524, Communication bus. Detailed Implementation

[0036] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0037] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0038] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0039] It should be noted that the data involved in this invention (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0040] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of the present invention. However, they do not mean that the applicant has used or necessarily used the solution.

[0041] The controlled system described in this invention is a system with multiple states and multiple actions, such as a new energy power system. The corresponding state space includes the states of various power generation equipment, energy storage equipment, and electrical equipment, and the corresponding action space includes the charging or discharging power of energy storage equipment, the charging power of electric vehicle charging piles, and the output of thermal power generating units. In specific implementations, it can also be other multi-state and multi-action systems. This invention does not specifically limit the controlled system.

[0042] Before implementing the supervised reinforcement learning method for controlling system operation scheduling described in this invention, a sampling data allocation algorithm needs to be established first to indicate:

[0043] (1) The higher the signal-to-noise ratio of the non-observed optimal action under each state s, the greater the number of samples of the non-observed optimal action.

[0044] In the noise-information ratio, noise refers to the sample standard deviation of the Q factor of the non-observed optimal action in each state s, and information refers to the performance difference between the Q factor of the observed optimal action in each state s and the Q factor of the non-observed optimal action in each state s.

[0045] (2) The number of samples of the best observed action in each state should be proportional to the product of the steady-state probability of each state and the sample standard deviation of the Q factor of the best observed action in each state.

[0046] (3) For observation-optimal actions and non-observation-optimal actions that have not been confirmed by the supervisor, the number of samples of observation-optimal actions in each state s should be greater than the number of samples of other non-observation-optimal actions.

[0047] Specifically, as shown in Figure 1, the process of determining the sampling data allocation algorithm includes:

[0048] Step 101: Based on the equipment information, load information, system status, control actions, and optimization target information of the controlled system, establish the objective function, action sampling quantity constraints, and consultation quantity constraints of the control strategy.

[0049] Step 102: Transform the objective function of the control policy to obtain the action Q factor for reinforcement learning.

[0050] Step 103: Based on the action Q factor, construct an index function for calculating the importance of the unobserved optimal action and an objective function for the probability function of selecting the approximate observed optimal action.

[0051] Step 104: Determine the criteria for the sampling data allocation algorithm based on the objective function, index function, action sampling quantity constraint, and consultation quantity constraint of the approximate observation optimal action selection probability function.

[0052] In detail, the equipment information, load information, system status, control actions, and optimization target information in step 101 can be determined according to the specific situation of the controlled system, and this invention does not limit the specific parameters included therein. An objective function is established based on the equipment information, load information, system status, control actions, and optimization target.

[0053] Taking a new energy power system as an example, the controlled system consists of power generation equipment (such as solar, wind, fuel cells, and thermal power generators), energy storage equipment, and power consumption equipment (such as building electricity and electric vehicle charging station electricity). Actions need to be determined based on the current state of the new energy power system (such as the charging or discharging power of energy storage equipment and the charging power of electric vehicles). Experienced engineers in new energy power systems can accurately determine the merits of two actions given a particular state. Furthermore, reinforcement learning can be driven by data through interaction with the environment.

[0054] In some implementations, the control objective of the controlled system is expressed using the following formula:

[0055] Among them, s t d represents the system state at time t; d represents the policy, which needs to be based on state s. t Define the action; c represents the cost function; T represents the number of simulation stages; T represents the total simulation duration; J(d) represents the average operating cost of strategy d.

[0056] Furthermore, step 102 above can be used to obtain the action Q factor expressed by the following formula according to formula (1): Q d (s,a)=c(s,a)+∑P(s′|s,a)Q d (s′,d(s′))-J(d) (2)

[0057] Among them, Q d (s,a) represents the Q-factor of action a in state s; d represents the given policy; for simplicity, the above Q... d (s,a) can also be represented as Q(s,a); c represents the target object value corresponding to state s and action a; the second term on the right side of the above equation represents the future target object value; s' represents the future state; P(s′|s,a) represents the probability of transitioning from state s to the next state s' by taking action a; Q d (s',d(s')) represents the Q-factor of state s' and its policy d'.

[0058] The Q factor represented by formula (2) can be obtained by taking samples. We obtain, where k = 1, 2, ..., N sa N saThe number of samples for (state, action) pairs is represented by Q(s,a) using the sample mean, as shown in the following formula:

[0059] Furthermore, according to formula (3), a greedy strategy based on this estimate can be obtained, namely:

[0060] For any state in the state space and any action in the control space, when N sa →∞, It converges to Q(s,a). Converges at:

[0061] When all (s, a) are visited an infinite number of times, we can obtain d. b (s) converges to the optimal policy d * (s)≡argmin d∈D Q(s,a), where D represents the set of all policies. Given a total sample size N, different allocations may lead to different policies. In addition to the samples, we also seek guidance from a supervisor. Given a state s and two actions i and j, the supervisor tells us the true order of Q(s,i) and Q(s,j), or Q(s,i) ≤ Q(s,j), or vice versa. This guidance provides much more accurate information than the samples and is therefore more valuable. Compared to the total sample size N, our requests to the supervisor are limited to a much smaller M, M << |S|(|A|-1), meaning we do not have sufficient budget to determine all optimal actions solely through interaction with the supervisor. Combining the samples and the supervisor's guidance, we aim to minimize the policy's control objective. Therefore, we can obtain the following objective function for the controlled system's control policy:

[0062] in, The objective function of the control policy is N, for example, the average cost of the policy. sa N represents the number of samples of the Q factor corresponding to state s and action a, and N represents the total number of samples.

[0063] M(s,a,a') represents the number of times actions a and a' in state s are consulted with the supervisor. To improve efficiency, consultation is generally limited to once, i.e., M(s,a,a')∈{0,1}, where M represents the total number of consultations with the supervisor. N and M are preset values ​​that can be set by the user according to actual needs. For convenience, the problem represented by formula (6) can be denoted as P1.

[0064] In detail, the supervisor described in this invention can be an engineer familiar with the control of the system being controlled, or it can be an intelligent algorithm used to evaluate the merits of two actions, that is, to provide a judgment on which of the two actions is superior. This invention does not limit the specific type of supervisor; the supervisor is capable of making an accurate judgment on the merits of two actions.

[0065] When implementing step 103, the problem P1 represented by formula (6) can be refined into a more specific optimization problem form based on the problem structure of online reinforcement learning, denoted as problem P2, as shown in the following formula:

[0066] The final strategy is the strategy that maximizes the probability of the target object (e.g., average operating cost) within a given ε. The constraints on the number of action samples and the number of consultations are the same as those in the previous problem, representing the total number of samples and the total number of consultations, respectively.

[0067] Compared with problem P1 represented by formula (6), problem P2 represented by formula (7) has an objective function that is easier to calculate, and the difficulty of solving the problem can be adjusted by changing the value of the constant ε, making it more practical.

[0068] Notice:

[0069] Among them, the first item ∩ s ∩ a≠b(s) (Q(s, b(s)) ≤ Q(s, a)) indicates that the observed optimal action is the true optimal action, and the second term ∑ s π(s)Q(s,b(s))≤ε indicates that the authenticity of the observed optimal motion is no greater than a given constant ε, which is determined by the given basic strategy.

[0070] Based on the Bonferroni inequality, we can obtain the following inequality for the right-hand side of formula (8):

[0071] Where s represents the state, a represents the unobserved optimal action, b(s) represents the observed optimal action, Q(s,b(s)) represents the Q-factor value of the observed optimal action b(s) in state s, Q(s,a) represents the Q-factor value of the unobserved optimal action a in state s, π(s) represents the steady-state probability distribution of state s, ε represents a constant, and N saLet N represent the number of samples of the Q factor corresponding to state s and action a, and let M(s,a,a') represent the total number of samples. Let M(s,a,a') represent the number of times actions a and a' in state s are consulted with the supervisor, and M represent the total number of consultations with the supervisor. The term on the right side of formula (9) is called the Approximate Observational Optimal Action Selection Probability Function, i.e., APCS, Approximate Probability of Correct Selection.

[0072] Furthermore, problem P2, represented by formula (7), can be expressed as problem P3, represented by the following formula:

[0073] Next, we optimize the supervisor's guidance, which informs us of the true order between Q(s,a) and Q(s,a'), where a and a' are any two actions. Of course, we can choose any pair of actions to send to the supervisor's guidance. However, some requests provide more information and improve APCS better than others. For example, when a = b(s) (b(s) is the observed optimal action in state s), and a' is the second-optimal action observed in state s, sending a request to the supervisor directly tells us which action between a and a' is better, thus reducing 1 - Pr{Q(s,b(s)≤Q(s,a)} to 0. On the other hand, when a,a' ≠ b(s), the request result does not affect APCS. Therefore, we need to select certain actions to request guidance from the supervisor. Furthermore, since we are looking at it from a Bayesian perspective, the sample mean... These are given and fixed. Their truth values ​​are random variables following a posterior distribution. Then, the supervisor's guidance can directly determine the order of the random truth values ​​Q(s,a) and Q(s,a'). and The order is consistent. Then, we no longer need to consider the possibility that their true order is other than that, and we can remove this term from APCS, so we choose the pair of actions that maximizes APCS.

[0074] Based on the above analysis, we can obtain the following formula to represent the importance of the unobserved optimal action a in state s:

[0075] I(s,a)≡1-Pr{Q(s,b(s))≤Q(s,a)} (12)

[0076] Furthermore, combining with formula (12), the APCS of formula (10) can be expressed as:

[0077] In step 104 above, based on the objective function, index function, action sampling quantity constraint, and consultation quantity constraint of the approximate observation optimal action selection probability function, the criteria for determining the sampling data allocation algorithm are as follows:

[0078] Criterion 1: For unobserved optimal actions i and j,

[0079] Where, N si N represents the number of samples for action i in state s. sj σ represents the number of samples for action j in state s. si σ represents the sample standard deviation of the Q factor for action i in state s. sj δ represents the sample standard deviation of the Q-factor of action j in state s. si δ represents the difference between the Q-factor of the observed optimal action in state s and the performance of action i in state s. sj The Q-factor of the observed optimal action in state s represents the performance difference between action j in state s and the optimal action in state s.

[0080] Criterion 2: For the observation-optimal action b in state i and the observation-optimal action b in state j,

[0081] Where, N ib N represents the number of samples for the optimal action b(i) observed in state i. jb Let π(i) represent the number of samples for the optimal action b(j) observed in state j, π(i) represent the steady-state probability distribution of state i, and π(j) represent the steady-state probability distribution of state j. ib σ represents the Q-factor sample standard deviation of the observed optimal action b(i) in state i. jb The Q-factor sample standard deviation represents the observed optimal action b(j) in state j;

[0082] Criterion 3: For the observed optimal action b in state s,

[0083] Where, N sb σ represents the number of samples of action b(s) in state s. sb N represents the Q-factor sample standard deviation of action b(s) in state s. sa σ represents the number of samples for the unobserved optimal action a in state s. sa Let represent the sample standard deviation of the Q factor for the unobserved optimal action a in state s, and M(s,b(s),a) represent the number of times action b(s) and a are consulted with the supervisor in state s.

[0084] Specifically, The Q-factor represents the performance difference between the observed optimal action b(s) and the unobserved optimal action a in state s.

[0085] Rule 1 states that for actions i and j that are not optimal in state s, their sampling size should be proportional to the square of the "noise-to-signal ratio". Here, "noise" refers to the sample standard deviation of the Q factor, and "signal" refers to the difference in Q factor performance between the optimal action in state s and another action a. The intention of Rule 1 is to allocate more sampling size to actions with inaccurate observations or whose true values ​​are close to the true values.

[0086] Criterion 2 states that for the optimal observed action in two different states i and j, the number of samples obtained should be proportional to the steady-state distribution of the corresponding state and the sample standard deviation of the Q-factor. Therefore, the more common a state is, the less accurate its corresponding Q-factor observations are, and the more samples should be obtained.

[0087] Criterion 3 states that for the observed optimal action b(s) in state s, the square of the normalized sample size should be the sum of the squares of the normalized sample sizes of other actions in the same state. Furthermore, considering the high accuracy of the supervisor's information, once the supervisor has been consulted, there is no need to waste data and samples asking the same question again (i.e., which action is better).

[0088] In one embodiment, step 104 is specifically implemented as follows:

[0089] Based on APCS, we introduce the Lagrange function F, from which we can obtain:

[0090] Where λ is a Lagrange multiplier.

[0091] First, considering an action a≠b(s), the partial derivative of formula (14) with respect to the sampled amount of action a can be obtained as follows:

[0092] According to formula (13)

[0093] In formula (16), the second term M(s,b(s),a) = 0 indicates that the supervisor was not consulted to determine which of Q(s,b(s)) and Q(s,a) is better. Note N sa This does not affect the first term in formula (16), from which we can obtain:

[0094] If M(s,b(s),a)=1, then:

[0095] If M(s,b(s),a)=1, then:

[0096] Combining formula (12), we can obtain:

[0097] Please note that Q(s,b(s))-Q(s,a) is a Gaussian random variable. in, Further results were obtained:

[0098] Combining formulas (19)-(23), we can obtain:

[0099] Combining formulas (19) and (20), we can obtain:

[0100] Next, considering the optimal action b(s), based on formula (14), we can obtain:

[0101] For the first term on the right side of formula (26), note The mean is variance is The Gaussian random variable, from which we can know in, Therefore, we can conclude that:

[0102] By integrating formulas (27) and (29), we can obtain:

[0103] For the second term of formula (26), note N. sb Only I(s,a) is affected, where a≠b(s), and M(s,b(s),a)=0. From this, we can obtain:

[0104] By integrating formulas (32) and (34), we can obtain:

[0105] By integrating (31) to (35), we can obtain:

[0106] By integrating (26), (30) and (36), we can obtain:

[0107] We will analyze N next. si With N sjThe relationship is given by the condition that i,j ≠ b(s). Note that the local optimal solution Karush-Kuhn-Tucker (KKT) condition indicates that... According to formulas (15) and (24), we can obtain:

[0108] By integrating formulas (38) and (39), we can obtain:

[0109] Rearranging formula (40) yields:

[0110] When the total sample size N approaches infinity, the sample size of the observed optimal action b(s) is much larger than the sample size of other actions, from which we can conclude:

[0111] By integrating (41) to (43), we can obtain:

[0112] Taking the logarithm of both sides, we get:

[0113] Since the growth rate of the logarithmic term is much slower than that of the other terms, the other terms can be omitted. Based on the above formula, we can further derive:

[0114] After rearranging the formula, we get:

[0115] Next, we will analyze the sampling size N for observing the optimal action. sb The sampling amount N of the non-observation optimal action sa , a≠b(s), M(s,b(s),a)=0. Formula (37), let N sb1 =N sb2 N sb1 Corresponding to Pr{∑ s π(s)Q(s,b(s))≤ε},N sb2 For Pr{Q(s, b(s))≤Q(s, a)}, we can obtain:

[0116] Considering the optimal action for observation under different states, named b(s1) and b(s2), then according to formula (35), we can obtain:

[0117] After sorting, we get:

[0118] By arranging (24) and (46), we can obtain:

[0119] Furthermore, the criteria for the sampling data allocation algorithm can be obtained according to formulas (44), (47), and (48).

[0120] Based on the determined sampling data allocation algorithm, a supervised reinforcement learning method for the operation scheduling of the controlled system can be executed. Specifically, as shown in Figure 2, the supervised reinforcement learning method for the operation scheduling of the controlled system includes:

[0121] Step 201: Sample the simulation results of each state-action pair of the controlled system according to the preset sampling quantity, and determine the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed based on the sampled simulation results.

[0122] During this step, each action and each state-action pair in each state can be denoted as (state, action). Action types include observed optimal and unobserved optimal.

[0123] The preset sampling quantity (N0) can be set based on the preset total sampling quantity N, the preset number of loops, the motion space, and the newly added sampling quantity Δ. Generally, the preset number of loops is greater than 10, and the preset total sampling quantity N is much larger than the preset sampling quantity N0 multiplied by the number of motions in the motion space.

[0124] The actions in this step can be determined based on the action space. The states in this step can be determined based on the state space.

[0125] In this step, the (state, action) can be directly applied to the simulation model of the controlled system (e.g., a digital twin model), and the simulation results for each (state, action) are determined through the simulation model. In practice, the execution results of a preset sample size can also be obtained from historical data based on the state and action, and the obtained execution results are used as the simulation results.

[0126] The total number of samples consumed in this step is the product of the total number of states, the total number of actions, and the preset number of samples. It can be calculated using the following formula during implementation:

[0127] Where n0 represents the preset sampling quantity; S represents the state quantity in the state space; and A represents the action quantity in the action space.

[0128] Step 202: Calculate the importance of each non-observed optimal action in each state based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state, and rank the observed optimal actions and their importance by Δ. M The non-observed optimal action is sent to the supervisor so that the supervisor can determine the optimal action.

[0129] In detail, the importance of each non-observed optimal action in each state can reflect the degree of closeness between the non-observed optimal action and the observed optimal action. The greater the importance, the greater the probability that the non-observed optimal action is the optimal action.

[0130] Based on this, we can construct the following formula to represent the importance of each unobserved optimal action in each state:

[0131] I(s,a)≡1-Pr{Q(s,b(s))≤Q(s,a)};

[0132] Where I(s,a) represents the importance of the non-observed optimal action a in state s, and I(s,a) quantifies the probability that the non-observed optimal action a in state s is truly better than the observed optimal action b; Q(s,b(s)) represents the Q factor of the observed optimal action b(s) in state s; Q(s,a) represents the Q factor of the non-observed optimal action a in state s.

[0133] Specifically, the above formula considers a minimization problem. If Pr{Q(s,b)<=Q(s,a)} is larger, it indicates that the selection of the observed optimal action b(s) is more correct. Therefore, the smaller I(s,a) is, the closer it is to 0, indicating that it is less necessary to consider the possibility that the non-observed optimal action a is the best action. Conversely, if Pr is smaller, it indicates that the selection of the observed optimal action b(s) may be incorrect. Therefore, the larger I(s,a) is, the closer it is to 1, indicating that it is necessary to consider the possibility that the non-observed optimal action a is the optimal action.

[0134] Therefore, after calculating I(s,a) for all unobserved optimal actions a, the magnitude of I can reflect the probability that the unobserved optimal action may be the optimal action. The larger I is, the greater the probability that the corresponding unobserved optimal action a is the optimal action.

[0135] In this step, D M As a preset value, D is generally used to conserve resources. M The value is usually 1.

[0136] During this step, to conserve resources for communication with supervisors, it can be determined whether the importance of each non-observed optimal action is less than a preset value, such as 0.5. If so, the process of ranking the observed optimal actions and their importance by Δ is omitted. M If no non-observable optimal action is sent to the supervisor, then the top D actions are selected. M The non-observation optimal action will be ranked by importance. M Both the non-observed optimal action and the observed optimal action are sent to the supervisor.

[0137] Step 203: Adjust the total amount of samples consumed. Analyze the Q-factor sample variance and the total amount of samples consumed for each state-action pair using the sampling data allocation algorithm to obtain the target sample size for each action in each state. The sampling data allocation algorithm includes criteria for determining the number of samples for each type of action, used to progressively maximize the probability of correctly selecting the optimal action for observation.

[0138] The specific criteria for the sampling data allocation algorithm in this step are described in the aforementioned embodiments and will not be repeated here.

[0139] When implementing this step, the total amount of samples consumed is adjusted by adding the newly added samples (Δ) to the already consumed samples.

[0140] In some implementations, as shown in Figure 3, this real-time process specifically includes:

[0141] Step 301: Analyze the Q-factor sample variance of each state-action pair according to the sampling data allocation algorithm to obtain the target sample quantity ratio of each action in each state.

[0142] Step 302: The number of samples that should be obtained for each action in each state is obtained by multiplying the target sample quantity ratio of each action in each state by the total number of samples consumed.

[0143] Step 303: Select the larger of the number of samples that should be obtained for each action in each state and the number of samples that have actually been used for each action in each state as the target number of samples for each action in each state.

[0144] Step 204: Determine the number of additional samples for each action based on the target sample size and the sample size already consumed for each action.

[0145] During this step, the number of additional samples for each action is obtained by subtracting the number of samples already consumed for each action from the target sample size for each action.

[0146] Step 205: Perform supplementary sampling based on the number of supplementary samples for each action, and use the simulation results of the supplementary sampling to redetermine the action type and the Q-factor sample variance of each state-action pair, and recalculate the importance of each unobserved optimal action in each state and the feedback supervisor steps.

[0147] In detail, the supplementary sampling process based on the number of supplementary samples can be referred to step 201, and its specific implementation process will not be repeated here.

[0148] The sample variance of the Q factor for each state-action pair can be calculated based on the Q factor obtained from sampling each state-action pair. For details, please refer to the existing variance calculation formula, which will not be elaborated here.

[0149] Step 206: Determine whether the total number of consumed samples is less than or equal to the preset total number of samples. If so, re-execute the steps to adjust the total number of consumed samples and the steps thereafter. Otherwise, output the optimal action for each state.

[0150] The preset total sampling quantity N can be set according to the initial value of the preset sampling quantity in step 201 and the action space, as long as it can meet the preset number of repeated cycles from step 201 to step 206.

[0151] In this step, if the total number of samples consumed is less than or equal to the action sampling amount constraint, it means that the number of samples consumed after the next iteration has not reached or is exactly equal to the preset total number of samples, and the feasibility of the action can continue to be observed. If the total number of samples consumed is greater than the action sampling amount constraint, it means that the number of samples consumed after the next iteration has exceeded the preset total number of samples, and the feasibility of the action cannot continue to be observed.

[0152] In one specific implementation, the algorithm flow of this embodiment is shown in Figure 6. In Figure 6, d0 represents the initial strategy, n is the preset sampling amount, and Δ N Indicates the newly added sampling amount, △ M Indicates the number of inquiries, n, △ N Table △ M It can be preset.

[0153] a and a' indicate actions. Let represent the action space, s represent the state, S represent the state space, and M(s,a,a') represent the number of consultations for which action a or a' is better in state s. During initialization, the number of consultations for any two actions in each state s is 0, i.e., M(s,a,a')=0.

[0154] σ sa The Q-factor sample standard deviation represents the state s-action a. I(s,a) represents the actual number of samples consumed, and I(s,a) represents the importance of action a in state s. During implementation, the top Δ values ​​based on importance are selected. M For action a, set M(s, b(s), a) = 1 for the selected action a, that is, each selected action needs to be combined with the optimal action and sent to the supervisor for consultation. For actions with importance ranking in △ M The subsequent action is to set M(s, b(s), ·) = 0, meaning there is no need to rank importance in △. M Consult the supervisor before proceeding.

[0155] N represents the preset total sample size, α sa This indicates the ratio of the number of samples that should be obtained for state s to the number of samples that should be obtained for action a. N represents the number of samples that should be obtained for state s-action a. saThe number of samples consumed for state s-action i is N. sa 'N represents the target sample size for state s and action i.' sa '-N sa This is the supplementary sampling amount for state s-action a.

[0156] d(s) represents the control strategy determined under state s.

[0157] After obtaining the control strategies for each state through the above embodiments, the controlled system can be controlled according to the control strategies for each state. Specifically, for example, the control strategies for each state are sent to the controlled system, and the control device of the controlled system controls the controlled device of the controlled system according to the control strategies for each state.

[0158] The supervised reinforcement learning method for controlled system operation scheduling provided in this embodiment is applicable to complex controlled systems with multiple states and actions, such as smart grids and new energy power systems, and can achieve the following technical effects:

[0159] (1) By optimizing the use of limited opportunities to communicate with supervisors and limited opportunities to interact with the environment, the supervision of supervisors and sampling data can be combined to improve the efficiency and performance of reinforcement learning.

[0160] (2) By filtering out the non-observed optimal actions and observed optimal actions with the highest importance ranking to the supervisor, the supervisor can determine the optimal action type, which can improve the accuracy of optimal action determination.

[0161] (3) By using the sampling quantity to determine the criterion, the probability of correctly selecting the optimal action can be gradually maximized. Simulation resources are concentrated on the important actions that truly need to be distinguished, achieving effective sampling, reducing learning time, lowering time costs, and making the observation results of each action increasingly accurate, ultimately obtaining the optimal action. At the same time, by controlling the sampling quantity, the data efficiency of online reinforcement learning during the operation and scheduling of the controlled system can be improved, significantly reducing the time required for online decision-making and reducing data usage.

[0162] Based on the same inventive concept, this paper also provides a supervised reinforcement learning device for the operation scheduling of a controlled system, as described in the following embodiments. Since the principle of the supervised reinforcement learning device for the operation scheduling of a controlled system is similar to that of the supervised reinforcement learning method for the operation scheduling of a controlled system, the implementation of the supervised reinforcement learning device for the operation scheduling of a controlled system can refer to the supervised reinforcement learning method for the operation scheduling of a controlled system, and the repetitions will not be repeated.

[0163] Specifically, as shown in Figure 4, the supervised reinforcement learning device for the operation scheduling of the controlled system includes:

[0164] The simulation calculation unit 401 is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed based on the sampled simulation results.

[0165] Supervision unit 402 is used to calculate the importance of each non-observed optimal action in each state based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state, and to rank the observed optimal actions and their importance in the top Δ. M Each unobserved optimal action is sent to the supervisor so that the supervisor can determine the optimal action.

[0166] The allocation calculation unit 403 is used to adjust the total amount of samples consumed. It uses the sampling data allocation algorithm to analyze the Q factor sample variance of each state-action pair and the total amount of samples consumed to obtain the target sample size of each action in each state. The sampling data allocation algorithm includes the criteria for determining the number of samples for each type of action, which is used to progressively maximize the probability of correctly selecting the optimal action to observe.

[0167] The supplementary sampling quantity determination unit 404 is used to determine the supplementary sampling quantity for each action based on the target sample quantity for each action and the sample quantity consumed for each action.

[0168] The supplementary sampling unit 405 is used to perform supplementary sampling based on the number of supplementary samples for each action. The simulation calculation unit 401 uses the simulation results of the supplementary sampling to re-execute the steps to determine the action type, and the supervision unit 402 recalculates the importance of each non-observed optimal action in each state and provides feedback to the supervisor.

[0169] Analysis unit 406 is used to determine whether the total amount of consumed samples is less than or equal to the preset total sampling amount. If so, the allocation calculation unit will re-execute the steps of adjusting the total amount of consumed samples and thereafter. Otherwise, it will output the optimal action observed in each state.

[0170] This embodiment can apply the supervisor's experience to the process of determining the control strategy of the controlled system. By sampling control in the process of determining the control strategy, the data efficiency of the optimization process of the operation and scheduling strategy of the new energy power system can be improved, the performance of the obtained strategy can be significantly improved, and the amount of data used can be reduced.

[0171] In one embodiment of the present invention, a computer device is also provided. As shown in FIG. 5, the computer device 502 may include one or more processors 504, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 502 may also include any memory 506 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 506 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 502. In one case, when the processor 504 executes associated instructions stored in any memory or combination of memories, the computer device 502 may perform any operation of the associated instructions. The computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0172] Computer device 502 may also include an input / output module 510 (I / O) for receiving various inputs (via input device 512) and providing various outputs (via output device 514). A specific output mechanism may include a presentation device 516 and an associated graphical user interface 518 (GUI). In other embodiments, the input / output module 510 (I / O), input device 512, and output device 514 may be omitted, and the device may function solely as a computer device within a network. Computer device 502 may also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the components described above together.

[0173] Communication link 522 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 522 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0174] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described method.

[0175] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the method described in any of the foregoing embodiments.

[0176] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0177] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0178] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0182] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0183] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A supervised reinforcement learning method for scheduling the operation of a controlled system, characterized in that, include: The simulation results of each state-action pair of the controlled system are sampled according to the preset sampling number. Based on the sampled simulation results, the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed are determined. The importance of each non-observed optimal action in each state is calculated based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state. The observed optimal actions and their importance are then ranked in order of importance. M Each unobserved optimal action is sent to the supervisor so that the supervisor can determine the optimal action. The total amount of samples consumed is adjusted, and the sample data allocation algorithm is used to analyze the Q-factor sample variance and the total amount of samples consumed for each state-action pair to obtain the target sample size for each action in each state. The sample data allocation algorithm includes the criteria for determining the number of samples for each type of action, which is used to progressively maximize the probability of correctly selecting the optimal action to observe. The number of additional samples for each action is determined based on the target sample size and the sample size already consumed for each action. Supplementary sampling is performed based on the number of supplementary samples for each action. The simulation results of the supplementary sampling are used to redetermine the action type steps and recalculate the importance of each non-observed optimal action in each state and the feedback supervisor steps. Determine whether the total number of samples consumed is less than or equal to the preset total number of samples. If so, re-execute the steps to adjust the total number of samples consumed and the steps thereafter. Otherwise, output the optimal action for each state.

2. The method as described in claim 1, characterized in that, The process of determining the sampling data allocation algorithm includes: Based on the equipment information, load information, system status, control actions, and optimization target information of the controlled system, establish the objective function, action sampling quantity constraints, and consultation quantity constraints of the control strategy. The objective function of the control strategy is transformed to obtain the action Q factor for reinforcement learning; Based on the action Q factor, construct an index function for calculating the importance of the unobserved optimal action and an objective function for the probability function of selecting the approximate observed optimal action; Based on the objective function, index function, action sampling quantity constraint, and consultation quantity constraint of the approximate observation optimal action selection probability function, the criteria for the sampling data allocation algorithm are determined.

3. The method as described in claim 2, characterized in that, The objective function of the approximate observation optimal action selection probability function, the action sampling quantity constraint, and the consultation quantity constraint are expressed by the following formula: Max APCS; ∑ s,a,a' M(s,a,a')=M; Where APCS represents the approximate observation optimal action selection probability function, s represents the state, a represents the non-observation optimal action, b(s) represents the observation optimal action, Q(s,b(s)) represents the Q factor value of the observation optimal action b(s) in state s, Q(s,a) represents the Q factor value of the non-observation optimal action a in state s, π(s) represents the steady-state probability distribution of state s, ε represents a constant, and N sa Let N represent the number of samples of the Q factor corresponding to state s and action a, N represent the total number of samples, M(s,a,a') represent the number of times the supervisor is consulted regarding actions a and a' in state s, and M represent the total number of consultations with the supervisor.

4. The method as described in claim 1, characterized in that, The importance of each unobserved optimal action in each state is calculated based on the Q-factor of each unobserved optimal action and the Q-factor of the observed optimal action in each state, including: The importance of each unobserved optimal action in each state is calculated using the following formula: I(s,α)=1-Pr{Q(s,b(s))≤Q(s,α)}; Where I(s,a) represents the importance of the unobserved optimal action a in state s; Q(s,b(s)) represents the Q factor of the observed optimal action b in state s; and Q(s,a) represents the Q factor of the unobserved optimal action a in state s.

5. The method as described in claim 1, characterized in that, The sampling data allocation algorithm is used to indicate: The higher the noise-to-signal ratio of the non-observed optimal action in each state s, the larger the number of samples of the non-observed optimal action. In the noise-to-signal ratio, the noise refers to the sample standard deviation of the Q factor of the non-observed optimal action in each state s, and the information refers to the performance difference between the Q factor of the observed optimal action in each state s and the Q factor of the non-observed optimal action in each state s. The number of samples of the observed optimal action in each state should be proportional to the product of the steady-state probability of each state and the sample standard deviation of the Q factor of the observed optimal action in each state. For both observation-optimal and non-observation-optimal actions that have not been confirmed by the supervisor, the number of samples for observation-optimal actions in each state s should be greater than the number of samples for other non-observation-optimal actions.

6. The method as described in claim 5, characterized in that, The sampling data allocation algorithm includes the following criteria: Criterion 1: For unobserved optimal actions i and j, Where, N si N represents the number of samples for action i in state s. sj σ represents the number of samples for action j in state s. si σ represents the sample standard deviation of the Q factor for action i in state s. sj The Q-factor sample standard for action j in state s Difference, δ si δ represents the difference between the Q-factor of the observed optimal action in state s and the performance of action i in state s. sj The Q-factor of the observed optimal action in state s represents the performance difference between action j in state s and the optimal action in state s. Criterion 2: For the observation-optimal action b in state i and the observation-optimal action b in state j, Where, N ib N represents the number of samples for the optimal action b(i) observed in state i. jb Let π(i) represent the number of samples for the optimal action b(j) observed in state j, π(i) represent the steady-state probability distribution of state i, and π(j) represent the steady-state probability distribution of state j. ib σ represents the Q-factor sample standard deviation of the observed optimal action b(i) in state i. jb The Q-factor sample standard deviation represents the observed optimal action b(j) in state j; Criterion 3: For the observed optimal action b in state s, Where, N sb σ represents the number of samples of action b(s) in state s. sb N represents the Q-factor sample standard deviation of action b(s) in state s. sa σ represents the number of samples for the unobserved optimal action a in state s. sa Let represent the sample standard deviation of the Q factor for the unobserved optimal action a in state s, and M(s,b(s),a) represent the number of times action b(s) and a are consulted with the supervisor in state s.

7. The method as described in claim 1, characterized in that, The target sample size for each action in each state is obtained by analyzing the Q-factor sample variance and the total number of samples consumed for each state-action pair using a sampling data allocation algorithm, including: The Q-factor sample variance of each state-action pair is analyzed based on the sampling data allocation algorithm to obtain the target sample quantity ratio of each action in each state. The number of samples that should be obtained for each action in each state is obtained by multiplying the proportion of the target sample quantity for each action in each state by the total number of samples consumed. The larger of the number of samples that should be obtained for each action in each state and the number of samples that have actually been used for each action in each state is selected as the target number of samples for each action in each state.

8. A supervised reinforcement learning device for scheduling the operation of a controlled system, characterized in that, include: The simulation calculation unit is used to sample the simulation results of each state-action pair of the controlled system according to a preset sampling number, and to determine the type of each action in each state, the Q factor of each state-action pair, the sample variance of the Q factor, and the total number of samples consumed based on the sampled simulation results. The supervisory unit calculates the importance of each non-observed optimal action in each state based on the Q-factor of each non-observed optimal action and the Q-factor of the observed optimal action in each state, and ranks the observed optimal actions and their importance in the top Δ. M Each unobserved optimal action is sent to the supervisor so that the supervisor can determine the optimal action. The allocation calculation unit is used to adjust the total amount of samples consumed. It uses the sampling data allocation algorithm to analyze the Q-factor sample variance of each state-action pair and the total amount of samples consumed to obtain the target sample size of each action in each state. The sampling data allocation algorithm includes the criteria for determining the number of samples for each type of action, which is used to progressively maximize the probability of correctly selecting the optimal action to observe. The supplementary sampling quantity determination unit is used to determine the supplementary sampling quantity for each action based on the target sample quantity for each action and the sample quantity consumed by each action. The supplementary sampling unit is used to perform supplementary sampling based on the number of supplementary samples for each action. The simulation calculation unit uses the simulation results of the supplementary sampling to re-execute the steps to determine the action type, and the supervision unit recalculates the importance of each non-observed optimal action in each state and provides feedback to the supervisor. The analysis unit is used to determine whether the total number of consumed samples is less than or equal to the preset total number of samples. If so, the allocation calculation unit will re-execute the steps to adjust the total number of consumed samples and the subsequent steps. Otherwise, the optimal action for each state will be output.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor of the computer device, it implements the method of any one of claims 1 to 7.

Images