Collaborative optimization method and apparatus for operation strategies of controlled system group

Abstract

A collaborative optimization method and apparatus for operation strategies of a controlled system group. The method comprises: on the basis of simulation results, determining the Q-factor sample variance of each action, performance differences between Q-factor fusion values of optimal actions and non-optimal actions, and the total number of consumed samples; using a sampling data allocation algorithm to analyze the quantities to obtain a target sample size of each action; on the basis of the target sample size of each action and the number of consumed samples of each action, determining a supplementary sampling quantity for each action; performing supplementary sampling on the basis of the supplementary sampling quantity, and using supplementary simulation results to re-determine the Q-factor sample variance of each action, the performance differences and the total number of consumed samples; and adjusting the total number of consumed samples, determining whether the total number of consumed samples is less than a preset total sampling quantity, if so, calculating the target sample size again and performing the steps subsequent thereto, and if not, outputting an optimal action.

Description

A method and device for cooperative optimization of operation strategy of a group of controlled systems TECHNICAL FIELD

[0001] The present application relates to the field of operation strategy optimization for a group of controlled systems (such as a group of energy storage power stations in a smart grid), and in particular to a method and device for cooperative optimization of operation strategy of a group of controlled systems. BACKGROUND

[0002] In the prior art, there are groups of controlled systems with multiple controlled systems, such as energy storage power stations in a smart grid. An energy storage power station is composed of a large number of energy storage devices, and the performance of a large power grid can be improved by reasonably scheduling the charging and discharging power. With a large number of new energy power generation connected to the power grid, the stable and economic operation of the energy storage power station becomes increasingly important. The existing operation strategy optimization of the energy storage power station involves a large state space (such as the power demand of each electrical device, the energy storage level of each energy storage device, and the power generation of each power generation device) and a large action space (such as the charging or discharging power of each energy storage device), which is difficult to accurately solve using reinforcement learning and other methods. Generally, a simplified model is used for solving, and the operation strategy is solved based on the data and information of a single energy storage power station. The energy storage power stations located in different places are relatively independent in the strategy optimization process, and the learning time of each energy storage power station is long, which consumes a large amount of training data. Even if the general reinforcement learning method is run for a long time, a good operation strategy cannot be found, and therefore a manually designed operation rule is needed to guide the operation of the power station.

[0003] SUMMARY

[0004] The present application is used to solve the problem of low efficiency and poor accuracy in the operation strategy optimization of a group of controlled systems with multiple controlled systems in the prior art. The similarity of decision-making problems of different energy storage power stations is not utilized, and the overall learning efficiency of the group of energy storage power stations is not improved by using transfer learning and other methods.

[0005] To solve the above technical problems, the present application provides a method for cooperative optimization of operation strategy of a group of controlled systems, the group of controlled systems comprising multiple controlled systems, the method comprising:

[0006] sampling each action simulation result of a target controlled system agent according to a preset sampling amount, determining a Q factor sample variance of each action, a performance difference of a Q factor fusion value of each optimal action and non-optimal action, and a total amount of consumed samples according to the sampling result, wherein the Q factor fusion value of each action is calculated by a Q factor fusion function, and the Q factor fusion function is a Q factor function of the target controlled system agent and its similar controlled system agent under the same state-action;

[0007] The Q-factor sample variance, performance difference and total consumed sample amount of each action are analyzed by using a sampling data distribution algorithm to obtain the target sample amount of each action, wherein the sampling data distribution algorithm comprises a determination criterion of the sampling amount of each type of action obtained according to the Q-factor fusion function, and is used for gradually maximizing the probability of correctly selecting the optimal action.

[0008] The supplementary sampling amount of each action is determined according to the target sample amount of each action and the consumed sample amount of each action.

[0009] Supplementary sampling is performed according to the supplementary sampling amount of each action, and the Q-factor sample variance, performance difference and total consumed sample amount of each action are re-determined by using the simulation result of the supplementary sampling.

[0010] The total consumed sample amount is adjusted, and it is judged whether the total consumed sample amount is less than the preset total sampling amount; if yes, the target sample amount and the subsequent steps are re-calculated, and if no, the optimal action is output.

[0011] The second aspect of the present application provides a running strategy cooperative optimization device of a controlled system group, wherein the controlled system group comprises a plurality of controlled systems, and the device comprises:

[0012] A sampling calculation unit is configured to sample the simulation result of each action of the target controlled system agent according to a preset sampling amount, and determine the Q-factor sample variance of each action, the performance difference of the Q-factor fusion value of each optimal action and non-optimal action, and the total consumed sample amount according to the sampling result, wherein the Q-factor fusion value of each action is calculated by a Q-factor fusion function, and the Q-factor fusion function is the Q-factor function of the target controlled system agent and its similar controlled system agent under the same state-action.

[0013] A target sampling amount determination unit is configured to analyze the Q-factor sample variance, performance difference and total consumed sample amount of each action by using a sampling data distribution algorithm to obtain the target sample amount of each action, wherein the sampling data distribution algorithm comprises a determination criterion of the sampling amount of each type of action obtained according to the Q-factor fusion function, and is used for gradually maximizing the probability of correctly selecting the optimal action.

[0014] A supplementary sampling amount determination unit is configured to determine the supplementary sampling amount of each action according to the target sample amount of each action and the consumed sample amount of each action.

[0015] A supplementary sampling unit is configured to perform supplementary sampling according to the supplementary sampling amount of each action, and re-determine the Q-factor sample variance, performance difference and total consumed sample amount of each action by using the simulation result of the supplementary sampling.

[0016] The analysis unit is configured to adjust the total amount of consumed samples, determine whether the total amount of consumed samples is less than a preset total sampling amount, if yes, calculate the target sampling amount by using the target sampling amount determination unit, and if no, output the optimal action.

[0017] The third aspect of the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the preceding embodiments when executing the computer program.

[0018] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by the processor of a computer device to implement the method of any one of the preceding embodiments.

[0019] The fifth aspect of the present application provides a computer program product, which comprises a computer program, wherein the computer program is executed by a computer device to implement the method of any one of the preceding embodiments.

[0020] The method and device for cooperative optimization of operation strategies of a group of controlled systems provided by the present application can improve the performance of the strategy learned from the shared environment, and can increase the difference between the Q factor of the optimal action and the Q factors of other actions, so as to facilitate the correct selection probability of the optimal action under the same observation uncertainty, and effectively improve the data efficiency of the strategy optimization process by cooperatively using the operation data of the controlled systems with the same action.

[0021] The sampling data distribution algorithm is established based on the Q factor fusion function of the target controlled system agent, the sampling amount of each action is distributed based on the Q factor fusion data and the sampling data distribution algorithm, and the probability of correctly selecting the optimal action can be maximized gradually. Meanwhile, the data efficiency of online reinforcement learning in the operation scheduling process of the controlled system can be improved by controlling the sampling amount, the time required for online decision making can be significantly improved, and the data usage can be reduced.

[0022] In order to make the above and other objects, features and advantages of the present application more apparent, the following will describe preferred embodiments of the present application in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only need to be some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor.

[0024] Fig. 1 shows a flow chart of the criterion determination process in the sampling data distribution algorithm of the embodiment of the present application;

[0025] Fig. 2 shows a flow chart of the operation strategy cooperative optimization method of the controlled system group of the embodiment of the present application;

[0026] Fig. 3 shows a flow chart of the target sample size calculation process of the embodiment of the present application;

[0027] Fig. 4 shows a structural diagram of the operation strategy cooperative optimization device of the controlled system group of the embodiment of the present application;

[0028] Fig. 5 shows a structural diagram of the computer equipment of the embodiment of the present application;

[0029] Fig. 6 shows a flow chart of the program design of the operation strategy cooperative optimization method of the controlled system of the embodiment of the present application.

[0030] Explanation of the accompanying drawings: 401, sampling calculation unit; 402, target sampling size determination unit; 403, supplementary sampling size determination unit; 404, supplementary sampling unit; 405, analysis unit; 502, computer equipment; 504, processor; 506, memory; 508, driving mechanism; 510, input / output module; 512, input device; 514, output device; 516, presentation device; 518, graphical user interface; 520, network interface; 522, communication link; 524, communication bus. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be described clearly and completely below with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0032] It should be noted that the terms "first", "second", and the like in the description and claims of the application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or apparatuses.

[0033] The specification provides method operation steps as described in the embodiments or flowcharts, but can include more or fewer operation steps based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of the many execution orders of the steps, and does not represent the only execution order. In actual system or device product execution, the method order shown in the embodiments or drawings can be executed in sequence or in parallel.

[0034] It should be noted that the data involved in the present application (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the acquisition, transmission, storage, use and processing of related data comply with relevant national and regional laws, regulations and standards.

[0035] It should be noted that in the embodiments of the present application, some software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0036] The controlled system group described in the present application refers to a cluster of multiple controlled systems distributed in different regions, and there is a certain similarity between the controlled systems, and all have control requirements, for example, a cluster of energy storage power stations, and the present application does not limit the specific controlled system cluster, and subsequent embodiments are all illustrated by taking the cluster of energy storage power stations as an example for ease of description.

[0037] In the prior art, the operation strategy optimization of the controlled system group with multiple controlled systems has the problems of low efficiency and poor precision, the similarity of decision problems of different energy storage power stations is not utilized, and the overall learning efficiency of the energy storage power station group is not improved by methods such as transfer learning.

[0038] To solve the above technical problems, the application shares the Q factor data of the same action of the target controlled system agent and its similar controlled system agent in the same state, that is, obtains the Q factor fusion function of the target controlled system agent (for fusing the Q factor of the target controlled system agent and its similar controlled system agent in the same state and action), and establishes a sampling data distribution algorithm through the Q factor fusion function, which can further widen the gap between the Q factor of the optimal action and other actions, so as to facilitate the improvement of the correct selection probability of the optimal action under the same observation uncertainty, and effectively improve the data efficiency of the strategy optimization process in cooperation with the operation data of the controlled system using the action consistency. Based on the Q factor fusion data and the sampling data distribution algorithm, the sampling amount of each action is distributed, which can gradually maximize the probability of correctly selecting the optimal action. At the same time, through the control of the sampling amount, the application can also improve the data efficiency of online reinforcement learning in the operation scheduling process of the controlled system agent, significantly improve the time required for online decision-making, and reduce the data usage.

[0039] Specifically, before the sampling data distribution algorithm is executed, the sampling data distribution algorithm is established, as shown in FIG. 1, and the sampling data distribution algorithm establishment process includes:

[0040] Step 101, according to the device information, system state, control action, and optimization target information of each controlled system, the performance function of the control strategy of each controlled system and the action sampling amount constraint condition are established.

[0041] Step 102, the performance function of the control strategy of each controlled system is converted to obtain the Q factor fusion function of each controlled system agent.

[0042] Step 103, according to the Q factor fusion function of each controlled system agent, the objective function of the approximate optimal action selection probability function is established.

[0043] Step 104, according to the objective function of the approximate optimal action selection probability function and the action sampling amount constraint condition, the criterion of the sampling data distribution algorithm is determined.

[0044] In detail, the device information, system state, control action, and optimization target information in step 101 can be determined according to the specific circumstances of the controlled system, and the application does not limit the specific parameters contained therein. The performance function of the control strategy is established according to the device information, system state, control action, and optimization target. Each controlled system is responsible for the related strategy optimization by an agent, and the performance function of the control strategy of each controlled system can be represented by the following formula:

[0045] Wherein, J represents the performance function, s t represents the state of the controlled system at time t, d i (s t) represents the policy used by the agent i of the controlled system to make decisions, c represents the objective function, and T represents the length of the simulation period.

[0046] The objective of the agent i of the controlled system is to find a policy d i * , which represents the following:

[0047] The Q-factor (also known as the action-value function) of the agent i of the controlled system can be represented by the following formula:

[0048] wherein, represents the Q-factor of the agent i of the controlled system at state s and action a; d i represents the policy; c represents the objective function; P i (s'|s,a) represents the probability of the agent i of the controlled system transitioning to state s' from state s and action a; represents the Q-factor of the agent i of the controlled system at state s' and policy d i (s'). In actual implementation, in order to simplify the description, the Q-factor (s,a) can also be represented as Q i (s,a).

[0049] During the learning process, the agent i can obtain a policy d. For an action a, the agent i can run some simulations to collect samples of Q i (s,a). The collected samples are denoted as represents the number of samples collected by the agent i for action a. In actual implementation, another agent j can have done similar work and has collected sample data represents the number of samples collected by the agent j for action a. The agent j hopes to share these data with the agent i. In order to simplify the discussion, the following assumptions are made:

[0050] Assume that the same state space S i =S j . The extension of the conclusion to cases other than the above assumption will be discussed later.

[0051] Now the agent i can generate an estimate based on the local samples and the shared samples

[0052] wherein, ​Qf(s,a) represents the Q-factor fusion function of the controlled system agent i, i.e. the function of the controlled system agent i and the Q-factor fusion of its similar controlled system agent j; s represents the state; a represents the action; f represents the fusion function, for example, the sum operation on the elements contained in f; Qf(s,a) represents the Q-factor observation mean value of the controlled system agent i at state s, action a and sample quantity k; Qf(s,a) represents the Q-factor observation mean value of the controlled system agent j at state s and action a sample quantity k; k represents the sample quantity, Qf(s,a) represents the sample quantity of the controlled system agent i at action a, Qf(s,a) represents the sample quantity of the controlled system agent j at action a.

[0053] Then different actions can be compared, and the action with the smallest estimated Q-factor is selected, i.e.

[0054] We want to adjust the sampling budget allocated to the entire action space to minimize the Q-factor of the current state, i.e.

[0055] where N is the total sample quantity, representing the total sampling quantity of agent i in this iteration. The problem represented by the above formula is simply denoted as P1. When the system reaches the next state, the next iteration begins. Under the convenience assumption of a policy-controlled Markov chain, it can be proved that when there is infinite sampling (N→∞), and the iteration continues, the estimated optimal action at each state converges to the same action as the optimal policy. We want to get more details of f, i.e. how the samples shared by agent j are used in agent i, and how to allocate N to speed up this learning process.

[0056] For the above formula (4), we consider a specific function f that combines samples from agent i and agent j to generate the estimate of and by directly adding the sample means

[0057] Thus we have:

[0058] We now take the Bayesian view that the observed samples are given and fixed. But the true value Q i (s,a),a∈A i is a random variable subject to the posterior distribution. When the sample quantity is large, the central limit theorem proves that Q i (s,a) and The difference between them is a Gaussian random variable. Given a Gaussian prior and Gaussian observation noise, the posterior distribution is also Gaussian. Therefore, we can make the following assumption:

[0059] For all s∈S i ,a∈A i , the prior distribution of Q i (s,a) is as follows

[0060] The above assumption basically means that we know nothing about Q i (s,a) except that each value has the same probability. Given the above assumption, the observation and the posterior distribution of Q ij (s,a) is where:

[0061] We are interested in maximizing the probability of choosing the optimal action, i.e.,

[0062] By simplifying b(s) to b in the above equation, we get the following problem, denoted as P2:

[0063] That is, after fusing the information of agents i and j, the probability that the optimal action b estimated by agent i at state s is the true optimal action. N a i represents the number of samples assigned by agent a to action a. In the case of online running, the sum of the computational amount assigned by agent i at the current state on all actions should be the given total computational amount N.

[0064] The above problem P2 is relatively easier to solve than the previous problem P1, because it only needs to evaluate the probability that the observed optimal action at the current state is the true optimal action, while the original problem model needs to evaluate the average performance of a given strategy.

[0065] Bonferroni's inequality states that:

[0066] where Y i ,i=1,2,...,k are events. Next, we can get:

[0067] The right side of the above equation can be called the approximate optimal action selection probability function (APCS),

[0068] From this, we can get the objective function of the approximate optimal action selection probability function represented by the following formula:

[0069] The above problem can be simply recorded as P3.

[0070] where APCS represents the approximate optimal action selection probability function, which is a lower bound of PCS; b represents the optimal action; a represents the non-optimal action; Q ij (s,b) represents the Q-factor fusion value of the optimal action b of the controlled system agent i and the controlled system agent j in the state s; Q ij (s,a) represents the Q-factor fusion value of the non-optimal action a of the controlled system agent i and the controlled system agent j in the state s.

[0071] Problem P3 is easier to solve than problem P2, because the original joint probability is decomposed into the performance comparison between the observed optimal action b and other actions a in the current state. This decomposition form can provide a basis for using parallel computing. However, as will be seen later, the present patent provides a faster solution formula.

[0072] In step 104, the criterion of the sampling data allocation algorithm is obtained for indicating:

[0073] After the data fusion of the controlled system and its similar controlled system agent, the greater the signal-to-noise ratio of the non-observed optimal action in the same state s, the greater the sampling amount of the non-observed optimal action, wherein the noise in the signal-to-noise ratio refers to the sample standard deviation of the Q-factor of the non-observed optimal action, and the information in the signal-to-noise ratio refers to the performance difference of the Q-factor fusion values of the observed optimal action and the non-observed optimal action;

[0074] The sampling amount of the observed optimal action in the same state should be greater than that of other non-observed optimal actions.

[0075] Specifically, according to the objective function of the approximate optimal action selection probability function and the action sampling amount constraint condition, the process of determining the criterion of the sampling data allocation algorithm includes:

[0076] First, the Lagrange equation is introduced:

[0077] where λ is the Lagrange multiplier, λ≥0, and then according to the Karush-Kuhn-Tucker (KKT) condition (assuming that the KKT regular condition is established), we have the necessary condition of the local optimal solution: λ≥0 (20)

[0078] For the non-observed optimal action a≠b, we have:

[0079] For the first term on the right side of equation (22), we can further get:

[0080] Please note Where:

[0081] So we can get:

[0082] Then we can get:

[0083] Where,

[0084] Further we can get:

[0085] According to similar analysis, for the observed optimal action b, we have:

[0086] Next, we can analyze the relationship between

[0087] First, we analyze two non-observed optimal actions p and q, p, q≠b, according to equation (29), we can get:

[0088] Combined with equations (31) and (32), we can get:

[0089] When the sample size N tends to infinity, we can reasonably assume that the observed optimal action distribution has more sample size, that is So we can get:

[0090] Similarly, we can get:

[0091] Then we can get:

[0092] Further, equation (36) can be adjusted as:

[0093] Next, take the logarithm of both sides of equation (37), we can get:

[0094] We can see that when N grows, the logarithmic term grows significantly slower than other terms, so we can ignore it from the above equation, thus we can get: ​

[0095] Therefore, we can conclude that:

[0096] Secondly, analysis and The relationship between a≠b can be obtained according to formula (29):

[0097] Substituting formula (41) into formula (30), we get:

[0098] Then we can get:

[0099] After readjusting formula (43), we can obtain:

[0100] In summary, based on the above analysis, the criteria for the sampling data allocation algorithm can be summarized as follows:

[0101] The sampling data allocation algorithm includes:

[0102] Criterion 1: For unobserved optimal actions p and q, q ≠ p.

[0103] in, This represents the sample size of the controlled system agent i when it is not observing the optimal action p; σ represents the sample size of agent i in the controlled system when performing the non-observed optimal action q; ip σ represents the sample standard deviation of the Q-factor of the controlled system agent i when performing the non-observed optimal action p; iq δ represents the sample standard deviation of the Q-factor of the controlled system agent i when performing the non-observed optimal action q; p δ represents the performance difference between the optimal action b and the non-optimal action p of controlled system agent i and its similar controlled system agent j; q This represents the performance difference between the optimal action b and the non-optimal action q of the controlled system agent i and its similar controlled system agent j.

[0104] Criterion 2: For observing the optimal action b and the non-optimal action a,

[0105] in, σ represents the sample size of agent i in the controlled system when observing the optimal action b; ib The sample standard deviation of the Q factor represents the Q factor of the controlled system agent i when observing the optimal action b. σ represents the sample size of the controlled system agent i when it performs a non-optimal action a; iaIt represents the sample standard deviation of the Q factor of the controlled system agent i when it performs a non-optimal action a.

[0106] Q ij This is a specific way of fusing information from agents i and j. Specifically, it involves directly adding the Q-factors of the same action in the same state, estimated by both agents using their respective data. The direct advantage of this approach is that, as long as the optimal actions of the two agents in the same state are identical, this method will further widen the gap between the Q-factors of the optimal action and other actions. This is beneficial for improving the probability of correct selection under the same observational uncertainty. This property can be described as optimal action consistency, i.e., the optimal actions of the two agents in the same state are identical.

[0107] If the state spaces of the subproblems of two agents are not completely identical, data sharing between the two agents can be achieved by normalization or by taking their intersection, selecting a state that satisfies optimal action consistency. In this mode, the agents sharing data may change depending on the state. This aligns well with the characteristics of actual energy storage power station cluster operations, where data is shared only between energy storage power stations that satisfy optimal action consistency under similar states, or by normalizing the data of each energy storage power station according to its load level, energy storage capacity level, and daily power generation level, and then sharing data between energy storage power stations that satisfy optimal action consistency under similar states. Criterion 1 states that after agent i fuses the data of agent j, for two actions that are not observed optimally, the sampling amount of different actions should be proportional to the square of the "noise-to-information ratio" (NNR). Here, noise refers to the sample standard deviation of a single observation of the Q-factor of an action, and information refers to the difference in the sample mean of the Q-factor of the observed optimal action from other actions. Note that fusing the samples of agent j using the aforementioned method further increases this "information" compared to before fusion, which is equivalent to further improving the speed at which the observation order of the superiority or inferiority of two actions converges to the true order.

[0108] Criterion 2 states that for the observed optimal action b, the sample size it should receive should be approximately equal to the sum of the sample sizes of all other actions after normalization according to their respective standard deviations. As a result of this allocation method, with continued sampling, the observed optimal action eventually receives a much larger sample size than other actions. In other words, with the same total sample size, after allocating the sample size using criteria 1 and 2, the observed optimal action converges to the true optimal action the fastest, and fusing data from agent j can further improve this convergence rate.

[0109] According to formula (24), we can obtain:

[0110] The first and second terms on the right side of the above formula are respectively and That is:

[0111] Then we can get:

[0112] The second term on the right side of the above formula quantifies the influence of agent j From this, it can be proved that if there is no influence of agent j, the above criterion becomes:

[0113] This means that is the observation performance difference between the observation optimal action b and the non-observation optimal action a determined by agent j. If agents i and j share the same optimal action in this state, it is indeed useful.

[0114] On the basis of the above sampling data allocation algorithm, a running strategy cooperative optimization method of the group of controlled systems can be executed. Specifically, according to the control requirement, a target controlled system is first determined from the group of controlled systems; then a controlled system similar to the target controlled system, i.e., a controlled system satisfying state-action consistency, is found from the group of controlled systems. The process of finding a controlled system similar to the target controlled system can refer to the prior art, which will not be described in detail here. Next, as shown in FIG. 2, the following steps are executed:

[0115] In step 201, the simulation results of each action of the target controlled system agent are sampled according to a preset sampling amount, and the Q factor sample variance of each action, the performance difference of the Q factor fusion value of each optimal action and non-optimal action, and the total amount of consumed samples are determined according to the sampling results.

[0116] The Q factor fusion value of each action is calculated by a Q factor fusion function, which is the Q factor function of the target controlled system agent and its similar controlled system agent under the same state-action, and is used to realize the Q factor data sharing between the target controlled system and its similar controlled system.

[0117] The performance difference of the Q factor fusion value of each optimal action and non-optimal action refers to the difference between the Q factor fusion value of the optimal action and the Q factor fusion value of the non-optimal action of the target controlled system agent in state s. In some embodiments, the performance difference can be expressed by the following formula:

[0118] Where a is a non-observation optimal action; b represents an observation optimal action; and s represents the current state. The Q-factor observation fusion mean value of the controlled system agent i and its similar controlled system agent j in state s and non-observation optimal action a; The Q-factor observation fusion mean value of the controlled system agent i and its similar controlled system agent j in state s and observation optimal action b.

[0119] In this step, the action types include optimal actions and non-optimal actions, and the Q-factor sample variances of various actions are calculated from the sampled Q-factors of various actions. The specific calculation method can refer to the existing variance calculation formula.

[0120] The action in this step can be determined according to the action space. The preset sampling quantity (N0) can be set according to the preset total sampling quantity N, the preset cycle number, the action space, and the newly added sampling quantity Δ. Generally, the preset cycle number is greater than 10, and the preset total sampling quantity N is much greater than the product of the preset sampling quantity N0 and the action quantity in the action space.

[0121] In this step, the total consumed sample quantity is the product of the preset sampling quantity and the action quantity in the action space plus the newly added sampling quantity, and the corresponding calculation formula is: Wherein, Total sampling quantity, n0 represents the preset sampling quantity, |A i | represents the sampling quantity in the action space of the target controlled system i, and Δ represents the newly added sampling quantity.

[0122] In this step, the action can directly act on the simulation model (such as a digital twin model) of the controlled system to determine the simulation results of each action. In specific implementation, the execution results of the preset sampling quantity can also be obtained from the historical data according to the state and the action, and the obtained execution results are taken as the simulation results.

[0123] In step 202, the Q-factor sample variances, performance differences, and total consumed sample quantities of various actions are analyzed by using a sampling data allocation algorithm to obtain target sample quantities of various actions.

[0124] The sampling data allocation algorithm includes sampling quantity determination criteria for various actions, which are obtained according to the Q-factor fusion function and are used to gradually maximize the probability of correctly selecting the optimal action. The specific contents included in the criteria are described in the foregoing embodiments, which will not be described here.

[0125] The specific implementation process of this step can be referred to in the subsequent embodiments, which will not be described here in detail.

[0126] In step 203, the supplementary sampling quantities of various actions are determined according to the target sample quantities of various actions and the consumed sample quantities of various actions.

[0127] In this step, the supplementary sampling quantities of various actions are obtained by subtracting the consumed sample quantities of various actions from the target sample quantities of various actions.

[0128] Step 204, according to the supplementary sampling amount of each action, supplementary sampling is carried out, and the Q factor sample variance, performance difference and total consumed sample amount of each action are re-determined by using the simulation result of supplementary sampling.

[0129] In this step, the Q factor sample variance and performance difference calculation process can refer to step 201, which will not be described here. The total consumed sample amount in this step is the sum of the real sampling amount of the target controlled system at each action.

[0130] Step 205, adjust the total consumed sample amount, and judge whether the total consumed sample amount is less than the preset total sampling amount, if yes, re-calculate the target sample amount and the subsequent steps, if not, output the optimal action.

[0131] When this step is implemented, the total consumed sample amount is increased by the new sampling amount Δ to obtain the adjusted total consumed sample amount. The action sampling amount constraint condition is the preset total sampling amount N, which can be set jointly according to the initial value of the preset sampling amount in step 201 and the action space, as long as the preset number of repeated cycles of steps 201 to 205 can be met.

[0132] In this step, if the total consumed sample amount is less than or equal to the action sampling amount constraint condition, it means that the consumed sample amount after the next iteration does not reach or just equals the preset total sampling amount, and the feasibility of the action can be continued to be observed. If the total consumed sample amount is greater than the action sampling amount constraint condition, it means that the consumed sample amount after the next iteration exceeds the preset total sampling amount, and the feasibility of the action cannot be observed.

[0133] In a specific embodiment, the flow of the algorithm of the present embodiment is shown in FIG. 6. In FIG. 6, the controlled system agent j is similar to the target controlled system agent i, n0 is the preset sampling amount, a is the action, is the action space. δ a represents the performance difference between the Q factor fusion value of the optimal action observed by the target controlled system agent i and its similar controlled system agent j in the state s and the Q factor fusion value of the non-observed optimal action a, represents the Q factor sample variance of the action a, represents the consumed sample amount, and Δ represents the new sampling amount. N represents the preset total sample amount, and α a represents the sample amount ratio relationship that the action a should obtain, represents the sample amount that the action a should obtain, represents the consumed sample amount of the action a, represents the target sample amount of the action a, represents the supplementary sampling amount of the action a.

[0134] For each state s, the optimal action b(s) is the action in the action space that can make the Q-factor fusion value minimum, that is, the action satisfying .

[0135] The embodiment pre-designs the Q-factor fusion function of the target controlled system agent, the Q-factor fusion function of the target controlled system agent shares / fuses the Q-factor data of the target controlled system and its similar controlled system, can further narrow the gap between the optimal action and other actions in the Q-factor, and is beneficial to improving the correct selection probability of the optimal action under the same observation uncertainty, cooperates with the operation data of the controlled system with the action consistency, and effectively improves the data efficiency of the strategy optimization process.

[0136] The sampling data distribution algorithm is established based on the Q-factor fusion function of the target controlled system agent, the sampling amount of each action is distributed based on the Q-factor fusion data and the sampling data distribution algorithm, and the probability of correctly selecting the optimal action can be maximized gradually. Meanwhile, the data efficiency of online reinforcement learning in the operation and scheduling process of the controlled system can be improved by controlling the sampling amount, the time required for online decision is significantly improved, and the data usage is reduced.

[0137] In an embodiment of the present application, as shown in FIG. 3, the step 204 analyzes the Q-factor sample variance, performance difference and total amount of consumed samples of each action by using the sampling data distribution algorithm to obtain the target sample amount of each action, including:

[0138] Step 301: According to the sampling data distribution algorithm, the Q-factor sample variance and performance difference of each action are analyzed to obtain the sampling ratio of each action.

[0139] Step 302: The sample amount obtained by the product of the sampling ratio of each action and the total amount of consumed samples is obtained, and the corresponding calculation formula is:

[0140] Step 303: The larger one of the sample amount obtained by each action and the actual used sample amount of each action is selected as the target sample amount of each action. In specific implementation, the corresponding calculation formula of this step is:

[0141] Wherein, indicates the target sampling amount of action a, indicates the sample amount obtained by action a, indicates the actual used sample amount of action a.

[0142] Based on the same inventive concept, the application further provides a running strategy coordination optimization device for a group of controlled systems, as described in the following embodiments. Since the running strategy coordination optimization device for a group of controlled systems solves problems in a similar principle to the running strategy coordination optimization method for a group of controlled systems, the implementation of the running strategy coordination optimization device for a group of controlled systems can be referred to the running strategy coordination optimization method for a group of controlled systems, and the repeated parts will not be described herein.

[0143] Specifically, as shown in FIG. 4, the running strategy coordination optimization device for a group of controlled systems comprises:

[0144] a sampling calculation unit 401 configured to sample each action simulation result of a target controlled system intelligent agent according to a preset sampling amount, and determine a Q factor sample variance of each action, a performance difference between a Q factor fusion value of each optimal action and a non-optimal action, and a total amount of consumed samples.

[0145] The Q factor fusion value of each action is calculated by a Q factor fusion function, and the Q factor fusion function is a Q factor function of the target controlled system intelligent agent and its similar controlled system intelligent agent under the same state-action.

[0146] a target sampling amount determination unit 402 configured to analyze the Q factor sample variance of each action, the performance difference, and the total amount of consumed samples by using a sampling data distribution algorithm to obtain a target sample amount of each action.

[0147] The sampling data distribution algorithm comprises a determination criterion of a sampling amount of each type of action obtained by analyzing the Q factor fusion function, and is used to gradually maximize the probability of correctly selecting an optimal action.

[0148] a supplementary sampling amount determination unit 403 configured to determine a supplementary sampling amount of each action according to the target sample amount of each action and the consumed sample amount of each action.

[0149] a supplementary sampling unit 404 configured to perform supplementary sampling according to the supplementary sampling amount of each action, and re-determine the Q factor sample variance of each action, the performance difference, and the total amount of consumed samples by using the simulation result of the supplementary sampling.

[0150] an analysis unit 405 configured to adjust the total amount of consumed samples, and determine whether the total amount of consumed samples is less than a preset total sampling amount. If yes, the target sample amount is recalculated by using the target sampling amount determination unit, and if no, an optimal action is output.

[0151] The embodiment pre-designs the Q factor fusion function of the target controlled system agent, the Q factor fusion function of the target controlled system agent shares / fuses the Q factor data of the target controlled system and its similar controlled system, the target controlled system and its similar controlled system refer to the controlled systems with the same optimal action of the controlled system agent in the same state, the difference between the Q factor of the optimal action and other actions can be further enlarged, so as to improve the probability of correct selection of the optimal action under the same observation uncertainty, and the operation data of the controlled system with the same action consistency are used in cooperation, so that the data efficiency of the strategy optimization process is effectively improved.

[0152] The sampling data distribution algorithm is established based on the Q factor fusion function of the target controlled system agent, the sampling amount of each action is distributed based on the Q factor fusion data and the sampling data distribution algorithm, and the probability of correct selection of the optimal action can be gradually maximized. Meanwhile, through the control of the sampling amount, the data efficiency of online reinforcement learning in the operation and scheduling process of the controlled system can be improved, the time required for online decision making is significantly improved, and the data usage is reduced.

[0153] When the application is applied to the operation strategy cooperative optimization of the energy storage power station group, the data efficiency of the operation and scheduling strategy optimization process of the energy storage power station group can be improved, the performance of the obtained strategy is significantly improved, and the data usage is reduced.

[0154] In an embodiment of the application, a computer device is also provided, as shown in FIG. 5, the computer device 502 can include one or more processors 504, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 502 can also include any memory 506 for storing any kind of information such as code, settings, data, etc. Without limitation, for example, the memory 506 can include any one or a combination of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory can store information using any technology. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 502. In one case, the computer device 502 can perform any operation of the associated instructions when the processor 504 executes the associated instructions stored in any memory or combination of memories. The computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0155] The computer device 502 can also include an input / output module 510 (I / O) for receiving input (via input device 512) and for providing output (via output device 514). One particular output mechanism can include a presentation device 516 and associated graphical user interface 518 (GUI). In other embodiments, the input / output module 510 (I / O), input device 512, and output device 514 can not be included, and the computer device 502 can be only one of a network of computer devices. The computer device 502 can also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the above-described components so that each component can communicate with each other component.

[0156] The communication links 522 can be implemented in any manner, such as through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication links 522 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0157] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is run by a processor to execute the steps of the above method.

[0158] The embodiments of the present application also provide a computer readable instruction, wherein when the processor executes the instruction, the program in the processor executes the method of any of the above embodiments.

[0159] It should be understood that the size of the sequence number of each process described above does not mean the order of execution in various embodiments of the present application. The execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0160] It should also be understood that in the embodiments of the present application, the term "and / or" is only a description of the association relationship between the associated objects, which means that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects before and after it.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0163] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. For example, the device embodiments described above are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0164] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0165] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0166] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the whole or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0167] The principles and implementation manners of the present application are described in the specific embodiments in the present application. The above embodiment descriptions are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed. In summary, the content of the present description should not be understood as a limitation of the present application.

Claims

1. A method for collaborative optimization of operation strategies of a group of controlled systems, characterized in that, The controlled system group comprises a plurality of controlled systems, and the method comprises: According to the preset sampling amount, the simulation results of each action of the target controlled system agent are sampled, and the Q factor sample variance of each action, the performance difference of the Q factor fusion value of each optimal action and non-optimal action, and the total amount of consumed samples are determined according to the sampling results, wherein the Q factor fusion value of each action is calculated by a Q factor fusion function, and the Q factor fusion function is the Q factor function of the target controlled system agent and its similar controlled system agent under the same state-action. The Q factor sample variance, the performance difference, and the total amount of consumed samples of each action are analyzed by using a sampling data allocation algorithm to obtain the target sample amount of each action, wherein the sampling data allocation algorithm comprises a determination criterion of the sampling amount of each type of action obtained by analyzing the Q factor fusion function, and is used for gradually maximizing the probability of correctly selecting the optimal action. According to the target sample amount of each action and the consumed sample amount of each action, the supplementary sampling amount of each action is determined. According to the supplementary sampling amount of each action, supplementary sampling is performed, and the Q factor sample variance, the performance difference, and the total amount of consumed samples of each action are re-determined by using the simulation results of the supplementary sampling. The total amount of consumed samples is adjusted, it is judged whether the total amount of consumed samples is less than the preset total sampling amount, if yes, the target sample amount and the subsequent steps are re-calculated, and if no, the optimal action is output.

2. The method of claim 1, wherein, The sampling data allocation algorithm determination process comprises: According to the device information, system state, control action, and optimization target information of each controlled system, a performance function and an action sampling amount constraint condition of each controlled system control strategy are established; The performance function of each controlled system control strategy is converted to obtain the Q factor fusion function of each controlled system agent; According to the Q factor fusion function of each controlled system agent, a target function of an approximate optimal action selection probability function is established; According to the target function of the approximate optimal action selection probability function and the action sampling amount constraint condition, the criterion of the sampling data allocation algorithm is determined.

3. The method of claim 2, wherein, The performance function of each controlled system control strategy is converted to obtain the Q factor fusion function of each controlled system agent, which comprises: The Q-factor fusion function of the agent of the controlled system is expressed by the following formula: wherein Qij(s, a) represents a fused value of the Q-factor of the controlled system agent i and the controlled system agent j in the state s and the action a; s represents a state; and a represents an action. Q(s, a) = E[r | s, a] + γmaxa' Q(s', a') Qs (s, a) represents the observed mean of the Q-factor of the controlled system agent j at state s and action a; S represents the state space; A i A represents the action space, the controlled system agent i is similar to the controlled system agent j.

4. The method of claim 2, wherein, According to the Q factor fusion function of each controlled system agent, a target function of an approximate optimal action selection probability function is established, which comprises: The objective function for the approximate optimal action selection probability function is expressed using the following equation: wherein APCS represents an approximate optimal action selection probability function; b represents an optimal action; a represents a non-optimal action; Q ij (s,b) represents a Q-factor fusion value of the controlled system agent i and the controlled system agent j in the state s and the optimal action b; Q ij (s,a) represents a Q-factor fusion value of the controlled system agent i and the controlled system agent j in the state s and the non-optimal action a, and the controlled system agent i and the controlled system agent j are similar.

5. The method of claim 1, wherein, The sampling data allocation algorithm is used to indicate that: After the data fusion of the controlled system agent and its similar controlled system agent, the larger the signal-to-noise ratio of the unobserved optimal action under the same state s, the larger the sampling amount of the unobserved optimal action, wherein the noise in the signal-to-noise ratio refers to the sample standard deviation of the Q factor of the unobserved optimal action, and the information in the signal-to-noise ratio refers to the performance difference of the Q factor fusion value of the observed optimal action and the unobserved optimal action; The sampling amount of the observed optimal action under the same state should be more than that of other unobserved optimal actions.

6. The method of claim 5, wherein, The sampling data allocation algorithm comprises: Criterion 1 : for non-observed optimal actions p and q, q≠p, wherein, represents the number of samples of the controlled system agent i at the unobserved optimal action p; N (q) represents the number of samples of the controlled system agent i at the unobserved optimal action q; σ ip N (q) represents the number of samples of the controlled system agent i at the unobserved optimal action Q-factor sample standard deviation of the agent i of the controlled system at the non-optimal action q; δ iq Q-factor sample standard deviation of the agent i of the controlled system at the non-optimal action q; δ p Performance difference of the Q-factor fusion value of the optimal action b and the non-optimal action p of the agent i of the controlled system and the similar agent j of the controlled system; δ q Performance difference of the Q-factor fusion value of the optimal action b and the non-optimal action q of the agent i of the controlled system and the similar agent j of the controlled system; Criterion 2: For an observed optimal action b and a non-optimal action a, wherein denotes the sample size of the controlled system agent i when observing the optimal action b; σ ib denotes the sample standard deviation of the Q-factor of the controlled system agent i when observing the optimal action b; S (i, a) represents the sample size of the controlled system agent i at the non-optimal action a; σ ia S (i, a) represents the sample size of the controlled system agent i at the non-optimal action a; σ 7. The method of claim 1, wherein, The Q factor sample variance, the performance difference, and the total amount of consumed samples of each action are analyzed by using a sampling data allocation algorithm to obtain the target sample amount of each action, which comprises: According to the sampling data allocation algorithm, the Q factor sample variance and the performance difference of each action are analyzed to obtain the sampling proportion of each action; The Q factor sample variance and the performance difference of each action are analyzed according to the sampling data allocation algorithm to obtain the sampling proportion of each action. The sample quantity that each action should obtain is obtained by multiplying the sampling proportion of each action and the total quantity of consumed samples; The greater of the sample quantity that each action should obtain and the actual used sample quantity of each action is selected as the target sample quantity of each action.

8. A device for collaborative optimization of the operation strategy of a controlled system group, characterized in that, The controlled system group includes a plurality of controlled systems, and the device includes: A sampling calculation unit is configured to sample each action simulation result of a target controlled system agent according to a preset sampling quantity, and determine a Q factor sample variance of each action, a performance difference between a Q factor fusion value of each optimal action and a non-optimal action, and a total quantity of consumed samples according to the sampling result, wherein the Q factor fusion value of each action is calculated by a Q factor fusion function, and the Q factor fusion function is a Q factor function of the target controlled system agent and a similar controlled system agent thereof under a same state-action; A target sampling quantity determination unit is configured to analyze the Q factor sample variance of each action, the performance difference, and the total quantity of consumed samples by using a sampling data distribution algorithm to obtain a target sample quantity of each action, wherein the sampling data distribution algorithm includes a determination criterion of each action sampling quantity obtained according to the Q factor fusion function, and is configured to gradually maximize a probability of correctly selecting an optimal action; A supplementary sampling quantity determination unit is configured to determine a supplementary sampling quantity of each action according to the target sample quantity of each action and the consumed sample quantity of each action; A supplementary sampling unit is configured to perform supplementary sampling according to the supplementary sampling quantity of each action, and to redetermine the Q factor sample variance of each action, the performance difference, and the total quantity of consumed samples by using the simulation result of the supplementary sampling; An analysis unit is configured to adjust the total quantity of consumed samples, and to determine whether the total quantity of consumed samples is less than a preset total sampling quantity, if yes, to redetermine the target sample quantity by using the target sampling quantity determination unit, and if no, to output an optimal action.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor of the computer device to implement the method in any one of claims 1 to 7.

Images