Man-machine hybrid enhanced power system intelligent scheduling method and related equipment

Abstract

The invention relates to the technical field of intelligent scheduling, in particular to a man-machine hybrid enhanced power system intelligent scheduling method and related equipment, and the method comprises the steps: obtaining a demonstration data set, inputting the demonstration data set into a neural network for supervised learning training, and obtaining a behavior cloning agent; executing a scheduling task of the power system through a decision action output by the behavior cloning agent to obtain a plurality of moving trajectories of the power system, and obtaining a sequence of each moving trajectory by a dispatcher; deducing a reward function through the sorted moving tracks, executing an optimization step of reinforcement learning on the behavioral cloning agent based on the reward function to obtain a hybrid enhancement strategy, and performing power dispatching based on the hybrid enhancement strategy; according to the method, a more efficient and safer decision can be realized in the intra-day optimization scheduling problem of the power system.

Description

Technical Field

[0001] This invention relates to the field of intelligent dispatching technology, and in particular to a human-machine hybrid enhanced intelligent dispatching method and related equipment for power systems. Background Technology

[0002] Driven by the "dual carbon" goals, the new power system exhibits characteristics such as high-proportion renewable energy penetration, multi-type flexible load integration, and deep interaction between power generation, grid, load, and storage. As a core business of the power system, dispatching and operation urgently require better solutions to improve decision-making quality in order to ensure system safety, balance power generation with load, and improve system operational economy. In the context of new power system construction, dispatching decision-making tasks and operational constraints exhibit multi-dimensional spatiotemporal coupling characteristics, accompanied by strong randomness brought about by a very high proportion of renewable energy. This significantly increases the difficulty of coordinated operation of power generation, grid, load, and storage, making traditional deterministic modeling methods inefficient in solving these problems. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a human-machine hybrid enhanced intelligent dispatching method and related equipment for power systems, so as to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] On one hand, embodiments of the present invention provide a human-machine hybrid enhanced intelligent dispatching method for power systems, the method comprising the following steps: Obtain a demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategies; The demonstration dataset is input into the first neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. By executing the decision-making actions output by the behavioral clone agent, the scheduling tasks of the power system are executed, multiple operating trajectories of the power system are obtained, and the dispatcher's ranking of each operating trajectory is obtained. The reward function is derived by sorting the running trajectories. Based on the reward function, the behavior clone agent is subjected to reinforcement learning optimization steps to obtain a hybrid reinforcement strategy. Power scheduling is then performed based on the hybrid reinforcement strategy.

[0005] Optionally, the behavioral clone agent is represented as: (1); in, The policy represented by the output of the behavioral clone agent. express The parameters, For the example dataset, s represents the observation state and a represents the decision action.

[0006] Optionally, the process of executing power system scheduling tasks through the decision-making actions output by the behavioral clone agent to obtain multiple operating trajectories of the power system and acquiring the dispatcher's ranking of each operating trajectory includes: In the initial stage of the operation of the behavioral clone agent, the behavioral clone agent outputs decision actions corresponding to the observed state of the power system, and uses the decision actions to execute the scheduling tasks of the power system to determine whether the power system is operating safely. Under the premise of ensuring the safe operation of the power system, random noise with different noise ratios is injected into the behavioral clone agent. The behavioral clone agent with injected random noise outputs decision actions corresponding to the observed state of the power system. The decision actions are used to execute the scheduling task of the power system, and the operating trajectory of the power system is obtained. All operating trajectories are formed into a trajectory set. Pair the running trajectories in the trajectory set into pairs, obtain the priority ranking of the two running trajectories in each pair by the scheduler, and obtain the ranking of each running trajectory.

[0007] Optionally, the trajectory set is represented as: ; in, Let U be the nth trajectory, U be a uniform distribution, and η be the noise ratio of the injected behavioral clone agent. The strategy after injecting noise η.

[0008] Optionally, deriving the reward function from the sorted running trajectories includes: The reward function is obtained by training the second neural network using a generalized loss function based on the sorted running trajectory.

[0009] Optionally, the generalized loss function is expressed as: ; in, This represents the reward corresponding to the observed state s. and Let i and j represent the i-th and j-th running trajectories, respectively.

[0010] Optionally, the formulas for action policy and Q-value updates in the reinforcement learning are as follows: ; ; in, π * represents the optimal action strategy; r ( s t ,a t () is the immediate reward for the state-action pair. It is a moment t The observation status, It is a moment t The action; For expectation operators; γ It is a discount factor; Representing state S t+1 Value functions; H [·] represents the entropy of the action set; α Weights for temperature coefficients; Represents state-action pairs ( s t , a t )of Q value; The update formulas for the actor, evaluator, and target evaluator in the reinforcement learning process are as follows: ; (11); ; In the formula, This represents the residual required for policy network updates; Indicates action strategy; ω for Q Network parameters, ϖ For the goal Q Network parameters, θ For policy network parameters, λ This is a soft update coefficient; r ( s t , a t () is the immediate reward for the state-action pair. It is a moment t The system observation status, It is a moment t The action. Represents the expectation operator. State-action pair of Q value, State-action pair goal Q value, The temperature coefficient for exploring the information entropy value of the action set. For action strategy, The residuals required for policy network updates.

[0011] On the other hand, embodiments of the present invention provide a human-machine hybrid enhanced intelligent dispatching device for power systems, comprising: The first module is used to acquire a demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategies. The second module is used to input the demonstration dataset into the neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. The third module is used to execute the power system's scheduling tasks through the decision-making actions output by the behavioral clone agent, obtain multiple operating trajectories of the power system, and acquire the dispatcher's ranking of each operating trajectory. The fourth module is used to derive the reward function from the sorted running trajectory, perform reinforcement learning optimization steps on the behavior clone agent based on the reward function, obtain a hybrid reinforcement strategy, and perform power scheduling based on the hybrid reinforcement strategy.

[0012] On the other hand, embodiments of the present invention provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0013] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0014] The embodiments of this invention include the following beneficial effects: This embodiment first acquires a demonstration dataset, then generates decision suggestions by learning the decision-making behavior of human experts, constructing a behavioral clone agent; the behavioral clone agent generates different operating trajectories, and prioritizes these trajectories using human feedback; finally, a reward function is derived to guide policy optimization. The proposed method enables the decision-making agent to obtain policies superior to those based on human demonstration data, thereby achieving more efficient and safer decision-making in the intraday optimal scheduling problem of power systems. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the steps of a human-machine hybrid enhanced intelligent dispatching method for power systems provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the behavioral cloning learning demonstration strategy provided in an embodiment of the present invention; Figure 3 This is a diagram of the human-machine hybrid enhanced intelligent scheduling architecture in an embodiment of the present invention; Figure 4 This is an architecture diagram of a human-machine hybrid enhanced intelligent dispatching device for power systems provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0023] In related technologies, the mainstream methods for optimizing new power systems under uncertainty are mainly divided into stochastic optimization (SO), robust optimization (RO), chance-constrained programming, and artificial intelligence methods. SO is based on probabilistic models of uncertain variables. To obtain the probability density function of random variables, most current research adopts the scenario method, simulating a large number of scenarios to eliminate uncertainty under the assumption that the distribution of random variables is known. The goal of RO is to find the optimal solution that satisfies the system's operational constraints within an uncertainty set (usually a box-shaped uncertainty set, elliptical uncertainty set, polyhedral uncertainty set, etc.). Chance-constrained programming is another commonly used uncertainty optimization method in power system optimization and scheduling. Its goal is to find the optimal solution that satisfies the constraints under a set confidence level. It generally adopts two approaches: assuming an empirical distribution and transforming the model into a deterministic model, or sampling scenarios and using heuristic algorithms to solve the problem.

[0024] In the field of distribution network optimization and scheduling, common intelligent optimization algorithms can be divided into heuristic intelligent algorithms and data-driven algorithms such as reinforcement learning. Heuristic intelligent algorithms have the advantages of low design cost, independence from model characteristics, and the ability to handle high-dimensional problems. The earliest well-known heuristic algorithms include genetic algorithms (GA) and particle swarm optimization (PSO). Data-driven approaches are based on operational data from real physical systems, simulation data, or data generated by other intelligent algorithms. They use parameter learning methods to fit variable relationships that are difficult to express using mathematical models.

[0025] However, the performance of traditional methods depends on the quality of the modeling of the original problem and the accuracy of the prediction of uncertain variables. As the solution time increases, the computational pressure and communication burden increase, and the optimization performance decreases accordingly. The solution obtained is not the globally optimal solution. The shortcomings of intelligent methods are that they require sufficient training before they can be applied to real physical systems, and the decisions made during their learning phase are often infeasible or even violate the safety constraints of the physical system.

[0026] The reasons for the above problems are as follows: SO uses probability distributions to describe uncertain variables, which makes it difficult to fully cover all uncertain variables; RO uses min-max optimization techniques, which can only obtain the most conservative strategy; chance-constrained programming uses assumed empirical distributions, which are difficult to fully adapt to strong uncertainty. Heuristic algorithms rely on the wisdom of swarm intelligence, depend on the initial solution, and are prone to getting trapped in local optima; reinforcement learning and other algorithms are based on preset reward functions, which are prone to exceeding limits in real systems.

[0027] To address the shortcomings of existing technologies, this invention proposes a novel human-machine hybrid augmented intelligence method that combines machine intelligence and human knowledge, incorporating human feedback, to handle the intraday optimal scheduling problem of power systems, specifically addressing the needs of power system dispatching decision-making. This method first constructs a behavioral clone agent through supervised learning, generating decision suggestions by learning from the decision-making behavior of human experts. Second, it guides its own learning using demonstration data to ensure safe decision-making in the early stages of training. Third, it generates different operating trajectories and prioritizes them using human feedback. Finally, it derives a reward function to guide policy optimization. The proposed method enables the decision-making agent to obtain policies superior to those based on human demonstration data, thereby achieving more efficient and safer decision-making in the intraday optimal scheduling problem of power systems.

[0028] like Figure 1 As shown, Figure 1 A human-machine hybrid enhanced intelligent dispatching method for power systems, provided in this embodiment of the invention, includes the following steps: S100, Obtain the demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategy; Figure 2 This diagram illustrates the learning of a demonstration policy using behavioral cloning. The goal of behavioral cloning is to reproduce the decision policy πBC from the demonstration data, which is typically presented as the trajectory of a decision task. The demonstration data from a human scheduler is represented as a demonstration dataset. ,in, For the example dataset, s represents the observation state, a represents the decision action, and the subscript is the index of the example data.

[0029] S200, the demonstration dataset is input into the first neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. Specifically, the human scheduler's decision action 'a' is used as a label. Behavior cloning (BC) is used to learn the mapping from observed state to action in the human scheduler's decision-making process. The demonstration dataset is then input into the first neural network for supervised learning training, resulting in a trained first neural network with strategies similar to those of human schedulers. The behavioral clone agent uses a pre-trained first neural network as its carrier, with the input being the observed state s and the output being the decision action a. Use a parameter as The first neural network representation. Maximum likelihood estimation can be used for training.

[0030] Existing technologies reduce learning efficiency by using system operational constraints as a loss function; or they employ manual post-verification to correct agent decisions, requiring human intervention at each decision-making stage. The embodiments provided in this invention enhance the security of intelligent decision-making through demonstration-guided learning.

[0031] S300 executes the power system's scheduling tasks by outputting decision actions from the behavioral clone agent, obtains multiple operating trajectories of the power system, and acquires the dispatcher's ranking of each operating trajectory. Specifically, in the initial stage of the operation of the behavioral clone agent, the output is to use the behavioral clone agent to execute the intelligent dispatch of the power system to ensure the safety of the power system operation; then random noise is injected into the behavioral clone agent to generate multiple operating trajectories and form a diverse set of trajectories.

[0032] Existing parametric learning algorithms, such as reinforcement learning, often lead to unstable system operation during the training phase. This invention establishes a behavioral clone agent using historical operating data. In the early stages of algorithm training, the output of the behavioral clone agent is used as the decision, ensuring the system's security.

[0033] In related technologies, human schedulers only participate in post-processing verification during algorithm training; they do not directly participate in the training process. This invention achieves priority ranking of running trajectories by involving human schedulers in the algorithm training process.

[0034] S400 derives the reward function from the sorted running trajectory, performs reinforcement learning optimization steps on the behavior clone agent based on the reward function, obtains a hybrid reinforcement strategy, and performs power scheduling based on the hybrid reinforcement strategy.

[0035] Specifically, based on existing mechanistic knowledge and historical experience, humans rank the operational trajectories of the decision-making outputs of cloned intelligent agents according to their preferences. By ranking the operational trajectories, a reward function that aligns with the intrinsic value orientation of human decision-making is derived, providing optimization direction for the algorithm. Based on this reward function, reinforcement learning optimization steps are executed to obtain a hybrid reinforcement strategy that surpasses human demonstrations.

[0036] In some embodiments, the behavioral clone agent is represented as: (1); in, The policy represented by the output of the behavioral clone agent. express The parameters, For the example dataset, s represents the observation state and a represents the decision action.

[0037] In some embodiments, the process of executing power system scheduling tasks through decision-making actions output by a behavioral clone agent to obtain multiple operating trajectories of the power system and acquiring the dispatcher's ranking of each operating trajectory includes: S310, In the initial stage of the operation of the behavioral clone agent, the behavioral clone agent outputs a decision action corresponding to the observed state of the power system, and uses the decision action to execute the scheduling task of the power system to determine whether the power system is operating safely. S320, under the condition that the power system is operating safely, random noise with different noise ratios is injected into the behavioral clone agent. The behavioral clone agent with injected random noise outputs a decision action corresponding to the observed state of the power system. The decision action is used to execute the scheduling task of the power system to obtain the operating trajectory of the power system. S330: Form a trajectory set from all running trajectories, pair the running trajectories in the trajectory set, obtain the priority ranking of the two running trajectories in each pair of running trajectories from the dispatcher, and obtain the ranking of each running trajectory.

[0038] In the initial stage of the operation of the behavioral clone agent, the behavioral clone agent is used to make decisions and outputs the decision of intelligent dispatching of the power system to ensure the safety of system operation. Then, random noise is injected into the behavioral clone agent to generate a diverse set of trajectories.

[0039] This invention employs a behavioral clone agent to provide decision outputs during the algorithm training phase, ensuring the safety of the power system during training. It also provides a user-friendly channel for human dispatchers to participate in algorithm training by offering two running trajectories injected with different levels of noise, avoiding the difficulty of directly scoring the trajectories; dispatchers only need to select a preference between the two trajectories. Furthermore, the invention derives a reward function that enables the algorithm to perform online adjustments and real-time updates, ensuring the agent's adaptability to the system.

[0040] In some embodiments, the trajectory set is represented as: (2);

[0041] in, Let U be the nth trajectory, U be a uniform distribution, and η be the noise ratio of the injected behavioral clone agent. The strategy after injecting noise η.

[0042] By presenting experienced dispatchers with two flight paths injected with varying levels of noise, the dispatchers utilize their expertise to professionally evaluate and select the flight path based on the cumulative objectives achieved. The cumulative objective represents the total target for the flight cycle (e.g., daily operational revenue). Through repeated comparisons, a ranking is determined for all flight paths. This ranking is denoted as [ranking information missing]. , represented as: (3); Here, R() is the reward function.

[0043] In some embodiments, deriving the reward function from the sorted running trajectories includes: The reward function is obtained by training the second neural network using a generalized loss function based on the sorted running trajectory.

[0044] Specifically, using the sorted running trajectories, the reward for a given observed state s is estimated through a second neural network, thereby deriving the reward. The second neural network is used as the reward function, denoted as... .

[0045] For any pair of trajectories and In other words, if The ranking is lower than ( ), can then be represented as: (4); in, Represented by the reward function For the trajectory The total reward obtained over multiple time periods. Represented by the reward function For the trajectory The total reward obtained across multiple time periods is derived by using a generalized loss function based on the sorted running trajectories to iteratively train the second neural network. Represented as: (5); in, Represents the loss function. This represents the expected value when the trajectory follows a distribution. It is the distribution of the running trajectory. The binary classification loss function is represented by the probability P, which is expressed as a flexible maximum normalized distribution. (6); in, This represents the reward corresponding to the observed state s; According to the classic Luce-Shephard rule, the loss function is... Equivalent conversion to: (7); During the iterative training of the second neural network, the generalized loss function is calculated. When the generalized loss function of the second neural network is determined to be lower than the set loss threshold, the trained second neural network is obtained, and the trained second neural network is used as the reward function.

[0046] Existing techniques fit reward functions to demonstration data using inverse reinforcement learning (such as generative adversarial networks providing reward signals). This invention, however, utilizes the order of the running trajectories to derive a reward function reflecting the decision preferences of a human scheduler.

[0047] Figure 3 This describes the execution architecture of a human-machine hybrid augmented intelligence scheduling method. In some embodiments, the formulas for action policy and Q-value updates in the reinforcement learning are as follows: (8); (9); in, π * represents the optimal action strategy; r ( s t , a t () is the immediate reward for the state-action pair. It is a moment t The observation status, It is a moment tThe action; For expectation operators; γ It is a discount factor; Representing state S t+1 Value functions; H [·] represents the entropy of the action set; α Weights for temperature coefficients; Represents state-action pairs ( s t , a t )of Q value.

[0048] In some embodiments, the update formulas for the actor, evaluator, and target evaluator in the reinforcement learning are as follows: (10); (11); (12); In the formula, This represents the residual required for policy network updates; Indicates action strategy; ω for Q Network parameters, ϖ For the goal Q Network parameters, θ For policy network parameters, λ This is a soft update coefficient; r ( s t , a t () is the immediate reward for the state-action pair. It is a moment t The system observation status, It is a moment t The action. Represents the expectation operator. State-action pair of Q value, State-action pair goal Q value, The temperature coefficient for exploring the information entropy value of the action set. For action strategy, The residuals required for policy network updates.

[0049] Compared with related technologies, this invention proposes a hybrid augmented intelligence (HAI) method that combines machine intelligence and human knowledge, and considers human feedback, by deriving the value orientation of human decision-making through a reward function, to address the intraday optimal scheduling problem of novel power systems. This invention has the following advantages: A Human-Machine Hybrid Augmented Intelligence-Based Intraday Optimization Scheduling Method: This paper proposes a multi-stage stochastic dynamic optimization scheduling method based on HAI, providing a new human-machine collaborative framework for solving power dispatching problems under multiple uncertainties. The main drawbacks of model-free optimization scheduling methods, such as security uncertainty, convergence uncertainty, and reliance on human-preset reward functions, are analyzed. A new mechanism for learning from human demonstration and feedback is established.

[0050] Expert demonstration-guided learning: Behavioral cloning is used to transform expert demonstrations into guidance strategies, guiding the scheduling agent to learn safely in the early training phase and avoiding frequent trial and error. Then, a smooth switching mechanism is adopted to allow the scheduling agent to explore freely and seek better scheduling strategies.

[0051] Potential Reward Function Derivation: To ensure that the scheduling agent's decision-making performance is at least comparable to the demonstration data policy, a reward function is derived using trajectory ranking based on human feedback. The derived reward points to potential directions for improvement in the scheduling policy, allowing for further optimization of the learned policy to surpass the expert demonstration.

[0052] See Figure 4 This invention provides a human-machine hybrid enhanced intelligent dispatching device for power systems, comprising: The first module is used to acquire a demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategies. The second module is used to input the demonstration dataset into the neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. The third module is used to execute the power system's scheduling tasks through the decision-making actions output by the behavioral clone agent, obtain multiple operating trajectories of the power system, and acquire the dispatcher's ranking of each operating trajectory. The fourth module is used to derive the reward function from the sorted running trajectory, perform reinforcement learning optimization steps on the behavior clone agent based on the reward function, obtain a hybrid reinforcement strategy, and perform power scheduling based on the hybrid reinforcement strategy.

[0053] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0054] See Figure 5 This invention provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0055] It is evident that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented in this device embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0056] Furthermore, this application also discloses a computer program product or computer program stored in a computer-readable storage medium. A processor of a computer device can read the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the described method. Similarly, the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0057] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0058] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0059] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0060] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0061] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0064] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0065] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A human-machine hybrid enhanced intelligent dispatching method for power systems, characterized in that, The method includes the following steps: Obtain a demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategies; The demonstration dataset is input into the first neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. By executing the decision-making actions output by the behavioral clone agent, the scheduling tasks of the power system are executed, multiple operating trajectories of the power system are obtained, and the dispatcher's ranking of each operating trajectory is obtained. The reward function is derived by sorting the running trajectories. Based on the reward function, the behavior clone agent is subjected to reinforcement learning optimization steps to obtain a hybrid reinforcement strategy. Power scheduling is then performed based on the hybrid reinforcement strategy.

2. The method according to claim 1, characterized in that, The behavioral clone agent is represented as follows: ； in, The policy represented by the output of the behavioral clone agent. express The parameters, For the example dataset, s represents the observation state and a represents the decision action.

3. The method according to claim 1, characterized in that, The decision-making actions output by the behavioral clone agent are used to execute the power system scheduling task, resulting in multiple operating trajectories of the power system. The dispatcher's ranking of each operating trajectory is then obtained, including: In the initial stage of the operation of the behavioral clone agent, the behavioral clone agent outputs decision actions corresponding to the observed state of the power system, and uses the decision actions to execute the scheduling tasks of the power system to determine whether the power system is operating safely. Under the premise of ensuring the safe operation of the power system, random noise with different noise ratios is injected into the behavioral clone agent. The behavioral clone agent with injected random noise outputs decision actions corresponding to the observed state of the power system. The decision actions are used to execute the scheduling task of the power system, and the operating trajectory of the power system is obtained. All operating trajectories are formed into a trajectory set. Pair the running trajectories in the trajectory set into pairs, obtain the priority ranking of the two running trajectories in each pair by the scheduler, and obtain the ranking of each running trajectory.

4. The method according to claim 3, characterized in that, The trajectory set is represented as follows: ； in, Let U be the nth trajectory, U be a uniform distribution, and η be the noise ratio of the injected behavioral clone agent. The strategy after injecting noise η.

5. The method according to claim 1, characterized in that, The derivation of the reward function from the sorted running trajectories includes: The reward function is obtained by training the second neural network using a generalized loss function based on the sorted running trajectory.

6. The method according to claim 4, characterized in that, The generalized loss function is expressed as: ； in, This represents the reward corresponding to the observed state s. and Let i and j represent the i-th and j-th running trajectories, respectively.

7. The method according to claim 6, characterized in that, The formulas for action policy and Q-value updates in reinforcement learning are as follows: ； ； in, π * represents the optimal action strategy; r ( s t , a t () is the immediate reward for the state-action pair. It is a moment t The observation status, It is a moment t The action; For expectation operators; γ It is a discount factor; Representing state S t+1 Value function; H [·] represents the entropy of the action set; α Weights for temperature coefficients; Represents state-action pairs ( s t , a t )of Q value; The update formulas for the actor, evaluator, and target evaluator in the reinforcement learning process are as follows: ； (11)； ； In the formula, This represents the residual required for policy network updates; Indicates action strategy; ω for Q Network parameters, ϖ For the goal Q Network parameters, θ For policy network parameters, λ This is a soft update coefficient; r ( s t , a t () is the immediate reward for the state-action pair. It is a moment t The system observation status, It is a moment t The action. Represents the expectation operator. State-action pair of Q value, State-action pair goal Q value, The temperature coefficient for exploring the information entropy value of the action set. For action strategy, The residuals required for policy network updates.

8. A human-machine hybrid enhanced intelligent dispatching device for power systems, characterized in that, The device includes: The first module is used to acquire a demonstration dataset, which contains multiple demonstration datasets, including the observed state of the power system and the corresponding demonstration strategies. The second module is used to input the demonstration dataset into the neural network for supervised learning training to obtain a behavioral clone agent; the input of the behavioral clone agent is the observed state of the power system, and the output is the decision action. The third module is used to execute the power system's scheduling tasks through the decision-making actions output by the behavioral clone agent, obtain multiple operating trajectories of the power system, and acquire the dispatcher's ranking of each operating trajectory. The fourth module is used to derive the reward function from the sorted running trajectory, perform reinforcement learning optimization steps on the behavior clone agent based on the reward function, obtain a hybrid reinforcement strategy, and perform power scheduling based on the hybrid reinforcement strategy.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1 to 7.

Images