Green electricity generation and consumption collaborative matching method and device based on security reinforcement learning, and medium

Abstract

The method, device and medium for green power generation and consumption collaborative matching based on security reinforcement learning belong to the technical field of power automation, and solve the problem of how to improve the online decision efficiency and physical security of green power generation and consumption collaborative matching; the safety shielding layer is introduced before joint action, the power flow safety constraint is locally linearized into a convex feasible region composed of equalities and inequalities according to the current operation section; when the initial action is located outside the feasible region, the system projects it into the feasible region through quadratic programming to obtain a safe action closest to the original action in the sense of weighted two norm; meanwhile, the constraint violation amount and the projection deviation amount are introduced into the strategy network loss function in the training stage, so that the agent gradually learns the physical boundary of the power grid, and the dependence on the large correction of the shielding layer in the online stage is reduced, thereby the online decision efficiency, economy and physical security are taken into account.

Description

Technical Field

[0001] This invention belongs to the field of power automation technology and relates to a method, device and medium for collaborative matching of green power generation and utilization based on security reinforcement learning. Background Technology

[0002] As the proportion of new energy sources such as wind and solar power connected to the grid continues to increase, and given the randomness, intermittency, and volatility of new energy output, the coupling relationship between grid supply and demand balance, line power flow control, node voltage constraints, and green energy consumption becomes more complex. At the same time, electricity market users' demands for green electricity consumption ratios, green electricity consumption certificates, and carbon emission reduction compliance targets are becoming increasingly diversified. Coordination and matching between the generation side, the consumption side, the energy storage side, and the market settlement side have become crucial technical issues for ensuring the economical operation of the system and the high proportion of new energy consumption.

[0003] Green electricity generation and consumption refers to the entire process of green electricity production and consumption. Existing generation-consumption matching and safe dispatch methods usually rely on optimal power flow, mixed integer programming, or rolling optimization models. These methods have the ability to express clear physical constraints, but under high-dimensional, multi-time period, multi-entity, and high-frequency fluctuation conditions, they are prone to heavy computational burden, long online solution time, and difficulty in adapting to rapidly changing operating sections.

[0004] Deep reinforcement learning (DRL) can form a nonlinear mapping from state to action through offline training, and quickly output scheduling decisions in the online stage, with the potential to improve real-time decision-making efficiency. Existing technologies, such as CN120545958A, disclose a smart grid scheduling optimization method based on adaptive evolutionary control. This method trains grid scheduling strategies based on the PPO algorithm, accurately capturing the complex relationship between renewable energy fluctuations and load changes, and achieving real-time state perception and decision-making. However, conventional reinforcement learning models are mainly based on trial-and-error and reward feedback learning strategies. Their original output actions may violate physical safety constraints such as power balance at grid nodes, upper and lower limits of node voltage, line thermal stability constraints, or equipment operating boundaries. If these actions are issued to the underlying actuators without physical safety verification, it may lead to voltage exceeding limits, line overload, or unexecutable scheduling commands.

[0005] Therefore, it is necessary to propose a green electricity generation and utilization collaborative matching method that takes into account both the efficiency of data-driven online decision-making and the security of power grid physical constraints. This method ensures that the output of the reinforcement learning strategy undergoes interpretable security verification and correction before actual distribution, and gradually learns the power grid operation boundary during the training phase. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to improve the online decision-making efficiency and physical security of green energy generation and utilization coordination matching.

[0007] The present invention solves the above-mentioned technical problems through the following technical solutions: The green energy generation and utilization collaborative matching method based on security reinforcement learning includes an initialization phase, an offline training phase, and an online deployment phase. In the online deployment phase, the following steps are executed iteratively within each scheduling cycle: Acquire power system operation section data and market data to construct state vectors. ; Input the state vector into the pre-trained policy network to output the initial joint action. It must contain at least a green electricity quota vector. Node load variation Energy storage charging and discharging power vector and discrete device candidate states ; The initial joint action is mapped to the change in node net injected power, and the node net injected power vector is solved. ; Construct a linearized safe and feasible region that includes at least node power balance constraints, line active power flow constraints, and node voltage magnitude constraints. ; Determine the initial joint action Does it satisfy the linearized safe feasible region? If the conditions are met, the initial joint action will be considered a safe action. If not satisfied, project the initial joint action into the linearized safe feasible region, and solve for the safe action closest to the initial joint action in the sense of weighted L2 norm using quadratic programming. ; Safety actions are issued to the lower-level implementing agencies and executed.

[0008] Furthermore, the solution involves the net injected power vector at the node. Specifically: Initial joint action on the user side Converted to net power injection increment at grid node side This can be represented using the following logic:

[0009] in, A mapping matrix for green electricity allocation to generation or settlement nodes; A mapping matrix from demand response users to load nodes; This is the mapping matrix from energy storage devices to access nodes; it also provides the net injected power vector of nodes during scheduling period t. :

[0010] in, The baseline net injected power is formed by the predicted output of new energy sources, the planned output of conventional power sources, and the base load demand.

[0011] Furthermore, the dynamically linearized safe feasible region Using the following logical representation:

[0012] in, This represents inequality constraints such as node voltage magnitude constraints, line active power flow constraints, and equipment operating boundaries. This indicates the node voltage magnitude constraint. Represents action variables, This represents the coefficient matrix and right-hand vector after linearization of inequality constraints such as line active power flow constraints and equipment operating boundary constraints; This represents the nodal power balance equation constraint, where and These represent the coefficient matrix and the right-hand side variables after linearizing the power balance equation constraints, respectively. and These are the lower and upper limits of the action variable, respectively.

[0013] Furthermore, the quadratic programming solution is represented using the following logical expression:

[0014]

[0015] in, The diagonal penalty weight matrix is Represents action variables.

[0016] Furthermore, in solving the quadratic programming problem, the candidate states of the discrete devices are considered. The processing steps are as follows: An action mask is generated based on the preset hard topology rules of the power grid, illegal switching state combinations are eliminated from the original discrete actions, and the candidate topology set is retained. For each candidate topology in the candidate topology set, solve the quadratic programming problem to obtain the continuous action projection correction solution under each candidate topology; Compare the projection correction costs under each candidate topology, and select the combination of discrete states and continuous actions that has the minimum correction cost and satisfies all constraints as the safe joint action.

[0017] Furthermore, when solving the quadratic programming problem is not feasible, a backup safety strategy is triggered, which includes one or more combinations of relaxing the upper and lower limits of the action variables in the quadratic programming constraints, replacing the action with a set of conservative rule variables preset by the system, and suspending the program to wait for manual intervention.

[0018] Furthermore, during the offline training phase, the following steps are performed: Get the immediate reward after the current action is performed. and the state vector at the next time step Interactive generation of experience sequences in a simulation environment And store it in the experience replay pool; A small batch of samples is randomly drawn from the experience replay pool. The value network is used to evaluate the value of safety actions and calculate the time difference error. The value network is updated by minimizing the value loss function. At the same time, the policy network parameters are updated based on the policy gradient algorithm to maximize the expected cumulative reward. Determine whether the training metrics have reached the preset convergence condition. If not, return to the online deployment stage to continue the next scheduling cycle of iterative sampling. If they have, end the training process, save and output the trained policy network parameters.

[0019] Furthermore, constraint violation penalties and projection bias penalties based on Lagrange multipliers are introduced during the policy network training process, specifically as follows: For the form The k-th linearized physical constraint is defined as follows:

[0020] in, This represents the k-th linearized physical constraint function; The training objective of the policy network By adding a Lagrange penalty term and a projection bias term, the following logical representation can be used:

[0021] in, It is the entropy temperature coefficient; This represents the logarithmic probability density of the policy network's output action in the current state; Output to the value network; Let be the Lagrange multiplier of the k-th constraint; This is the projection bias penalty coefficient; Lagrange multipliers Dual gradient ascent update based on historical violation severity is represented as follows:

[0022] Where ρ is the multiplier learning rate. The allowable relaxation threshold, This indicates projection onto the non-negative interval.

[0023] An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the aforementioned green electricity generation cooperative matching method based on security reinforcement learning, the processor being configured to execute the program stored in the memory.

[0024] A storage medium storing a computer program, which, when executed by a processor, performs the steps of the aforementioned green electricity generation collaborative matching method based on security reinforcement learning.

[0025] The advantages of this invention are: (1) This invention constructs a collaborative framework that uses data-driven rapid decision-making and physical constraint-based security verification in the online deployment phase, and introduces constraint penalties for active learning in the offline training phase. This enables the reinforcement learning policy network to learn the nonlinear mapping between high-dimensional states and scheduling actions through offline training, and to quickly generate initial decisions such as green electricity allocation, demand response, and energy storage regulation in the online deployment phase.

[0026] Because power grid operation constraints have rigid boundaries, this invention introduces a safety shield layer before joint actions are initiated. Based on the current operating profile, the safety shield layer locally linearizes the power flow safety constraints into a convex feasible region composed of equality and inequality equations. When the initial action is outside the feasible region, the system projects it into the feasible region through quadratic programming to obtain a safe action that is closest to the original action in the sense of the weighted L2 norm. This process, under the constructed linearized model and preset safety margins, ensures that the issued actions meet preset constraints such as node power balance, line power flow, node voltage, and equipment operating boundaries. Simultaneously, during the training phase, this invention incorporates constraint violation amounts and projection deviations into the policy network loss function, enabling the agent to gradually learn the physical boundaries of the power grid. This reduces the reliance on significant modifications to the shielding layer during the online phase, thus balancing online decision-making efficiency, economy, and physical security.

[0027] (2) The present invention improves the physical executability of reinforcement learning scheduling results: by performing constraint verification and secondary planning correction on the output actions of the policy network through a security shielding layer, the final issued actions meet the preset linearization security constraints, reducing the risk of node voltage exceeding limits, line power flow exceeding limits and equipment operating out of bounds.

[0028] (3) The present invention improves the timeliness of online decision-making: by transferring part of the complex multi-objective scheduling problem to the offline training stage, the online stage mainly performs neural network forward inference and necessary low-dimensional convex optimization correction, which has better real-time response potential compared to solving large-scale optimal power flow or mixed integer programming in each cycle.

[0029] (4) This invention achieves the integration of user-side objectives and physical-side constraints: through user-node mapping and action-node net injected power conversion, green electricity allocation, green electricity consumption certificate performance, demand response and energy storage regulation are uniformly incorporated into the power grid security verification framework.

[0030] (5) The present invention enhances the multi-resource collaborative scheduling capability: The joint action space of the present invention covers green electricity allocation, flexible load regulation and energy storage charging and discharging control, and can link multiple resources to achieve comprehensive regulation when the source and load are unbalanced or local lines are restricted.

[0031] (6) The present invention improves the stability of model training: by using Lagrange constraint penalty and projection bias penalty, the policy network is guided to reduce out-of-limit actions during training, thereby improving the policy convergence stability and engineering deployment robustness. Attached Figure Description

[0032] Figure 1 This is a diagram of the green electricity generation and utilization collaborative matching system architecture of Embodiment 1 of the present invention; Figure 2 This is a flowchart of the collaborative matching method for green electricity generation based on security reinforcement learning in Embodiment 1 of the present invention; Figure 3 This is a geometric schematic diagram of the QP orthogonal projection according to Embodiment 1 of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: Example 1 like Figure 1 As shown, this embodiment provides a system architecture for performing green electricity generation and consumption collaborative matching tasks. The system relies on the power dispatch data network, green electricity trading platform, user-side metering system and power grid operation monitoring platform to realize data interaction.

[0035] like Figure 1As shown, the system comprises a hardware layer and a software layer. The hardware layer includes: a first layer of physical devices and user layer hardware, including Remote Terminal Units (RTUs) and Power Management Units (PMUs) deployed in substations, new energy power plants, energy storage stations, and user-side facilities, and smart meters deployed on the user-side; a second layer of data communication layer hardware, including SCADA communication gateways and power dispatch data network interface devices; and third and fourth layers of computing and decision-making layer hardware, including computing servers deployed in the cloud or dispatch center. The software layer corresponds to... Figure 1 The algorithm consists of a third-layer decision layer and a fourth-layer application execution layer. The third layer is the decision layer, which includes a state space construction module, a reinforcement learning policy network module, an action mapping module, a security shielding verification module, and a quadratic planning correction module. The fourth layer is the application execution layer, which includes a data preprocessing module and a control command issuance module.

[0036] like Figure 2 As shown, this embodiment discloses a collaborative matching method for green electricity generation and utilization based on security reinforcement learning to execute an online security decision-making process, including an initialization phase, an offline training phase, and an online deployment phase. The specific process of each phase is as follows: During the initialization phase, the following operations are performed: S101 constructs a power system simulation environment that includes historical grid parameters, line capacity, node load, renewable energy output, energy storage operation status, green electricity trading data, and user preference data, and completes system initialization and simulation environment reset, including at least the following operations: loading grid topology and equipment parameters, resetting renewable energy output and load sequences to the start time, initializing energy storage charge state, resetting reinforcement learning policy network and value network parameters, and clearing the experience recycling pool.

[0037] Preferably, the simulation environment described in this embodiment can be encapsulated based on the OpenAI Gym interface or built based on a power system power flow calculation platform.

[0038] During the online deployment phase, the following steps are executed cyclically within each scheduling cycle: S102, Collect system status data and construct a status vector: At the start of scheduling cycle t, the system acquires real-time operational section data and market data from SCADA, the load forecasting platform, the renewable energy power forecasting platform, the green electricity trading platform, and the user-side metering system. After data preprocessing operations such as outlier removal, missing value imputation, unit unification, and normalization, a state vector is constructed. :

[0039] in, The predicted output vector of available green electricity for each new energy power station or power generation node; This is the basic electricity demand prediction vector for each load node; This is a user preference feature matrix, with each row corresponding to a user's acceptance of green electricity premiums, green electricity consumption ratio target, or green electricity consumption certificate fulfillment progress in the electricity market. The physical operating characteristics of the power grid include node voltage safety margin, line power flow safety margin, energy storage charge status, and current active power loss benchmark value; Market characteristics include carbon quotas, green electricity trading prices, and constraints such as green certificates or green electricity consumption vouchers.

[0040] S103, The policy network (Actor) generates the initial joint action: The state vector Input a pre-trained reinforcement learning policy network and output the initial joint action. .

[0041] In this embodiment, the initial joint action This includes green electricity allocation, user-side response adjustment, and controllable equipment adjustment. To avoid mathematical conflicts between discrete actions and continuous quadratic programming projection, this embodiment prioritizes representing energy storage control actions as continuous charging and discharging power for controllable equipment adjustment. If there are devices such as circuit breakers or flexible switches that must be switched discretely, candidate action enumeration, action masking, or rule verification methods are used instead of directly projecting them as continuous QP variables. The initial joint action is represented using the following logic. :

[0042] in, This represents the green electricity quota vector, which is the amount of green electricity or corresponding green electricity consumption vouchers allocated to each electricity user, corresponding to the green electricity allocation amount; This represents the amount of node load change caused by load adjustment commands. Positive values ​​indicate that flexible loads are guided to increase electricity consumption to absorb surplus green electricity, while negative values ​​indicate that loads are reduced or transferred to alleviate power shortages or grid congestion. This corresponds to the user-side response adjustment amount. This represents the energy storage charging and discharging power vector of the energy storage device, which is mapped to the physical node of the energy storage inverter connected to the grid. Discharging is positive and charging is negative, or it can be defined according to the system convention. Represents the candidate state of discrete devices such as flexible switches, and The term "joint" indicates the adjustable amount of controllable equipment.

[0043] The above actions must meet inherent device-level constraints such as the user's maximum absorption capacity, demand response protocol capacity, upper limit of energy storage charging and discharging power, upper and lower limits of energy storage state of charge, and discrete device candidate state set.

[0044] S104, The action mapping module maps the initial joint action on the user side to the change in the net injected power of the node: Since the output of deep reinforcement learning is often an abstract vector (such as the initial joint action in S103), However, in the actual power grid security constraint verification, the power injection value of physical nodes is required. In order to enable green electricity allocation, green electricity consumption certificate allocation, and demand response commands to be subject to the linearized security feasible region verification in the security shield layer, this invention sets up an action mapping module to convert user-side actions into net power injection increments on the grid node side based on the user-node mapping matrix, the demand response resource access node matrix, and the energy storage access node matrix. This can be represented using the following logic:

[0045] in, A mapping matrix for green electricity allocation to generation or settlement nodes; A mapping matrix from demand response users to load nodes; This is a mapping matrix from energy storage devices to access nodes; In the formula, This represents the vector of green electricity quotas allocated logically to each demander. This is mapped inversely to the physical generator nodes of the new energy sources that actually produce this green electricity, and is transformed into the physical power injection increment of the new energy nodes. This indicates the amount of load change triggered by a demand response command. Respond to users' load reduction commands in response to demand; This represents the command vector for charging and discharging operations of the energy storage cluster. This is mapped to the physical nodes of the energy storage inverter connected to the grid, and converted into the power injection or absorption of the corresponding nodes.

[0046] In a preferred embodiment, the above mapping matrix , , All are node-device association matrices constructed based on the actual topological connections of the power grid and market registration information. The data directly comes from the static topology database of the power grid energy management system and the market entity registration files of the power trading center. Preferably, if a device j is connected to a power grid node i, the element in the i-th row and j-th column of the matrix is ​​1; otherwise, it is 0.

[0047] Based on this, the node net injected power vector for scheduling period t is obtained. :

[0048] in, The baseline node net injected power is formed by the projected output of new energy sources, the planned output of conventional power sources, and the base load demand. This embodiment serves as a key factor connecting market matching decisions and power grid physical security constraints.

[0049] S105, The safety shielding layer constructs a linearized safe feasible region based on the operational cross-section: Before the actual action is issued, the safety shield layer constructs a dynamically linearized safe feasible region based on the current power grid topology, operating section, and power flow sensitivity matrix. The dynamically linearized safe feasible region Using the following logical representation:

[0050] in, This represents inequality constraints such as node voltage magnitude constraints, line active power flow constraints, and equipment operating boundaries. This indicates the node voltage magnitude constraint. Represents action variables, This represents the coefficient matrix and right-hand vector after linearization of inequality constraints such as line active power flow constraints and equipment operating boundary constraints; This represents the nodal power balance equation constraint, where and These represent the coefficient matrix and the right-hand side variables after linearizing the power balance equation constraints, respectively. and These are the lower and upper limits of the action variable, respectively.

[0051] Preferably, for line active power flow constraints, DC power flow approximation or power transfer distribution factor matrix can be used; for node voltage magnitude constraints, it is preferable to perform first-order Taylor linearization based on the current AC power flow operating point and approximate it using the voltage-active / reactive sensitivity matrix.

[0052] Specifically, the node power balance described in this embodiment can be modeled as a network-wide balance constraint or a node-level balance constraint based on the actual system model, and can be approximated at the node level as follows:

[0053] The active power flow constraint of the line is represented as:

[0054] The node voltage magnitude constraint is represented by the AC power flow linearization as follows:

[0055] in, This represents the current operating section's ground-state voltage amplitude. The voltage sensitivity matrix is ​​obtained from the AC power flow Jacobian matrix. and These are the lower and upper limits of node voltage safety, respectively. This represents the active power transfer distribution factor matrix. This indicates the upper limit of the transmission capacity constrained by the active power flow of the line. This represents the planned active power output vector of a conventional power source. The linearization constraints described above are used for rapid online verification. In formal engineering implementations, they can be combined with safety margins or verification back-calculations to improve reliability.

[0056] S106, Safety shielding layer execution action verification and QP projection correction: The security shield first determines the initial joint action. Does it satisfy the linearized safe feasible region? The specific verification logic is as follows: If If the continuous action is not corrected, it will be directly regarded as a safe action. ;like This triggers the security shielding correction mechanism, which uses quadratic programming to find the safest action closest to the initial joint action in the sense of the weighted L2 norm, represented by the following logic:

[0057]

[0058] in, In this embodiment, the weight matrix is ​​a diagonal penalty matrix. The diagonal elements can be set differently based on the control inertia and adjustment cost of the action variables. For example, a larger weight can be assigned to energy storage power variables with higher adjustment costs or those that are unfavorable to frequent changes, while a smaller weight can be assigned to more flexible green electricity allocation or demand response fine-tuning variables. This embodiment uses QP to solve the problem of voltage overruns that may be caused by the random and exploratory actions generated by RL.

[0059] like Figure 3 The geometric principle of QP orthogonal projection is given, where the QP quadratic programming solution objective and physical constraint boundary are in simplified form, and their specific expressions are shown in the above equation.

[0060] Preferably, based on the fact that the objective function in the quadratic programming is a convex quadratic function and linear constraints are used, therefore in t is not empty and For semi-positive definite or positive definite problems, quadratic programming problems can be solved quickly using solvers such as OSQP and Gurobi.

[0061] For discrete actions such as flexible switches To avoid failing to meet real-time requirements by directly using time-consuming mixed-integer quadratic programming, this example employs a decoupling mechanism combining mask filtering with branch-condition QP solution, specifically: First, an action mask is generated based on the preset hard topology rules of the power grid, and obviously illegal switching state combinations in the original discrete actions are directly eliminated, leaving only a few feasible topology sets. Secondly, for each retained candidate topology, a quadratic programming model of the same form as in S106 is constructed and solved using the update sensitivity matrix and constraint coefficients under that candidate topology. That is, the objective function is to minimize the weighted L2 projection deviation, and the constraint is the linearized safe and feasible region corresponding to that candidate topology, so as to obtain the continuous action projection correction solution under each candidate topology. Then, the correction costs of the QP projection of continuous actions under each candidate topology are compared, and the combination of discrete states and continuous actions with the minimum cost that satisfies all constraints is selected as the final safe joint action.

[0062] The above decoupling mechanism cleverly avoids the problem of discrete variables becoming illegal continuous values ​​after QP projection, while ensuring computational efficiency.

[0063] Furthermore, under extreme operating conditions, if the feasible region is reduced due to factors such as excessive source load deviation, insufficient equipment capacity, overly tight line constraints, or the inability to switch the state of discrete equipment, etc. If the value is empty or QP is not feasible, the system triggers a backup security policy, which includes at least the following: Relaxing the constraints in the feasible region or adjusting the action variable space, including relaxing the upper and lower limits of action variables in the QP problem. , For example, lowering the lower limit of green electricity consumption and expanding the upper limit of the power operation space of energy storage charging and discharging or conventional power sources.

[0064] If QP still has no solution after the boundary is relaxed, the optimization solution is skipped and the final action is replaced with a set of conservative rule variables preset by the system, such as setting all devices to maintain the current state or suspending the program to wait for manual intervention.

[0065] By setting up the aforementioned backup security strategies, it is possible to effectively prevent the system from having no output or outputting unexecutable instructions when optimization is not feasible, thereby improving the closed-loop reliability of the project deployment.

[0066] S107, Perform safety actions and calculate reward feedback: Security actions are issued from the cloud after verification or correction of the security shield layer. As a safety scheduling instruction, the underlying execution mechanism executes the safety scheduling instruction, rather than the unverified raw action. The execution results are fed back to the data platform for subsequent model retraining or online performance evaluation. The system then transitions to the next state. The scheduling cycle is calculated based on the green electricity consumption efficiency, demand response cost, energy storage regulation cost, active power grid loss, and constraint correction cost. Instant rewards This can be represented using the following logic:

[0067] in, The utility of green electricity matching can be represented by a logarithmic utility function or a piecewise utility function, which can be used to characterize the needs of users with a high willingness to use green electricity. The cost of demand response can be represented by a quadratic cost function, which can be used to characterize the compensation cost caused by frequent or large adjustments. This indicates the system's active power loss; This indicates the energy storage regulation cost or the cost of depreciation over its lifespan; This represents the correction magnitude of the security shield layer, used to guide the policy network to reduce out-of-limit actions; α, β, γ, δ, and μ are all non-negative weight coefficients that can be dynamically adjusted according to the scheduling objective.

[0068] During the offline training phase, perform the following steps: S108, Receive instant rewards and the next state, and store them in the experience pool: The system obtains the immediate reward after the currently executed action. and the system state at the next moment Interactive generation of experience sequences in a simulation environment The data is then stored in the experience replay pool for subsequent offline network training.

[0069] S109, Value Network (Critic Network) Assessment of Security Actions Calculate the TD error and update the value network parameters: A small batch of samples is randomly drawn from the experience replay pool, and the value of safety actions is evaluated and the time difference error is calculated using the Critic network. The parameters of the Critic network are updated by minimizing the value loss function. Meanwhile, the Actor network parameters are updated based on the policy gradient algorithm to maximize the expected cumulative reward.

[0070] Furthermore, a safety shielding layer is simultaneously activated during the offline training phase, and the deviation between the original action and the safe action is used as one of the training signals.

[0071] S110, Determine whether the training meets the convergence condition: Determine whether the training metric has reached the preset convergence condition. If it is not met, return to step S102 to continue the next scheduling cycle of iterative sampling; if it is met, execute step S111.

[0072] In a preferred embodiment, the convergence condition is considered met when the following conditions are met simultaneously: the cumulative reward curve tends to stabilize, the linearization constraint violation rate decreases to below a preset threshold, and the projection correction magnitude converges.

[0073] S111, End training, output the optimal safe scheduling strategy: The training process ends, and the trained policy network parameters are saved and output as the final optimal safety scheduling policy model for power grid safety scheduling in real-time production environments.

[0074] Example 2 This embodiment, based on the green electricity generation and utilization collaborative matching method of Embodiment 1, uses the Soft Actor-Critic algorithm under the maximum entropy reinforcement learning framework to illustrate the neural network structure and training process of the safety reinforcement learning model. Furthermore, this invention can also employ other Actor-Critic algorithms with continuous action output capabilities to implement the aforementioned green electricity generation and utilization collaborative matching method.

[0075] It should be noted that during the online deployment phase, only the parameters of the trained policy network are extracted, and the forward inference and security verification processes in steps S102-S107 of Example 1 are executed. Specifically, the specific content of the training of the policy network, value network, and constraint penalties is explained as follows: The policy network (Actor network) uses state vectors The input is a continuous action branch and a discrete candidate action branch. The continuous action branch outputs the mean and logarithmic standard deviation of the Gaussian policy, which are reparameterized and mapped to the [-1,1] interval using a Tanh transform. Then, they are linearly scaled to the green electricity quota vector according to the actual physical boundaries of each action variable. Load adjustment command and energy storage charge / discharge power vector The discrete candidate action branch is used to output the probability or logarithmic probability of each candidate device state.

[0076] It should be noted that in this embodiment, the policy network output needs to be verified or corrected by the security shield layer before it can be used as the actual security action in the value assessment and experience playback of the Critic network, rather than directly using the unverified original action.

[0077] Preferably, for discrete candidate action branches, Gumbel-Softmax approximation sampling can be used during offline training to support backpropagation, while action masks and maximum probability selection are used to obtain legal discrete actions during online deployment. .

[0078] It should be noted that, unlike existing technologies, this embodiment employs a decoupling mechanism combining mask filtering with branch condition QP solution when processing mixed action spaces, effectively avoiding the online solution overhead of mixed integer programming.

[0079] The value network (Critic network) is used to evaluate the expected cumulative reward of state-action pairs. It includes two independent Q-networks and a corresponding target Q-network for temporal difference target computation.

[0080] Furthermore, to enable the policy network to actively avoid physical security boundaries in the later stages of training, rather than relying entirely on passive correction by the security shielding layer, this embodiment introduces constraint violation penalties and projection bias penalties based on Lagrange multipliers during the training process. Specifically, for the form of... The k-th linearized physical constraint is defined as follows:

[0081] in, This represents the k-th linearized physical constraint function, such as line active power flow constraint or node voltage magnitude constraint.

[0082] Furthermore, in the training objective of the policy network By adding a Lagrange penalty term and a projection bias term, the following logical representation can be used:

[0083] in, This is the entropy temperature coefficient, used to encourage strategy exploration; This represents the logarithmic probability density of the policy network's output action in the current state; Output to the value network; Let be the Lagrange multiplier of the k-th constraint; This is the projection bias penalty coefficient.

[0084] Furthermore, Lagrange multipliers Dual gradient ascent update based on historical violation severity is represented as follows:

[0085] Where ρ is the multiplier learning rate. The allowable relaxation threshold, This indicates projection onto the non-negative interval. If the policy network continues to generate out-of-bounds actions, This will increase, prompting network updates to tend towards reducing related constraint violations; when actions are within a safe range, constraint penalties tend to decrease, and the network continues to optimize green electricity matching benefits and operational economy.

[0086] It should be noted that in existing technologies, the Lagrange penalty term targets a general violation of safety constraints. Its design aims to guide the policy to avoid unsafe regions by penalizing constraint violations, but it cannot directly reduce the deviation between the policy network output and safe actions. This invention, however, introduces a projection bias penalty term into the training objective function. The projection bias penalty term directly uses the distance between the original output of the policy network and the action after correction by the security shield as the penalty object, so that the policy network gradually learns the correction mode of the security shield during the training process, thereby reducing the dependence on large corrections of the security shield in the online stage and achieving a synergistic improvement in online decision-making efficiency, economy and physical security.

[0087] Example 3 An apparatus includes a memory and a processor, the memory being used to store a program that supports the processor in executing the green electricity generation cooperative matching method based on security reinforcement learning as described in Embodiment 1 or 2, the processor being configured to execute the program stored in the memory.

[0088] Example 4 A storage medium storing a computer program, which, when executed by a processor, performs the steps of the green electricity generation collaborative matching method based on security reinforcement learning in Embodiment 1 or 2.

[0089] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein can be considered as a ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-based system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0090] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0091] In the description of this specification, references to terms such as "in a preferred embodiment," "preferred," "in this embodiment," "specific," or "furthermore," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative matching method for green electricity generation and utilization based on security reinforcement learning, characterized in that, It includes an initialization phase, an offline training phase, and an online deployment phase; in the online deployment phase, the following steps are executed cyclically within each scheduling cycle: Acquire power system operation section data and market data to construct state vectors. ; Input the state vector into the pre-trained policy network to output the initial joint action. It must contain at least a green electricity quota vector. Node load variation Energy storage charging and discharging power vector and discrete device candidate states ; The initial joint action is mapped to the change in net injected power at the nodes, and the net injected power vector at the nodes is solved. ; Construct a linearized safe and feasible region that includes at least node power balance constraints, line active power flow constraints, and node voltage magnitude constraints. ; Determine the initial joint action Does it satisfy the linearized safe feasible region? If the conditions are met, the initial joint action will be considered a safe action. If not satisfied, project the initial joint action into the linearized safe feasible region, and solve for the safe action closest to the initial joint action in the sense of weighted L2 norm using quadratic programming. ; Safety actions are issued to the lower-level implementing agencies and executed.

2. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 1, characterized in that, The solution node net injected power vector Specifically: Initial joint action on the user side Converted to net power injection increment at grid node side This can be represented using the following logic: in, A mapping matrix for green electricity allocation to generation or settlement nodes; A mapping matrix from demand response users to load nodes; This is the mapping matrix from energy storage devices to access nodes; it also provides the net injected power vector of nodes during scheduling period t. : in, The baseline net injected power is formed by the predicted output of new energy sources, the planned output of conventional power sources, and the base load demand.

3. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 1, characterized in that, The dynamically linearized safe feasible region Using the following logical representation: in, This represents inequality constraints such as node voltage magnitude constraints, line active power flow constraints, and equipment operating boundaries. This indicates the node voltage magnitude constraint. Represents action variables, This represents the coefficient matrix and right-hand vector after linearization of inequality constraints such as line active power flow constraints and equipment operating boundary constraints; This represents the nodal power balance equation constraint, where and These represent the coefficient matrix and the right-hand side variables after linearizing the power balance equation constraints, respectively. and These are the lower and upper limits of the action variable, respectively.

4. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 3, characterized in that, The quadratic programming solution is represented by the following logic: in, The diagonal penalty weight matrix is Represents action variables.

5. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 4, characterized in that, In solving the quadratic programming problem, the candidate states of the discrete devices are considered. The processing steps are as follows: An action mask is generated based on the preset hard topology rules of the power grid, illegal switching state combinations are eliminated from the original discrete actions, and the candidate topology set is retained. For each candidate topology in the candidate topology set, solve the quadratic programming problem to obtain the continuous action projection correction solution under each candidate topology; Compare the projection correction costs under each candidate topology, and select the combination of discrete states and continuous actions that has the minimum correction cost and satisfies all constraints as the safe joint action.

6. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 4, characterized in that, When solving the quadratic programming problem is not feasible, a backup safety strategy is triggered, which includes one or more combinations of relaxing the upper and lower limits of the action variables in the quadratic programming constraints, replacing the action with a set of conservative rule variables preset by the system, and suspending the program to wait for manual intervention.

7. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 1, characterized in that, During the offline training phase, perform the following steps: Get the immediate reward after the current action is performed. and the state vector at the next time step Interactive generation of experience sequences in a simulation environment And store it in the experience replay pool; A small batch of samples is randomly drawn from the experience replay pool. The value network is used to evaluate the value of safety actions and calculate the time difference error. The value network is updated by minimizing the value loss function. At the same time, the policy network parameters are updated based on the policy gradient algorithm to maximize the expected cumulative reward. Determine whether the training metrics have reached the preset convergence condition. If not, return to the online deployment stage to continue the next scheduling cycle of iterative sampling. If they have, end the training process, save and output the trained policy network parameters.

8. The green electricity generation and utilization collaborative matching method based on security reinforcement learning according to claim 7, characterized in that, During the training of the policy network, constraint violation penalties and projection bias penalties based on Lagrange multipliers are introduced, specifically as follows: For the form The k-th linearized physical constraint is defined as follows: in, This represents the k-th linearized physical constraint function; The training objective of the policy network By adding a Lagrange penalty term and a projection bias term, the following logical representation can be used: in, It is the entropy temperature coefficient; This represents the logarithmic probability density of the policy network's output action in the current state; Output to the value network; Let be the Lagrange multiplier of the k-th constraint; This is the projection bias penalty coefficient; Lagrange multipliers Dual gradient ascent update based on historical violation severity is represented as follows: Where ρ is the multiplier learning rate. The allowable relaxation threshold, This indicates projection onto the non-negative interval.

9. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the collaborative matching method for green electricity generation based on security reinforcement learning as described in any one of claims 1 to 8, wherein the processor is configured to execute the program stored in the memory.

10. A storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, performs the steps of the collaborative matching method for green electricity generation based on security reinforcement learning as described in any one of claims 1 to 8.

Images