Power transaction strategy risk management and control method and system based on reinforcement learning

Abstract

The invention provides a power transaction strategy risk management and control method and system based on reinforcement learning, and the method comprises the following steps: obtaining historical load and weather forecast data of a power transaction subject, inputting the historical load and weather forecast data into a load prediction model to obtain load prediction data, and generating a reference transaction strategy through an optimization algorithm; determining an emergency transaction strategy according to a historical prediction error of the model, and circularly executing the following steps until a decision strategy network of the target agent converges: constructing a current state vector of the target agent in a simulation environment; calculating a strategy fusion coefficient through the decision strategy network; obtaining a final transaction strategy according to the strategy fusion coefficient; and executing the strategy award obtaining signal, and updating adjustable parameters of the strategy network through a reinforcement learning algorithm and the signal. By implementing the technical scheme provided by the invention, prediction deviation can be effectively adapted, the problem of sharp deterioration of strategy performance caused by prediction errors is avoided, and stable and effective risk management and control are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power transaction subjects, and in particular to a power transaction strategy risk management and control method and system based on reinforcement learning. BACKGROUND

[0002] Power transaction subjects, such as power plants, independent energy storage power stations, and virtual power plants (VPP), etc., aggregate a large amount of distributed energy into a unified entity through advanced control and communication technology, and participate in the bidding transaction of the electricity market to achieve economic benefits. However, due to the natural intermittency and volatility of the output of renewable energy such as wind and solar energy aggregated inside, and the real-time changes of market prices, the power transaction subject faces great uncertainty when formulating a transaction strategy.

[0003] To cope with the above transaction risks, the existing technology generally adopts a prediction-based decision-making method. This method first uses a deep learning model such as a long short-term memory (LSTM) to predict the future net load of the power transaction subject according to multi-dimensional data such as weather, historical load, and market price, and then directly uses the deterministic curve obtained by the prediction as an accurate reflection of the future market state, and formulates a theoretically optimal bidding strategy based on this. However, this strictly separated technical path of prediction and decision-making has an inherent technical problem: the decision-making stage completely relies on and trusts the single deterministic curve output by the prediction model, and takes it as the only and accurate basis for the future state. This processing method results in the transaction strategy finally formulated being only a theoretical optimal solution for this specific prediction scenario. When the actual market situation deviates from the prediction, the execution effect of the strategy will deteriorate sharply, making it difficult to stably and effectively control the transaction risks of the power transaction subject. SUMMARY

[0004] Therefore, the present application provides a power transaction strategy risk management and control method and system based on reinforcement learning to solve the above problems.

[0005] In a first aspect, a power transaction strategy risk management and control method based on reinforcement learning is provided, which is applied to, and the method comprises: obtaining historical load data and weather forecast data of a power transaction subject, and inputting the historical load data and weather forecast data into a preset load prediction model for processing to obtain load prediction data, and generating a benchmark transaction strategy through a constraint optimization algorithm from the load prediction data, wherein the benchmark transaction strategy includes predicted declared power and predicted declared price of each transaction period; determining an emergency transaction strategy according to historical prediction error data of the load prediction model; Repeat the following steps until the decision policy network of the target agent converges: In a pre-defined simulation environment containing multiple market participant agents, the current state vector of the target agent is constructed. The target agent is any one of the multiple market participant agents. The simulation environment has the function of determining the transaction result by clearing the market based on the trading strategies of multiple market participant agents, and the function of calculating penalties based on the deviation between the transaction result and the trading strategy. Based on the current state vector, the policy fusion coefficient is calculated through the decision policy network inside the target intelligent body; Based on the strategy fusion coefficient, the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy are weighted and calculated with the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy to obtain the final trading strategy. The final trading strategy is executed in a simulation environment to obtain reward signals. The adjustable parameters in the decision-making strategy network are then updated using reinforcement learning algorithms and the reward signals.

[0006] The above technical solution constructs a risk-reward boundary for decision-making by generating a benchmark trading strategy based on prediction and an emergency trading strategy based on historical worst-case error in parallel, fundamentally overcoming the blind reliance of existing technologies on a single plan. The reinforcement learning agent does not make direct decisions but calculates a strategy fusion coefficient based on a state vector containing a quantitative risk assessment of the benchmark trading strategy. This coefficient is used to dynamically weight and combine the two strategies, thereby deriving a resilient final strategy that can adapt to uncertainty. A cyclical learning process including bias penalty and reward until convergence ensures that this risk-reward trade-off capability can continuously self-optimize. The resulting strategy effectively adapts to prediction bias, avoiding the problem of drastic performance degradation due to prediction errors in existing technologies, and achieving stable and effective risk management.

[0007] Optionally, load forecasting data can be used to generate a benchmark trading strategy through a constrained optimization algorithm, specifically including: The load forecast data for each trading period is used as the boundary constraint for each forecasted electricity volume, and the upper and lower limits of power, power ramp rate, capacity status of energy storage units and boundary constraints of each distributed energy source within the power trading entity are constructed as constraint conditions. A function for expected transaction revenue is constructed using the predicted declared electricity volume and the predicted declared price as variables. Under constraints, the maximum value of the expected trading revenue function is obtained to get the predicted bid volume and predicted bid price for each trading period. The predicted bid volume and predicted bid price are then combined to obtain the benchmark trading strategy.

[0008] The above technical solution uses physical parameters such as the upper and lower limits of power output, ramp rate, and energy storage capacity of each distributed energy source within the power trading entity as constraints for the optimization algorithm. In the benchmark trading strategy output by the algorithm, the declared electricity volume and price for each trading period have been pre-verified for physical feasibility, ensuring that the strategy can be directly applied to actual physical equipment and will not fail due to exceeding the equipment's operating limits.

[0009] Optionally, based on historical forecasting error data from the load forecasting model, an emergency trading strategy can be determined, specifically including: By comparing the historical load forecast data of the power trading entities with the corresponding historical actual load data, all historical trading days with historical forecast error data greater than the preset error threshold are filtered out to obtain the error trading day set. For each trading session, extract the actual load data for all historical trading days in the error trading day set for each corresponding trading session; The average load value for each trading period is obtained by averaging the actual load data. All average load values ​​are combined according to time sequence to form an average high-error load curve; The average high-error load curve is multiplied by a preset conservative coefficient to generate emergency power reporting. A preset conservative bid price is matched to the emergency bid volume, and an emergency trading strategy is formed by combining the emergency bid volume.

[0010] The above technical solution filters out multiple historical high-error trading days by setting an error threshold, and calculates the average value of the actual load data for these dates by time period. Replacing a single extreme value with a statistical average of multiple data points makes the benchmark load curve on which the generated emergency trading strategy is based smoother and more representative, avoiding overly conservative or distorted emergency strategies due to single extreme anomalies such as astronomical or meteorological events.

[0011] Optionally, a preset conservative declaration price can be matched for emergency electricity demand, specifically including: When the emergency electricity demand is non-negative, the conservative demand price will be set as the minimum allowable price in the electricity market. When the emergency electricity demand is negative, the conservative demand price will be set as the highest permissible price in the electricity market.

[0012] The above technical solution sets the electricity sales price for the emergency trading strategy to the lowest permissible market price and the electricity purchase price to the highest permissible market price. In the market clearing mechanism that prioritizes prices, this price will be placed at the highest priority in the transaction sequence, thereby ensuring that emergency electricity applications can be matched and traded preferentially by the market system, directly reducing the possibility of deviation penalties due to failed transactions.

[0013] Optionally, in a pre-defined simulation environment containing multiple market participant agents, the current state vector of the target agent is constructed, specifically including: The risk vector is obtained by calculating the historical performance of the benchmark trading strategy and the load forecasting model through a pre-defined feedforward neural network. Market prices and behavioral data of competing agents are obtained from a simulation environment. Competing agents are other agents among multiple market participants besides the target agent. Based on the benchmark trading strategy, risk vector, market price, and behavioral data, the current state vector is constructed.

[0014] The above technical solution combines the parameters of the benchmark trading strategy, a quantified risk vector, real-time market prices, and competitor behavior data as input to the decision network. This provides the decision network with information across four dimensions: its own plan, plan risk, market environment, and competitors, enabling it to learn and build more complex game-theoretic decision-making models that go beyond simple price prediction.

[0015] Optionally, a risk vector is obtained by calculating the historical performance of the benchmark trading strategy and the load forecasting model using a pre-defined feedforward neural network, specifically including: Each predicted electricity volume, each predicted price, and the average absolute error value are used together as input features of the feedforward neural network. The average absolute error value is calculated by the load forecasting model on the reserved historical validation set. The risk vector is obtained by forward propagation of the input features through a feedforward neural network.

[0016] The above technical solution uses a feedforward neural network to take the declared electricity volume and price in the prediction benchmark strategy, along with the quantitative indicators of the historical performance of the load forecasting model, as inputs for calculation. This concretizes the abstract concept of "risk" into a deterministic output (risk vector) calculated from explicit inputs (trading plan and historical forecast accuracy) through a nonlinear function, making the risk assessment process repeatable and based on direct data.

[0017] Optionally, the final trading strategy is executed in a simulation environment to obtain a reward signal. The adjustable parameters in the decision-making strategy network are then updated using a reinforcement learning algorithm and the reward signal. Specifically, this includes: The final trading strategy is input into the simulation environment to determine the actual transaction volume and the actual transaction price. The transaction revenue is calculated based on the actual transaction volume and the actual transaction price. The deviation penalty is calculated based on the difference between the declared electricity volume and the actual transaction electricity volume in the final trading strategy, and according to the preset deviation penalty rules. By combining trading profits with deviation penalties, a reward signal is obtained; The adjustable parameters, including weight parameters and bias parameters, are updated based on the reward signal using the policy gradient algorithm.

[0018] The above technical solution clarifies that the reward signal is composed of the actual transaction profit and the deviation penalty caused by the discrepancy between the declaration and the execution. When updating weights through algorithms such as policy gradient, any decision that increases the deviation penalty will directly result in a negative reward signal, thereby reducing the probability of the algorithm making similar decisions in the next iteration. This establishes a direct and quantifiable reward and penalty transmission path, forcing the agent's optimization direction to converge towards the goal of balancing profit and compliance.

[0019] Optionally, trading profits can be combined with deviation penalties to obtain reward signals, specifically including: Calculate the net profit of the target agent. The net profit equals the transaction profit minus the bias penalty. Obtain the average net profit of competing agents in each trading session; The net profit is weighted and calculated with the average net profit to generate a reward signal.

[0020] The above technical solution weights the target agent's own net profit (transaction profit minus bias penalty) with the average net profit of competing agents obtained from the simulation environment to generate the final reward signal. Its technical effect is that the reward signal value used to update the decision network depends not only on the agent's own profitability but also directly on the difference between its profitability level and the market average profitability level. A decision that is profitable but below the market average generates a negative reward signal, which will directly cause the adjustable parameters to adjust in subsequent update steps in a direction that reduces the probability of making similar decisions.

[0021] Optionally, the adjustable parameters can be updated based on the reward signal using a policy gradient algorithm, specifically including: Calculate the policy gradient of the policy fusion coefficients output by the decision policy network; The policy gradient is weighted using the reward signal to generate the update values ​​for each adjustable parameter; Adjust the adjustable parameters in the decision-making strategy network based on the update amount.

[0022] The above technical solution calculates the policy gradient of the policy fusion coefficients output by the decision network and then weights this policy gradient vector using a scalar reward signal to generate an update vector with the same dimension as the network's adjustable parameters. Its technical effect is that it transforms a single reward value, used to evaluate the quality of a decision, into an update vector that corresponds one-to-one with each adjustable parameter in the network (including weight and bias parameters), with a clear adjustment direction and magnitude. When the reward signal is positive, the direction of this update vector is consistent with the policy gradient direction; when the reward signal is negative, the direction is opposite. Applying this update vector directly to the network's adjustable parameters can proportionally amplify the output probability of a high-reward decision from the previous step, and conversely, suppress it.

[0023] Secondly, a risk management system for power trading strategies based on reinforcement learning is provided. The system includes a strategy generation module, an intelligent decision-making module, and a reinforcement learning module, wherein: The strategy generation module is configured to acquire historical load data and weather forecast data of the power trading entity, and input the historical load data and weather forecast data into the preset load forecast model for processing to obtain load forecast data. The load forecast data is then used to generate a benchmark trading strategy through a constrained optimization algorithm. The benchmark trading strategy includes the predicted electricity volume and predicted price for each trading period. The strategy generation module is also configured to determine emergency trading strategies based on historical forecast error data from the load forecasting model. The intelligent decision-making module is configured to construct the current state vector of the target intelligent agent in a preset simulation environment containing multiple market participant intelligent agents. The target intelligent agent is any one of the multiple market participant intelligent agents. The simulation environment has the function of determining the transaction result by market clearing according to the trading strategies of multiple market participant intelligent agents, and the function of calculating penalties according to the deviation between the transaction result and the trading strategy. The intelligent decision-making module is also configured to calculate the policy fusion coefficients based on the current state vector through the decision policy network inside the target intelligent body; The intelligent decision-making module is also configured to perform weighted calculations based on the strategy fusion coefficient, which calculates the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy and the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy, to obtain the final trading strategy. The reinforcement learning module is configured to execute the final trading strategy in a simulation environment, obtain reward signals, and update the adjustable parameters in the decision policy network using reinforcement learning algorithms and reward signals.

[0024] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the foregoing.

[0025] A fourth aspect of this application provides a computer-readable storage medium storing instructions that, when executed, perform the method described in any of the preceding descriptions.

[0026] In summary, implementing one or more technical solutions provided in this application has at least the following technical effects or advantages: By generating a benchmark trading strategy based on prediction and an emergency trading strategy based on historical worst-case error in parallel, a risk-reward boundary for decision-making is constructed, fundamentally overcoming the blind reliance of existing technologies on a single plan. The reinforcement learning agent does not make direct decisions but calculates a strategy fusion coefficient based on a state vector containing a quantitative risk assessment of the benchmark trading strategy. This coefficient is used to dynamically weight and combine the two strategies, thereby deriving a resilient final strategy that can adapt to uncertainty. A cyclical learning process incorporating bias penalties and rewards until convergence ensures that this risk-reward trade-off capability can continuously self-optimize. The resulting strategy effectively adapts to prediction bias, avoiding the problem of drastic performance degradation due to prediction errors in existing technologies, and achieving stable and effective risk management. Attached Figure Description

[0027] Figure 1 This is an exemplary system architecture diagram of a power trading strategy risk management method or a power trading strategy risk management system based on reinforcement learning, which applies the present application. Figure 2 This is a flowchart illustrating a risk management method for power trading strategies based on reinforcement learning, as described in an embodiment of this application. Figure 3 This is a schematic diagram of a module of a power trading strategy risk management system based on reinforcement learning, as described in an embodiment of this application.

[0028] Explanation of reference numerals in the attached figures: 100, System architecture; 101, First terminal device; 102, Second terminal device; 103, Third terminal device; 104, Network; 105, Server; 301, Policy generation module; 302, Intelligent decision-making module; 303, Reinforcement learning module. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0030] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0031] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0032] Figure 1 This paper illustrates an exemplary system architecture diagram of an embodiment of a reinforcement learning-based power trading strategy risk management method or a reinforcement learning-based power trading strategy risk management system that can be applied to this application.

[0033] like Figure 1 As shown, the system architecture 100 may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as model training applications, video recognition applications, web browser applications, social platform software, etc.

[0035] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0036] When terminals 101, 102, and 103 are hardware devices, video capture devices can also be installed on them. These video capture devices can be various devices capable of capturing video, such as cameras, sensors, etc. Users can use the video capture devices on terminals 101, 102, and 103 to capture video.

[0037] Server 105 can be a server that provides various services, such as a backend server for processing data displayed on terminal devices 101, 102, and 103. The backend server can analyze and process the received data and can feed back the processing results (such as recognition results) to the terminal devices.

[0038] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. In particular, if the target data does not need to be obtained remotely, the above system architecture may exclude the network and include only terminal devices or servers.

[0040] Figure 2This is a flowchart illustrating a reinforcement learning-based risk management method for power trading strategies, as described in this application. This method can be implemented using a computer program, a microcontroller, or run on a reinforcement learning-based power trading strategy risk management system. The computer program can be integrated into the application or run as a standalone tool application. The specific steps of the reinforcement learning-based power trading strategy risk management method are described in detail below.

[0041] S201: Obtain historical load data and weather forecast data from the power trading entity, input the historical load data and weather forecast data into the preset load forecasting model for processing to obtain load forecast data, and generate a benchmark trading strategy by using the load forecast data through a constrained optimization algorithm. The benchmark trading strategy includes the predicted electricity volume and predicted price for each trading period.

[0042] For example, historical load and weather forecast data of the power trading entity are acquired and input into a preset load forecasting model to obtain load forecast data for each future trading period. This load forecast data is then input into a constraint optimization algorithm. This algorithm solves the problem with the goal of maximizing trading revenue while satisfying the constraints of the power trading entity's physical operating boundaries and power balance, ultimately generating the benchmark trading strategy. This strategy specifies the predicted electricity volume and predicted price for each trading period.

[0043] For example, the load forecasting model can be implemented using an LSTM consisting of an input layer, two hidden layers, and an output layer. The input layer receives multi-dimensional time-series features such as historical load and weather data; each of the two hidden layers contains 128 neurons for deep extraction of time dependencies; the output layer contains 24 neurons, corresponding to the load forecast values ​​for the next 24 trading periods.

[0044] In one possible implementation, load forecast data is used to generate a benchmark trading strategy through a constrained optimization algorithm. Specifically, this includes: using load forecast data for each trading period as boundary constraints for each forecasted electricity volume; and constructing constraints based on the upper and lower limits of power, power ramp-up rate, capacity status of energy storage units, and boundary constraints of each distributed energy source within the power trading entity; constructing an expected trading revenue function using the forecasted electricity volume and forecasted price as variables; and, under the constraints, solving for the maximum value of the expected trading revenue function to obtain the forecasted electricity volume and forecasted price for each trading period, and combining each forecasted electricity volume and forecasted price to obtain the benchmark trading strategy.

[0045] In the embodiments of this application, constraints refer to a series of boundary rules and limiting equations that must be followed by the algorithm during optimization calculations. These constraints represent the objective limitations faced by the power trading entity in physical operation and market transactions. For example, the power upper and lower limit constraints indicate that the output of each power generation device within the power trading entity must be within the range of its nameplate power limit, ensuring the executability of the calculation results in the real world.

[0046] Specifically, the constraints are summarized and constructed. This process includes: quantifying the physical operating parameters of each distributed energy source (such as photovoltaic, wind power, and energy storage batteries) aggregated within the power trading entity, including their minimum and maximum output power, power regulation ramp-up rate, and the current available capacity status of energy storage units, into mathematical inequalities; setting the load forecast data for each trading period as the upper or lower limit of the predicted electricity volume for that period. A predictive trading revenue function is constructed, using the predicted electricity volume and predicted price for each future period as the two unknown variables. This function is used to calculate the total economic return under a given bidding strategy. Finally, under the constraints, the predictive trading revenue function is solved to find the combination of predicted electricity volume and predicted price that maximizes the function value. This combination, containing specific values ​​for each period, is then used as the output of the benchmark trading strategy.

[0047] For example, the capacity state of the energy storage unit can be used as a constraint, specifically implemented as follows: The state of charge (SOC_t) of the energy storage unit in each trading period t is used as a variable, which must satisfy the following iterative expression and boundary conditions: State transition constraint: SOC_t = SOC_(t-1) + η_chg*Pchg_t - (1 / η_dis)*Pdis_t; Capacity boundary constraint: SOC_min ≤ SOC_t ≤ SOC_max; Charging / discharging power constraint: 0 ≤ Pchg_t ≤ Pchg_max and 0 ≤ Pdis_t ≤ Pdis_max. Where Pchg_t is the charging power in period t, Pdis_t is the discharging power in period t, η_chg is the charging efficiency, η_dis is the discharging efficiency, SOC_min is the minimum state of charge, SOC_max is the maximum state of charge, Pchg_max is the maximum charging power, and Pdis_max is the maximum discharging power. This set of constraints ensures that the scheduling scheme of the energy storage unit conforms to its physical operating laws.

[0048] S202: Determine emergency trading strategies based on historical forecasting error data from the load forecasting model.

[0049] For example, a historical prediction error threshold is set, and all historical trading days with prediction errors greater than this threshold are selected to form a set of high-error trading days. The actual load data of all trading days in this set are averaged point-by-point according to time periods to obtain an average high-error load curve representing a typical high-risk scenario. This curve is then multiplied by a conservative coefficient less than 1 to generate the emergency electricity demand for each time period. A conservative bid price, set to the lowest or highest allowed price in the electricity market, is matched to the generated emergency electricity demand. Specifically, if the emergency electricity demand is non-negative (electricity sales), the price is set to the lowest allowed price in the electricity market; if it is negative (electricity purchase), the price is set to the highest allowed price in the market. The final determined electricity demand and price are combined to form a complete emergency trading strategy.

[0050] In one possible implementation, an emergency trading strategy is determined based on historical forecast error data from the load forecasting model. Specifically, this includes: comparing historical load forecast data from the power trading entity with corresponding historical actual load data to filter all historical trading days with historical forecast error data exceeding a preset error threshold, thus obtaining a set of error trading days; for each trading period, extracting the actual load data for all historical trading days in the error trading day set for each corresponding trading period; averaging each actual load data to obtain the average load value for each trading period; combining all average load values ​​chronologically to form an average high-error load curve; multiplying the average high-error load curve by a preset conservative coefficient to generate emergency declared electricity volume; matching the emergency declared electricity volume with a preset conservative declared price, and combining this with the emergency declared electricity volume to constitute an emergency trading strategy.

[0051] In this embodiment, the average high-error load curve refers to a time series of data representing the typical actual load conditions of electricity trading entities when the load forecasting model has historically performed poorly. This curve is not derived from a single worst historical record, but rather generated by statistically averaging the actual load data of multiple historical trading days with large forecast errors, serving as a more general benchmark for high-risk scenarios. For example, if 10 high-error trading days are selected, the load value at a certain point on the curve (e.g., 14:00) is the arithmetic mean of the actual load at 14:00 on these 10 trading days.

[0052] Specifically, a historical forecast error threshold is set as a screening criterion, for example, 20%. By comparing historical load forecast data with actual load data daily, all dates with forecast errors exceeding this threshold are filtered out, forming a set of high-error trading days. For each trading period in the future trading cycle, the actual load data corresponding to all trading days in the aforementioned set for that period is extracted, and the average value of these data is calculated to obtain the average load value for each period. Then, the average load values ​​of all periods are combined chronologically to form an average high-error load curve. The value at each point in time on this curve is multiplied by a preset conservative coefficient less than 1 (e.g., 0.9) to reduce the safety margin of the result, thereby generating the emergency electricity demand for each period. The generated emergency electricity demand for each period is matched with a preset conservative bid price that prioritizes ensuring the transaction, and the electricity and price are combined to finally form an emergency trading strategy.

[0053] In one possible implementation, a preset conservative bid price is matched for the emergency bid electricity, specifically including: when the emergency bid electricity is non-negative, the conservative bid price is set to the minimum allowable bid price in the electricity market; when the emergency bid electricity is negative, the conservative bid price is set to the maximum allowable bid price in the electricity market.

[0054] In this embodiment, a conservative bid price refers to a bid price whose primary purpose is to ensure the successful execution of the transaction to the greatest extent possible, thereby avoiding the risk of deviation penalties. This price is not set with profit maximization as the guiding principle, but rather strategically selects extreme prices within the electricity market rules to obtain the highest market clearing priority. For example, when acting as a power seller, setting the price to the lowest price allowed by the market (i.e., the floor price) ensures priority acceptance in a price-ranked market mechanism.

[0055] Specifically, this step determines the positive or negative direction of the input emergency electricity declaration to ascertain the trading role of the electricity trading entity during that period. If the emergency electricity declaration is determined to be non-negative, it indicates that the electricity trading entity is either a seller or in a self-balancing state during that period. In this case, to ensure that the declared electricity can be preferentially absorbed by the market, the conservative bid price is directly set to the minimum allowable price stipulated in the electricity market operation rules. Conversely, if the emergency electricity declaration is determined to be negative, it indicates that the electricity trading entity is a buyer during that period and needs to purchase electricity from the market. In this case, to ensure that the required electricity can be successfully purchased, the conservative bid price is directly set to the maximum allowable price stipulated in the electricity market operation rules.

[0056] Repeat steps S203 to S206 until the decision policy network of the target agent converges.

[0057] S203: In a preset simulation environment containing multiple market participant agents, construct the current state vector of the target agent. The target agent is any one of the multiple market participant agents. The simulation environment has the function of determining the transaction result by market clearing according to the trading strategies of multiple market participant agents, and the function of calculating penalties according to the deviation between the transaction result and the trading strategy.

[0058] For example, a pre-defined feedforward neural network is used to calculate the risk vector. Specifically, the reported electricity volume and price for each time period in the benchmark trading strategy, along with the mean absolute error representing the historical performance of the load forecasting model, are used as input features to the network. After forward propagation, a risk vector quantifying the expected risk of the benchmark trading strategy for each time period is output. Simultaneously, real-time market prices and behavioral data from other competing agents are acquired from the simulation environment. The parameters of the benchmark trading strategy, the calculated risk vector, and the acquired real-time market and behavioral data are concatenated and combined according to a pre-defined format to form a current state vector that provides a complete snapshot of information for the agent's decision-making.

[0059] For example, the feedforward neural network can be implemented using a multilayer perceptron (MLP). The number of neurons in its input layer is equal to the sum of the parameter dimensions and historical performance metric dimensions of the benchmark trading strategy; it can be configured with three fully connected hidden layers, each containing 64 neurons, and using ReLU as the activation function; the number of neurons in the output layer is consistent with the number of trading sessions, and it outputs a multidimensional risk vector.

[0060] For example, the market clearing function in a simulation environment, which determines transaction results, can be implemented by simulating a unified clearing pricing mechanism. The specific steps include: In each trading session, the simulation environment collects all electricity sales declarations submitted by agents (sorted by price from low to high) to form an aggregate supply curve, and collects all electricity purchase declarations (sorted by price from high to low) to form an aggregate demand curve. The intersection of the aggregate supply curve and the aggregate demand curve is found; the price at this intersection is the market clearing price for that session, and the electricity volume at this intersection is the total market transaction volume. All electricity sellers with prices lower than or equal to the market clearing price, and all electricity buyers with prices higher than or equal to the market clearing price, are considered to have successfully completed their transactions. Based on the above principles, the actual transaction volume and actual transaction price (usually the unified market clearing price) for each agent in that session are determined.

[0061] In one possible implementation, in a pre-defined simulation environment containing multiple market participant agents, the current state vector of the target agent is constructed. Specifically, this includes: calculating the historical performance of the benchmark trading strategy and the load forecasting model using a pre-defined feedforward neural network to obtain a risk vector; acquiring market prices and behavioral data of competing agents from the simulation environment, where competing agents are other agents among the multiple market participant agents besides the target agent; and constructing the current state vector based on the benchmark trading strategy, risk vector, market price, and behavioral data.

[0062] In this embodiment, the current state vector refers to a structured numerical vector that contains all the preset input information used for the target agent's decision-making. This vector integrates the benchmark trading strategy, the risk vector, and real-time external market data into a numerical input that can be directly processed by its decision-making strategy network. For example, the vector can be a one-dimensional array, where the first 48 elements represent the declared electricity volume and price for the next 24 hours in the benchmark trading strategy, the following 24 elements represent the corresponding risk value, and the last part represents the real-time market price and competitor behavior data.

[0063] For example, the decision policy network can also be implemented using an MLP, where the number of neurons in the input layer matches the dimension of the current state vector, and the output layer is a single neuron using the sigmoid activation function to ensure that the output policy fusion coefficient is strictly between 0 and 1.

[0064] Specifically, this step generates a risk vector using a pre-trained feedforward neural network that serves as a risk assessment model. This network receives inputs from the declared electricity volume and price in the benchmark trading strategy, as well as historical performance indicators from the load forecasting model, and outputs a multi-dimensional risk vector corresponding to the trading period, where each dimension quantifies the expected trading risk for that period. Simultaneously, real-time market prices are collected from the simulation environment, and the trading behavior data of competing agents (excluding the target agent) in the previous round are observed. The parameters of the benchmark trading strategy (serving as the basis for the plan), the risk vector generated in the previous step, and the collected real-time market prices and the behavioral data of competing agents are concatenated and combined according to a preset order and format to ultimately construct the current state vector.

[0065] For example, the behavioral data of competing agents obtained from the simulation environment can be further processed into a set of statistical feature vectors. For instance, the declaration data of all other agents in the environment besides the target agent in the previous trading session can be statistically analyzed to calculate their average declaration price, total declared electricity volume, standard deviation of declaration price, and other statistical quantities. These statistical quantities can be combined to form a feature vector with a fixed dimension, which can be used as behavioral data reflecting the overall competitive situation of the market and input into the state vector of the target agent.

[0066] In one possible implementation, a risk vector is obtained by calculating the historical performance of the benchmark trading strategy and the load forecasting model through a pre-set feedforward neural network. Specifically, this includes using each predicted electricity volume, each predicted price, and the average absolute error value as input features of the feedforward neural network, where the average absolute error value is calculated by the load forecasting model on a reserved historical validation set; and using the feedforward neural network to perform forward propagation calculation on the input features to obtain the risk vector.

[0067] In this embodiment, a risk vector refers to a numerical sequence that corresponds one-to-one with each future trading session and is used to quantify the expected risk of a benchmark trading strategy. This vector is the output of the risk assessment model, and its core function is to provide the subsequent reinforcement learning agent with a clear and objective basis for judging the reliability of the benchmark trading strategy, transforming the abstract concept of "risk" into computable data. For example, a risk vector corresponding to the next 24 hours can be an array containing 24 elements, where the value of the 14th element is 0.85, indicating that the expected risk of executing the benchmark trading strategy during the 14:00-15:00 period is relatively high.

[0068] Specifically, this step prepares all the input features required by the feedforward neural network. This process includes: obtaining the predicted electricity volume and price for each future time period from the benchmark trading strategy, and acquiring a mean absolute error (MAE). This MAE is a single scalar value calculated by testing the historical prediction performance of the model on a historical validation set reserved during training, and is used to represent the overall reliability of the prediction model. The predicted electricity volume, price, and this MAE representing global reliability for each time period are combined into one or more input feature vectors. The combined input features are then fed into a pre-trained feedforward neural network, which serves as a risk assessment model. The network performs a forward propagation calculation, and the output of the network is the risk vector.

[0069] S204: Based on the current state vector, the policy fusion coefficient is calculated through the decision policy network inside the target intelligent body.

[0070] In this embodiment, the strategy fusion coefficient refers to a quantitative decision value output by the target agent, representing the trade-off between the benchmark trading strategy and the contingency trading strategy. This coefficient is not a direct trading instruction, but a scalar between 0 and 1. Its core function is to dynamically determine the proportion of high-yield and high-safety plans adopted in the final trading strategy, reflecting the agent's risk preference. For example, a strategy fusion coefficient of 0.8 indicates that the agent judges the current risk to be low, and the final trading strategy should consist of 80% from the benchmark trading strategy and 20% from the contingency trading strategy.

[0071] Specifically, this step takes the current state vector, constructed in the previous step and containing information such as the benchmark trading strategy, risk vector, and real-time market data, and inputs it as a whole into the decision-making strategy network inside the target intelligent body. This decision-making strategy network is a deep neural network trained using reinforcement learning, whose weights and bias parameters have learned the complex mapping relationship from state information to optimal decisions. After receiving the current state vector, the network performs a forward propagation calculation, that is, it transforms the values ​​in the state vector through nonlinear transformations of each layer of the network, ultimately obtaining a single scalar with a value between 0 and 1 at the network's output layer. This output scalar is the strategy fusion coefficient.

[0072] S205: Based on the strategy fusion coefficient, the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy are weighted and calculated with the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy to obtain the final trading strategy.

[0073] In this embodiment, the final trading strategy refers to a trading scheme generated through intelligent dynamic fusion that will ultimately be submitted to the electricity market for execution. This strategy does not directly select either a high-yield benchmark trading strategy or a high-security emergency trading strategy, but rather represents an optimized compromise between the two. Its specific composition is determined by the strategy fusion coefficient output by the reinforcement learning agent based on real-time risk assessment. For example, if the benchmark trading strategy requests 100MW of electricity, the emergency strategy requests 20MW, and the fusion coefficient is 0.7, then the requested electricity for that period in the final trading strategy will be 76MW (100MW × 0.7 + 20MW × (1 - 0.7)).

[0074] Specifically, this step calculates for each trading period within the future forecast period. For any given trading period, the submitted electricity volume and price in the benchmark trading strategy, as well as the submitted electricity volume and price in the emergency trading strategy, are obtained. Using the strategy fusion coefficient (denoted by α) obtained in the previous step, the final submitted electricity volume and price for that period are calculated using the following weighted calculation formula: Final submitted electricity volume = (α × benchmark submitted electricity volume) + ((1-α) × emergency submitted electricity volume); Final submitted price = (α × benchmark submitted price) + ((1-α) × emergency submitted price). The final submitted electricity volume and final submitted price calculated for all trading periods are combined sequentially to form the final trading strategy.

[0075] S206: Execute the final trading strategy in a simulation environment to obtain a reward signal, and update the adjustable parameters in the decision strategy network using a reinforcement learning algorithm and the reward signal.

[0076] For example, the final trading strategy is input into a simulation environment, which determines the actual trading volume and price based on built-in market clearing rules, and calculates the trading profit and the deviation penalty based on the discrepancy between the declared and actual trading volumes. The trading profit, representing opportunity, and the deviation penalty, representing risk, are combined into a single scalar reward signal to quantify the strategy's performance. A strategy gradient algorithm is employed, which calculates the gradient of each adjustable parameter in the decision-making strategy network (i.e., the connection weights between neurons and the bias parameters of each neuron) with respect to the previous decision. This gradient is then weighted using the reward signal to determine the direction and magnitude of parameter adjustments, thereby updating the decision-making network.

[0077] In one possible implementation, the final trading strategy is executed in a simulation environment to obtain a reward signal. Adjustable parameters in the decision-making strategy network are then updated using a reinforcement learning algorithm and the reward signal. Specifically, this includes: inputting the final trading strategy into the simulation environment to determine the actual transaction volume and price; calculating the trading revenue based on the actual transaction volume and price; calculating a deviation penalty based on the deviation between the declared volume and the actual transaction volume in the final trading strategy, according to a preset deviation penalty rule; combining the trading revenue and the deviation penalty to obtain the reward signal; and updating the adjustable parameters, including weight parameters and bias parameters, using a policy gradient algorithm based on the reward signal.

[0078] In this embodiment, the reward signal refers to a scalar feedback value used to quantitatively evaluate the performance of the final trading strategy in the previous step. This signal is the core basis for guiding the reinforcement learning agent to learn and optimize. It combines the two opposing financial results—trading gains representing opportunities and deviation penalties representing risks—into a single comprehensive score. For example, if the gain of a trade is 1000 yuan and the deviation penalty is 200 yuan, the combined reward signal can be 800; if the gain is 1000 yuan and the penalty is 1500 yuan, the reward signal is -500. This negative value will directly cause the adjustable parameters of the network to be adjusted in a direction that reduces the probability of making that decision in the previous step when the network's adjustable parameters are updated using a reinforcement learning algorithm (such as the policy gradient algorithm).

[0079] Specifically, this step submits the final trading strategy to the simulation environment, where the environment determines the actual trading volume and price for each time period based on built-in market clearing rules. Based on this result, the trading profit is obtained by multiplying the actual trading volume by the price, and a deviation penalty is calculated based on the difference between the declared volume and the actual trading volume, according to a preset deviation penalty rule. The calculated trading profit and deviation penalty are then mathematically combined (e.g., profit minus penalty) to construct a reward signal. A policy gradient algorithm is used to update the adjustable parameters in the decision-making policy network based on this reward signal. The update process here refers to the algorithm calculating the gradient of each adjustable parameter in the network (i.e., the connection weights between neurons and the bias parameters of each neuron) in relation to the previous decision, weighting this gradient using the reward signal to determine the adjustment direction and magnitude of each parameter, and completing the update.

[0080] In one possible implementation, the reward signal is obtained by combining the transaction gains with the deviation penalty. Specifically, this includes: calculating the net gain of the target agent, where the net gain equals the transaction gains minus the deviation penalty; obtaining the average net gain of the competing agents in each transaction period; and weighting the net gain with the average net gain to generate the reward signal.

[0081] In this embodiment, the reward signal refers to a comprehensive feedback value that includes not only the agent's own profit information but also the profit information of other competitors in the market. This signal is used to compare the target agent's own net profit with the average net profit of other competing agents in the environment to generate a final reward signal that reflects its relative performance in the market. For example, it can be constructed using the function "Reward Signal = (Any Net Profit) - β × (Average Net Profit of Competing Agents)", where β is a weighting coefficient. In this way, even if the agent's own net profit is positive, a negative reward signal may still be generated if it is lower than the average level of competitors.

[0082] Specifically, this step calculates the target agent's net profit, which is the value obtained by subtracting the deviation penalty from the transaction profit calculated in the previous step. It represents the target agent's absolute profit or loss during that transaction period. Simultaneously, it obtains the net profit data of all other competing agents during that transaction period from the simulation environment and calculates an arithmetic average of these data to obtain the average net profit of the competing agents, representing the average profitability level of the current market. The calculated target agent's own net profit is then weighted and summed with the average net profit to generate the final reward signal. One specific weighting method involves subtracting a product obtained by multiplying the average net profit by a preset weighting coefficient β from the target agent's own net profit.

[0083] In one possible implementation, the adjustable parameters are updated based on the reward signal using a policy gradient algorithm. Specifically, this includes: calculating the policy gradient of the policy fusion coefficients output by the decision policy network; weighting the policy gradient using the reward signal to generate the update amount for each adjustable parameter; and adjusting each adjustable parameter in the decision policy network according to the update amount.

[0084] In this embodiment, the policy gradient is a vector that corresponds one-to-one with all adjustable parameters in the decision policy network. This vector represents the steepest ascent direction required to adjust each adjustable parameter to increase the probability of the policy fusion coefficient output in the previous step. The policy gradient is obtained by performing differential operations on the decision policy network, and its core function is to provide a precise and quantitative adjustment "direction map" for optimizing network parameters. For example, if the agent outputs a decision of 0.7 and receives a positive reward, the calculated policy gradient indicates how to fine-tune each parameter in the network to increase the probability of outputting 0.7 in the same state in the next step.

[0085] Specifically, this update step calculates the policy gradient of the decision policy network when it outputs the policy fusion coefficients, based on the current state vector of the target agent when it outputs the policy fusion coefficients. The scalar reward signal obtained in the previous step is then weighted with the calculated policy gradient vector. This weighting is typically implemented as a scalar-vector multiplication, resulting in an update amount with the same dimension as the adjustable parameters in the network, but containing the adjustment direction and magnitude. This update amount is applied to each adjustable parameter in the decision policy network. Specifically, the adjustable parameters refer to the connection weights between neurons constituting the neural network and the bias parameters of each neuron. In this way, one iterative update of the decision policy network is completed.

[0086] For example, the condition for determining whether the decision policy network has converged can be specifically set as follows: during the training process, continuously monitor the average reward signal obtained by the target agent in multiple consecutive rounds (e.g., the most recent 100 training rounds). When the moving average curve of the average reward signal value tends to flatten, or its growth rate in the most recent N (N is a positive integer) rounds is lower than a preset small threshold (e.g., 0.1%), it can be determined that the decision policy network has reached a convergent state, and the cyclic training is terminated.

[0087] Figure 3 This is a schematic diagram of a module of a power trading strategy risk management system based on reinforcement learning, as described in an embodiment of this application. This system can be implemented through software, hardware, or a combination of both, forming all or part of the overall system. Figure 3 As shown, the system includes: a policy generation module 301, an intelligent decision-making module 302, and a reinforcement learning module 303, wherein: The strategy generation module 301 is configured to acquire historical load data and weather forecast data of the power trading entity, input the historical load data and weather forecast data into the preset load forecast model for processing to obtain load forecast data, and generate a benchmark trading strategy by using the load forecast data through a constraint optimization algorithm. The benchmark trading strategy includes the predicted declared electricity volume and predicted declared price for each trading period. The strategy generation module 301 is also configured to determine emergency trading strategies based on historical forecast error data of the load forecasting model; The intelligent decision-making module 302 is configured to construct the current state vector of the target intelligent agent in a preset simulation environment containing multiple market participant intelligent agents. The target intelligent agent is any one of the multiple market participant intelligent agents. The simulation environment has the function of determining the transaction result by market clearing according to the trading strategies of multiple market participant intelligent agents, and the function of calculating penalties according to the deviation between the transaction result and the trading strategy. The intelligent decision-making module 302 is also configured to calculate the policy fusion coefficient based on the current state vector through the decision policy network inside the target intelligent body; The intelligent decision-making module 302 is also configured to perform weighted calculations on the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy and the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy, based on the strategy fusion coefficient, to obtain the final trading strategy; The reinforcement learning module 303 is configured to execute the final trading strategy in a simulation environment, obtain a reward signal, and update the adjustable parameters in the decision strategy network through the reinforcement learning algorithm and the reward signal.

[0088] Based on the above embodiments, as an optional embodiment, the strategy generation module 301 is further configured to: The load forecast data for each trading period is used as the boundary constraint for each forecasted electricity volume, and the upper and lower limits of power, power ramp rate, capacity status of energy storage units and boundary constraints of each distributed energy source within the power trading entity are constructed as constraint conditions. A function for expected transaction revenue is constructed using the predicted declared electricity volume and the predicted declared price as variables. Under constraints, the maximum value of the expected trading revenue function is obtained to get the predicted bid volume and predicted bid price for each trading period. The predicted bid volume and predicted bid price are then combined to obtain the benchmark trading strategy.

[0089] Based on the above embodiments, as an optional embodiment, the strategy generation module 301 is further configured to: By comparing the historical load forecast data of the power trading entities with the corresponding historical actual load data, all historical trading days with historical forecast error data greater than the preset error threshold are filtered out to obtain the error trading day set. For each trading session, extract the actual load data for all historical trading days in the error trading day set for each corresponding trading session; The average load value for each trading period is obtained by averaging the actual load data. All average load values ​​are combined according to time sequence to form an average high-error load curve; The average high-error load curve is multiplied by a preset conservative coefficient to generate emergency power reporting. A preset conservative bid price is matched to the emergency bid volume, and an emergency trading strategy is formed by combining the emergency bid volume.

[0090] Based on the above embodiments, as an optional embodiment, the strategy generation module 301 is further configured to: When the emergency electricity demand is non-negative, the conservative demand price will be set as the minimum allowable price in the electricity market. When the emergency electricity demand is negative, the conservative demand price will be set as the highest permissible price in the electricity market.

[0091] Based on the above embodiments, as an optional embodiment, the intelligent decision-making module 302 is further used for: The risk vector is obtained by calculating the historical performance of the benchmark trading strategy and the load forecasting model through a pre-defined feedforward neural network. Market prices and behavioral data of competing agents are obtained from a simulation environment. Competing agents are other agents among multiple market participants besides the target agent. Based on the benchmark trading strategy, risk vector, market price, and behavioral data, the current state vector is constructed.

[0092] Based on the above embodiments, as an optional embodiment, the intelligent decision-making module 302 is further used for: Each predicted electricity volume, each predicted price, and the average absolute error value are used together as input features of the feedforward neural network. The average absolute error value is calculated by the load forecasting model on the reserved historical validation set. The risk vector is obtained by forward propagation of the input features through a feedforward neural network.

[0093] Based on the above embodiments, as an optional embodiment, the reinforcement learning module 303 is further used for: The final trading strategy is input into the simulation environment to determine the actual transaction volume and the actual transaction price. The transaction revenue is calculated based on the actual transaction volume and the actual transaction price. The deviation penalty is calculated based on the difference between the declared electricity volume and the actual transaction electricity volume in the final trading strategy, and according to the preset deviation penalty rules. By combining trading profits with deviation penalties, a reward signal is obtained; The adjustable parameters, including weight parameters and bias parameters, are updated based on the reward signal using the policy gradient algorithm.

[0094] Based on the above embodiments, as an optional embodiment, the reinforcement learning module 303 is further used for: Calculate the net profit of the target agent. The net profit equals the transaction profit minus the bias penalty. Obtain the average net profit of competing agents in each trading session; The net profit is weighted and calculated with the average net profit to generate a reward signal.

[0095] Based on the above embodiments, as an optional embodiment, the reinforcement learning module 303 is further used for: Calculate the policy gradient of the policy fusion coefficients output by the decision policy network; The policy gradient is weighted using the reward signal to generate the update values ​​for each adjustable parameter; Adjust the adjustable parameters in the decision-making strategy network based on the update amount.

[0096] It should be noted that the system provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0097] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the disclosure in this specification. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope of this application is defined by the claims.

Claims

1. A risk management method for power trading strategies based on reinforcement learning, characterized in that, Applied to a risk management platform, the method includes: Historical load data and weather forecast data of the power trading entity are obtained, and the historical load data and weather forecast data are input into a preset load forecasting model for processing to obtain load forecast data. The load forecast data is then used to generate a benchmark trading strategy through a constrained optimization algorithm. The benchmark trading strategy includes the predicted electricity volume and predicted price for each trading period. Based on the historical forecasting error data of the load forecasting model, an emergency trading strategy is determined; Repeat the following steps until the decision policy network of the target agent converges: In a preset simulation environment containing multiple market participant agents, the current state vector of the target agent is constructed. The target agent is any one of the multiple market participant agents. The simulation environment has the function of determining the transaction result by market clearing according to the trading strategies of the multiple market participant agents, and the function of calculating penalties according to the deviation between the transaction result and the trading strategy. Based on the current state vector, the policy fusion coefficient is calculated through the decision policy network inside the target intelligent body; Based on the strategy fusion coefficient, the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy are weighted and calculated with the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy to obtain the final trading strategy; The final trading strategy is executed in the simulation environment to obtain a reward signal. The adjustable parameters in the decision strategy network are then updated using a reinforcement learning algorithm and the reward signal.

2. The method according to claim 1, characterized in that, The step of generating a benchmark trading strategy from the load forecast data using a constrained optimization algorithm specifically includes: The load forecast data for each trading period is used as the boundary constraint for each forecasted electricity volume, and the upper and lower limits of power, power ramp rate, capacity status of energy storage units, and the boundary constraints of each distributed energy source within the power trading entity are constructed as constraint conditions. A function for expected transaction revenue is constructed using the predicted electricity volume and the predicted price as variables. Under the constraints, the maximum value of the expected trading return function is calculated to obtain the predicted declared electricity volume and the predicted declared price for each trading period. The predicted declared electricity volume and the predicted declared price are then combined to obtain the benchmark trading strategy.

3. The method according to claim 1, characterized in that, The step of determining the emergency trading strategy based on the historical forecasting error data of the load forecasting model specifically includes: By comparing the historical load forecast data of the power trading entity with the corresponding historical actual load data, all historical trading days with historical forecast error data greater than a preset error threshold are filtered out to obtain a set of error trading days. For each of the aforementioned trading periods, extract the actual load data for all historical trading days in the error trading day set for each corresponding trading period; The average load value for each of the actual load data is calculated by averaging the actual load data. All the average load values ​​are combined in time sequence to form an average high-error load curve; The average high-error load curve is multiplied by a preset conservative coefficient to generate emergency power reporting. A preset conservative bid price is matched to the emergency bid electricity, and the emergency bid electricity is combined to form the emergency transaction strategy.

4. The method according to claim 3, characterized in that, The process of matching the emergency electricity demand with a preset conservative demand price specifically includes: When the emergency electricity demand is non-negative, the conservative demand price is set to the minimum allowable price in the electricity market. When the emergency electricity demand is negative, the conservative demand price is set to the highest permissible price in the electricity market.

5. The method according to claim 1, characterized in that, The step of constructing the current state vector of the target agent in a preset simulation environment containing multiple market participant agents specifically includes: The risk vector is obtained by calculating the historical performance of the benchmark trading strategy and the load prediction model through a preset feedforward neural network. Market prices and behavioral data of competing agents are obtained from the simulation environment. The competing agents are other agents among the multiple market participant agents besides the target agent. The current state vector is constructed based on the benchmark trading strategy, the risk vector, the market price, and the behavioral data.

6. The method according to claim 5, characterized in that, The step of calculating the risk vector by using a preset feedforward neural network to analyze the historical performance of the benchmark trading strategy and the load forecasting model specifically includes: The predicted electricity volume, the predicted price, and the average absolute error value are used together as the input features of the feedforward neural network. The average absolute error value is calculated by the load forecasting model on the reserved historical validation set. The risk vector is obtained by forward propagation of the input features through the feedforward neural network.

7. The method according to claim 1, characterized in that, The step of executing the final trading strategy in the simulation environment to obtain a reward signal, and updating the adjustable parameters in the decision-making strategy network using a reinforcement learning algorithm and the reward signal, specifically includes: The final trading strategy is input into the simulation environment to determine the actual transaction volume and the actual transaction price. The transaction revenue is calculated based on the actual transaction volume and the actual transaction price. Based on the deviation between the declared electricity volume in the final trading strategy and the actual transaction electricity volume, and according to the preset deviation penalty rules, the deviation penalty is calculated. The reward signal is obtained by combining the transaction profit with the deviation penalty; The adjustable parameters, including weight parameters and bias parameters, are updated based on the reward signal using a policy gradient algorithm.

8. The method according to claim 7, characterized in that, The step of combining the transaction profit with the deviation penalty to obtain the reward signal specifically includes: Calculate the net profit of the target agent, where the net profit equals the transaction profit minus the deviation penalty; Obtain the average net profit of the competing intelligent agent in each of the transaction periods; The reward signal is generated by weighting the net profit with the average net profit.

9. The method according to claim 7, characterized in that, The step of updating the adjustable parameters based on the reward signal using a policy gradient algorithm specifically includes: Calculate the policy gradient of the policy fusion coefficients output by the decision policy network; The policy gradient is weighted using the reward signal to generate update values ​​for each of the adjustable parameters; The adjustable parameters in the decision strategy network are adjusted according to the update amount.

10. A risk management system for power trading strategies based on reinforcement learning, characterized in that, The system, which performs the method as described in any one of claims 1-9, comprises a policy generation module, an intelligent decision-making module, and a reinforcement learning module, wherein: The strategy generation module is configured to acquire historical load data and weather forecast data of the power trading entity, input the historical load data and weather forecast data into a preset load forecasting model for processing to obtain load forecast data, and generate a benchmark trading strategy by using the load forecast data through a constraint optimization algorithm. The benchmark trading strategy includes the predicted declared electricity volume and predicted declared price for each trading period. The strategy generation module is also configured to determine emergency trading strategies based on historical prediction error data of the load forecasting model. The intelligent decision-making module is configured to construct the current state vector of a target intelligent agent in a preset simulation environment containing multiple market participant intelligent agents. The target intelligent agent is any one of the multiple market participant intelligent agents. The simulation environment has the function of determining the transaction result by market clearing according to the trading strategies of the multiple market participant intelligent agents, and the function of calculating penalties according to the deviation between the transaction result and the trading strategy. The intelligent decision-making module is also configured to calculate the policy fusion coefficient based on the current state vector through the decision policy network inside the target intelligent body; The intelligent decision-making module is also configured to perform weighted calculations on the declared electricity volume and declared price for each corresponding trading period in the benchmark trading strategy and the declared electricity volume and declared price for each corresponding trading period in the emergency trading strategy, based on the strategy fusion coefficient, to obtain the final trading strategy; The reinforcement learning module is configured to execute the final trading strategy in the simulation environment to obtain a reward signal, and update the adjustable parameters in the decision strategy network through the reinforcement learning algorithm and the reward signal.

Images