Reinforcement learning action space dimensionality reduction method, system, device, and medium

Abstract

The application discloses a reinforcement learning action space dimension reduction method, system, device and medium, comprehensively considers various adjustable resources in a new power system, constructs a plurality of meta-action spaces for reinforcement learning training of power grid real-time scheduling plan arrangement, discretizes the plurality of meta-action spaces for continuous action space, obtains a plurality of discretized meta-action spaces, considers the adjustment characteristics of various adjustable resources, optimizes and combines the discretized meta-action spaces according to a set rule and algorithm, forms a macro-action space, shields illegal action space for the macro-action space, obtains a shielded macro-action space, carries out reinforcement learning agent training based on the shielded macro-action space, verifies the rationality of the shielded macro-action space in the training process, if the shielded macro-action space is reasonable, generates a reinforcement learning training action space, otherwise returns to the continuous action space discretization step and carries out cyclic iteration until the reinforcement learning training action space is generated.

Description

Technical Field

[0001] This invention belongs to the field of power dispatch automation technology, specifically relating to a method, system, device, and medium for dimensionality reduction of reinforcement learning action space. Background Technology

[0002] The scale of power grid operation and control in new power systems is growing exponentially, and the controlled objects are shifting from traditional conventional generating units to various adjustable resources on both the source and load sides. In traditional real-time dispatch planning, the objects of real-time dispatch planning are mostly large-capacity conventional generating units with continuous regulation and control capabilities, employing a centralized control mode. With the massive integration of distributed renewable energy, electric vehicles, and energy storage, to meet the needs of bidirectional power flow and diverse loads, the objects of regulation have shifted from single conventional energy generating units to multi-electrical quantity regulation involving flexibly modified units, renewable energy, energy storage, pumped storage, and electric vehicles. In actual operation, due to factors such as load forecast accuracy, weather changes, transmission line overload tripping, and temporary equipment failures, the planning needs to be rapidly adjusted to cope with uncertainties. Therefore, as the scale of controlled objects increases, reinforcement learning algorithms are gradually being adopted for real-time power grid dispatch planning. The solution space of a reinforcement learning algorithm is the product of the action space and the state space, and its solution complexity is determined by both the action space and the state space. Without considering the state space, the continuous action space will form an infinitely large solution space, which is the fundamental reason for the poor stability and performance of reinforcement learning algorithms in training continuous control tasks.

[0003] Chinese patent application CN115940294A, entitled "Multi-level Power Grid Real-time Dispatch Strategy Adjustment Method, System, Equipment, and Storage Medium," proposes a method for adjusting a multi-level power grid real-time dispatch strategy. The method includes: dividing the power grid into several zones according to tie-line cross-sections; obtaining the adjustable unit adjustment range and adjustable load adjustment range of each zone; obtaining transmission power prediction information for each zone; and, based on the transmission power prediction information of each zone, calling a preset zone transmission line transmission prediction model to obtain the predicted transmission power between zones; and, based on the adjustable unit adjustment range, adjustable load adjustment range, and predicted transmission power between zones, calling a preset dispatch reinforcement learning model to arrange the real-time dispatch strategy for each zone, thus obtaining the real-time dispatch strategy for each zone.

[0004] The method for reducing the dimensionality of the high-dimensional action space of large-scale multi-level power grids proposed in the above patent applications reduces the computational scope of the power grid by dividing the dispatching range of the multi-level power grid, thereby reducing the dimensionality of the action space. Its action space only considers the active power adjustments of conventional adjustable generating units and new energy adjustable generating units. With the construction of new power systems, various adjustable resources such as distributed new energy, electric vehicles, energy storage, and pumped storage can participate in the power grid optimization dispatching process. The method in this patent application does not consider the participation of various adjustable resources in dispatching plan arrangement and action space dimensionality reduction. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device and medium for dimensionality reduction of reinforcement learning action space, so as to overcome the defects of the existing technology. In the context of new power system, this invention considers the participation of various adjustable resources in real-time planning and scheduling, thereby improving the training speed of agents and the real-time planning and scheduling speed.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The reinforcement learning action space dimensionality reduction method includes the following steps:

[0008] Taking into account various adjustable resources in the new power system, a number of meta-action spaces are constructed for reinforcement learning training of real-time power grid dispatching plan arrangement.

[0009] By discretizing the continuous action space for several meta-action spaces, several discretized meta-action spaces are obtained.

[0010] Considering the adjustment characteristics of various adjustable resources, the discretized meta-action space is optimized and combined according to the set rules and algorithms to form the macro-action space;

[0011] By masking illegal action spaces in the macro action space, a masked macro action space is obtained.

[0012] The reinforcement learning agent is trained based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, the process returns to the continuous action space discretization step and iterates until the reinforcement learning training action space is generated.

[0013] Furthermore, the meta-action space includes adjustable resources on the source side and the load side;

[0014] The adjustable resources on the source side include conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power.

[0015] The adjustable resources on the load side include distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles.

[0016] Furthermore, the step of discretizing the continuous action space for several meta-action spaces to obtain several discretized meta-action spaces is specifically as follows:

[0017] The degree of discretization of the action space for adjustable resources on the source side is determined based on the unit's ramp rate, as shown in the following formula:

[0018] Δp1=[ω1·ρ]

[0019] Where Δp1 is the discrete granularity of the source side, in MW, ρ is the unit ramp rate, in MW / min, ω1 is the empirical coefficient of the source side, and [] indicates rounding;

[0020] The source-side discrete granularity satisfies the following constraints:

[0021]

[0022] in, This is the upper limit of the unit's output. The lower limit of the unit's output is α1, which is the discrete empirical coefficient on the source side.

[0023] The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula:

[0024] Δp2=[ω2·λ]

[0025] Where Δp2 is the load-side discrete granularity in MW, λ is the adjustable load response rate in MW / min, and ω2 is the load-side empirical coefficient.

[0026] If the adjustable resources on the load side can only be reduced, the discrete granularity of the load side must satisfy the following constraints:

[0027]

[0028] If the adjustable resources on the load side can be adjusted both vertically and horizontally, the discrete granularity of the load side satisfies the following constraints:

[0029]

[0030] Where P is the current value of the adjustable load, P max α2 represents the maximum adjustable load value, and α2 is the empirical coefficient for load-side dispersion.

[0031] Furthermore, considering the adjustment characteristics of various adjustable resources, the discretized meta-action space is optimized and combined according to the set rules and algorithms to form the macro-action space, specifically as follows:

[0032] Different macro-action spaces are constructed according to the positive and negative values ​​of the power generation and consumption imbalance.

[0033] If the power generation and consumption imbalance is positive, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the upper limit of output, pumped storage power stations that are generating power but have not reached the upper limit of output, energy storage loads that have not reached the upper limit of capacity, and electric vehicle charging piles that are charging.

[0034] If the power generation and consumption imbalance is negative, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging.

[0035] The formula for calculating the power generation-consumption imbalance is as follows:

[0036]

[0037] in, The total load forecast for the next time period. The projected total output of new energy sources for the next time period. ΔPunbalance represents the total output of the conventional units at this moment. t+1 This represents the power generation and consumption imbalance at the next moment.

[0038] Furthermore, the illegal action space masking of the macro action space to obtain the masked macro action space specifically involves:

[0039] Illegal actions are spatially shielded based on the response methods of source-side and load-side adjustable resources;

[0040] The response methods of the source-side adjustable resources include:

[0041] 1) Conventional thermal power plants can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0042] 2) Conventional hydropower can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0043] 3) When the pumped storage power station is in pumping mode, the power output of the power station is constant and cannot be adjusted, and there is no pumping-power generation state transition;

[0044] 4) Centralized wind power and centralized photovoltaic systems can adjust unit output according to response targets;

[0045] The response methods of the load-side adjustable resources include:

[0046] 1) Some energy storage loads only suspend their current charging / discharging state and do not switch between charging and discharging states;

[0047] 2) Distributed wind power and distributed photovoltaic power can reduce the output of some or all units according to the response target;

[0048] 3) Electric vehicle charging stations only pause the current charging / discharging state and do not switch between charging and discharging states.

[0049] Furthermore, the verification of the rationality of the shielded macro action space specifically includes:

[0050] The validity of the masked macro action space is verified based on the crash rate of the agent during the maximum number of training rounds. If the crash rate does not exceed the preset value, the verification is valid; otherwise, the verification is invalid.

[0051] Reinforcement learning action space dimensionality reduction systems include:

[0052] Meta-action space construction module: used to comprehensively consider various adjustable resources in the new power system and construct several meta-action spaces for reinforcement learning training of real-time power grid dispatching plan arrangement;

[0053] Discretization module: used to discretize the continuous action space for several meta-action spaces to obtain several discretized meta-action spaces;

[0054] Macro Action Space Construction Module: This module takes into account the adjustment characteristics of various adjustable resources and optimizes and combines the discretized meta action spaces according to the set rules and algorithms to form the macro action space.

[0055] Shielding module: Used to shield illegal action space in the macro action space, resulting in shielded macro action space;

[0056] Training module: Used to train reinforcement learning agents based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, it returns to the discretization module and performs iterative loops until the reinforcement learning training action space is generated.

[0057] Furthermore, in the meta-action space construction module, the meta-action space includes adjustable resources on the source side and the load side;

[0058] The adjustable resources on the source side include conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power.

[0059] The adjustable resources on the load side include distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles.

[0060] Furthermore, in the discretization module, continuous action spaces are discretized for several meta-action spaces to obtain several discretized meta-action spaces, specifically as follows:

[0061] The degree of discretization of the action space for adjustable resources on the source side is determined based on the unit's ramp rate, as shown in the following formula:

[0062] Δp1=[ω1·ρ]

[0063] Where Δp1 is the discrete granularity of the source side, in MW, ρ is the unit ramp rate, in MW / min, ω1 is the empirical coefficient of the source side, and [] indicates rounding;

[0064] The source-side discrete granularity satisfies the following constraints:

[0065]

[0066] in, This is the upper limit of the unit's output. The lower limit of the unit's output is α1, which is the discrete empirical coefficient on the source side.

[0067] The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula:

[0068] Δp2=[ω2·λ]

[0069] Where Δp2 is the load-side discrete granularity in MW, λ is the adjustable load response rate in MW / min, and ω2 is the load-side empirical coefficient.

[0070] If the adjustable resources on the load side can only be reduced, the discrete granularity of the load side must satisfy the following constraints:

[0071]

[0072] If the adjustable resources on the load side can be adjusted both vertically and horizontally, the discrete granularity of the load side satisfies the following constraints:

[0073]

[0074] Where P is the current value of the adjustable load, P max α2 represents the maximum adjustable load value, and α2 is the empirical coefficient for load-side dispersion.

[0075] Furthermore, in the macro-action space construction module, considering the adjustment characteristics of various adjustable resources, the discretized meta-action spaces are optimized and combined according to the set rules and algorithms to form the macro-action space, specifically as follows:

[0076] Different macro-action spaces are constructed according to the positive and negative values ​​of the power generation and consumption imbalance.

[0077] If the power generation and consumption imbalance is positive, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the upper limit of output, pumped storage power stations that are generating power but have not reached the upper limit of output, energy storage loads that have not reached the upper limit of capacity, and electric vehicle charging piles that are charging.

[0078] If the power generation and consumption imbalance is negative, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging.

[0079] The formula for calculating the power generation-consumption imbalance is as follows:

[0080]

[0081] in, The total load forecast for the next time period. The projected total output of new energy sources for the next time period. ΔPunbalance represents the total output of the conventional units at this moment. t+1 This represents the power generation and consumption imbalance at the next moment.

[0082] Furthermore, within the shielded space, the illegal action space is shielded from the macro-action space to obtain the shielded macro-action space, specifically as follows:

[0083] Illegal actions are spatially shielded based on the response methods of source-side and load-side adjustable resources;

[0084] The response methods of the source-side adjustable resources include:

[0085] 1) Conventional thermal power plants can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0086] 2) Conventional hydropower can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0087] 3) When the pumped storage power station is in pumping mode, the power output of the power station is constant and cannot be adjusted, and there is no pumping-power generation state transition;

[0088] 4) Centralized wind power and centralized photovoltaic systems can adjust unit output according to response targets;

[0089] The response methods of the load-side adjustable resources include:

[0090] 1) Some energy storage loads only suspend their current charging / discharging state and do not switch between charging and discharging states;

[0091] 2) Distributed wind power and distributed photovoltaic power can reduce the output of some or all units according to the response target;

[0092] 3) Electric vehicle charging stations only pause the current charging / discharging state and do not switch between charging and discharging states.

[0093] Furthermore, in the training module, verifying the rationality of the masked macro action space specifically involves:

[0094] The validity of the masked macro action space is verified based on the crash rate of the agent during the maximum number of training rounds. If the crash rate does not exceed the preset value, the verification is valid; otherwise, the verification is invalid.

[0095] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the reinforcement learning action space dimensionality reduction method.

[0096] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning action space dimensionality reduction method.

[0097] Compared with the prior art, the present invention has the following beneficial technical effects:

[0098] In the context of new power systems, this invention proposes a reinforcement learning action space dimensionality reduction method that considers the participation of various adjustable resources in real-time planning and scheduling, thereby improving the training speed of agents and the speed of real-time planning and scheduling.

[0099] The meta-action space constructed by this invention includes source-side conventional thermal power, conventional hydropower, pumped storage power plants, centralized wind power, centralized photovoltaic power, and load-side distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles. It is more adaptable to the transformation of the new power system from a centralized control mode to a decentralized control mode. The real-time power grid planning agent trained based on this action space generates a real-time power grid plan that is more adapted to the development of the new power system.

[0100] The continuous action space discretization method, macro-action space construction method, and shielded macro-action space construction method proposed in this invention can greatly reduce the dimensionality of the meta-action space, thereby greatly improving the training efficiency of reinforcement learning agents and meeting the requirements of real-time power grid planning for the speed and real-time performance of online training of agents. Attached Figure Description

[0101] The accompanying drawings are provided to further understand the invention and constitute a part of this invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0102] Figure 1 This is a schematic diagram of the reinforcement learning action space dimensionality reduction method of the present invention;

[0103] Figure 2 This is a schematic diagram of the reinforcement learning action space dimensionality reduction system of the present invention. Detailed Implementation

[0104] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0105] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0106] Example 1

[0107] This embodiment provides a method for dimensionality reduction of the action space in reinforcement learning, such as... Figure 1 As shown, it includes the following steps:

[0108] Taking into account various adjustable resources in the new power system, a number of meta-action spaces are constructed for reinforcement learning training of real-time power grid dispatching plan arrangement.

[0109] By discretizing the continuous action space for several meta-action spaces, several discretized meta-action spaces are obtained.

[0110] Considering the adjustment characteristics of various adjustable resources, the discretized meta-action space is optimized and combined according to the set rules and algorithms to form the macro-action space;

[0111] By masking illegal action spaces in the macro action space, a masked macro action space is obtained.

[0112] In the simulation environment, reinforcement learning agent training is carried out based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated; otherwise, the process returns to the continuous action space discretization step and iterates until the reinforcement learning training action space is generated.

[0113] Example 2

[0114] This embodiment provides a method for dimensionality reduction of the action space in reinforcement learning, which specifically includes the following steps:

[0115] (1) Taking into account various adjustable resources in the new power system, we construct several meta-action spaces for reinforcement learning training of real-time power grid dispatching plan arrangement.

[0116] Step (1) involves constructing the meta-action space for reinforcement learning training of real-time grid dispatching plans. Under the new power system's source-grid-load-storage coordinated control model, the meta-action space encompasses all adjustable resources on both the source and load sides. The source side includes conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power. Since the calculation cycle for real-time grid dispatching plans is every 15 minutes, the adjustable resources on the load side only include resources with a response rate within 15 minutes, including distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles. These action spaces are reinforcement learning meta-actions, i.e., indivisible basic actions.

[0117] (2) Discretize the continuous action space for the meta-action space to obtain several discretized meta-action spaces.

[0118] Step (2) involves discretizing the continuous action space. Since the real-time power grid dispatching plan is calculated every 15 minutes during the day, the start-up and shutdown of all adjustable units during this time period are not considered, nor are the pumped storage power stations' pumping-to-generation state transitions or the charging-to-discharging state transitions of electric vehicles. The action space consists entirely of power adjustments of adjustable resources, and is therefore a continuous action space. A continuous action space would result in an infinitely large solution space, which would slow down the training speed of the reinforcement learning agent. Therefore, discretizing the continuous space reduces the dimensionality of the action space, ensuring both control accuracy and computational efficiency.

[0119] The degree of discretization of the action space for adjustable resources on the source side is determined based on the unit's ramp rate, as shown in the following formula:

[0120] Δp1 = [ω1·ρ], where Δp1 is the discrete granularity in MW, ρ is the unit ramp rate in MW / min. Generally, thermal power units have a smaller ramp rate, so ω1 can be set relatively large to increase the degree of discretization. New energy units have a larger ramp rate, so ω1 can be set relatively small to increase the degree of discretization. [] indicates rounding. To improve the degree of discretization, all action spaces are integers. ω1 is the source-side empirical coefficient, which can be iteratively corrected in subsequent reinforcement learning agent training.

[0121] Meanwhile, in order to improve computational efficiency while ensuring control accuracy and solution rationality, it is necessary to set constraints on the discrete granularity: Where P unmax P is the upper limit of the unit's output. unmin The lower limit of the unit's output is α1, which is the discrete empirical coefficient on the source side. It can be set to 1 during the initial training and can be iteratively adjusted according to the training effect.

[0122] The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula:

[0123] Δp2 = [ω2·λ], where Δp2 is the load-side discrete granularity in MW, and λ is the adjustable load response rate in MW / min. For adjustable resources with faster response rates, ω2 can be set relatively smaller to increase the degree of discretization. [] indicates rounding. To improve the degree of discretization, all action spaces are integers. ω2 is an empirical coefficient on the load side, which can be iteratively corrected during subsequent reinforcement learning agent training.

[0124] Meanwhile, in order to improve computational efficiency while ensuring control accuracy and solution rationality, constraints need to be set on the discrete granularity. If it is a resource that can only be reduced, the constraint is as follows: If the resource is adjustable both upwards and downwards, the constraint is: Where P is the current value of the adjustable load, and α2 is the discrete empirical coefficient of the load side, which can be taken as 1 during the initial training and can be gradually iterated and corrected according to the training effect.

[0125] (3) Considering the adjustment characteristics of various resources, the discretized meta-action space is optimized and combined according to the set rules and algorithms to form the macro-action space.

[0126] The action spaces mentioned in the above steps are all meta-action spaces, which are the power adjustment actions of all adjustable resources. In the actual real-time scheduling of power grid dispatch, it is necessary to consider the adjustment characteristics of various adjustable resources. Therefore, the meta-action spaces can be optimized and combined according to the set rules and algorithms to form macro-action spaces, thereby further reducing the dimensionality of the action space.

[0127] Since the adjustment characteristics of various adjustment resources are different within 15 minutes (see Table 1 and Table 2), different macro-action spaces can be constructed according to the power generation and consumption imbalance.

[0128] Table 1. Response characteristics of adjustable resources on the source side within 15 minutes.

[0129]

[0130]

[0131] Table 2 Response characteristics of load-side adjustable resources within 15 minutes

[0132]

[0133] The formula for the power generation and consumption imbalance is as follows:

[0134] The total load forecast for the next time period. The projected total output of new energy sources for the next time period. ΔPunbalance represents the total output of the conventional units at this moment. t+1 The power generation and consumption imbalance at the next moment needs to be determined based on ΔpPunbalance. t+1 To develop a power generation plan for the next moment.

[0135] If ΔPunbalance t+1 A positive value indicates that the overall load level of the power grid will increase in the next moment. In order to meet the power grid's power generation and consumption balance, when arranging the power generation plan for the next moment, the source-side action space should only include units with upward regulation characteristics, and the load-side action space should only include units with downward regulation characteristics. Therefore, the macro action space in this scenario includes conventional thermal power and conventional hydropower that have not reached their output limit, pumped storage power stations that are generating power but have not reached their output limit, energy storage loads that have not reached their capacity limit, and electric vehicle charging piles that are charging.

[0136] If ΔPunbalance t+1 A negative value indicates that the overall load level of the power grid will decrease in the next moment. In order to meet the power grid's power generation and consumption balance, when arranging the power generation plan for the next moment, the source-side action space should only include units with downward regulation characteristics, and the load-side action space should only include units with upward regulation characteristics. Therefore, the macro action space in this scenario includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging.

[0137] If ΔPunbalancet+1 Set to 0, maintain the status quo.

[0138] (4) The illegal action space is masked in the macro action space to obtain the masked macro action space.

[0139] In conventional power grid real-time dispatching program training using reinforcement learning, various illegal action constraints are used as relaxation constraints in the reinforcement learning reward function to help the agent make learning decisions. This process increases the number of training rounds and training time. Therefore, this invention proposes to fully consider various illegal action constraints when constructing the action space to improve subsequent training efficiency. The constraints of various regulation resources are shown in the response mode columns in Tables 1 and 2, which consider both output upper and lower limit constraints and state transition constraints.

[0140] (5) In the simulation environment, reinforcement learning agent training is carried out based on the shielded macro action space. During the training process, the rationality of the shielded macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, the continuous action space discretization step is returned and iterated until the reinforcement learning training action space is generated.

[0141] To ensure that the reduced action space can meet the requirements of agent training stability and performance, the rationality of the action space needs to be verified. The evaluation criterion is the crash rate of the agent in the maximum number of training rounds. The crash rate refers to the power flow not converging, the power grid imbalance is too large, the line is overloaded and disconnected, resulting in island operation, the agent's decision does not meet the constraints, etc. Generally, the crash rate does not exceed 5%, which means that the action space setting is considered reasonable. Otherwise, the empirical coefficients (source side empirical coefficient, source side discrete empirical coefficient, load side empirical coefficient and load side discrete empirical coefficient) in step (2) can be iteratively adjusted.

[0142] Example 3

[0143] This embodiment provides a reinforcement learning action space dimensionality reduction system, see [link / reference] Figure 2 ,include:

[0144] Meta-action space construction module: used to comprehensively consider various adjustable resources in the new power system and construct several meta-action spaces for reinforcement learning training of real-time power grid dispatching plan arrangement;

[0145] The meta-action space includes adjustable resources on both the source and load sides;

[0146] The adjustable resources on the source side include conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power.

[0147] The adjustable resources on the load side include distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles.

[0148] Discretization module: Used to discretize the continuous motion space of several meta-motion spaces, resulting in several discretized meta-motion spaces; specifically, the source-side adjustable resources determine the degree of motion space discretization based on the unit's ramp rate, as shown in the following formula:

[0149] Δp1=[ω1·ρ]

[0150] Where Δp1 is the discrete granularity of the source side, in MW, ρ is the unit ramp rate, in MW / min, ω1 is the empirical coefficient of the source side, and [] indicates rounding;

[0151] The source-side discrete granularity satisfies the following constraints:

[0152]

[0153] in, This is the upper limit of the unit's output. The lower limit of the unit's output is α1, which is the discrete empirical coefficient on the source side.

[0154] The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula:

[0155] Δp2=[ω2·λ]

[0156] Where Δp2 is the load-side discrete granularity in MW, λ is the adjustable load response rate in MW / min, and ω2 is the load-side empirical coefficient.

[0157] If the adjustable resources on the load side can only be reduced, the discrete granularity of the load side must satisfy the following constraints:

[0158]

[0159] If the adjustable resources on the load side can be adjusted both vertically and horizontally, the discrete granularity of the load side satisfies the following constraints:

[0160]

[0161] Where P is the current value of the adjustable load, P max α2 represents the maximum adjustable load value, and α2 is the empirical coefficient for load-side dispersion.

[0162] Macro-action space construction module: It is used to consider the adjustment characteristics of various adjustable resources, optimize and combine the discretized meta-action spaces according to the set rules and algorithms to form macro-action spaces; specifically, different macro-action spaces are constructed according to the positive and negative values ​​of the power generation and consumption imbalance.

[0163] If the power generation and consumption imbalance is positive, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the upper limit of output, pumped storage power stations that are generating power but have not reached the upper limit of output, energy storage loads that have not reached the upper limit of capacity, and electric vehicle charging piles that are charging.

[0164] If the power generation and consumption imbalance is negative, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging.

[0165] The formula for calculating the power generation-consumption imbalance is as follows:

[0166]

[0167] in, The total load forecast for the next time period. The projected total output of new energy sources for the next time period. ΔPunbalance represents the total output of the conventional units at this moment. t+1 This represents the power generation and consumption imbalance at the next moment.

[0168] Shielding module: Used to shield illegal action space in macro action space to obtain shielded macro action space; specifically, illegal action space shielding is performed according to the response mode of source-side adjustable resources and load-side adjustable resources;

[0169] The response methods of the source-side adjustable resources include:

[0170] 1) Conventional thermal power plants can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0171] 2) Conventional hydropower can adjust the unit output within the upper and lower limits of the unit output according to the response target;

[0172] 3) When the pumped storage power station is in pumping mode, the power output of the power station is constant and cannot be adjusted, and there is no pumping-power generation state transition;

[0173] 4) Centralized wind power and centralized photovoltaic systems can adjust unit output according to response targets;

[0174] The response methods of the load-side adjustable resources include:

[0175] 1) Some energy storage loads only suspend their current charging / discharging state and do not switch between charging and discharging states;

[0176] 2) Distributed wind power and distributed photovoltaic power can reduce the output of some or all units according to the response target;

[0177] 3) Electric vehicle charging stations only pause the current charging / discharging state and do not switch between charging and discharging states.

[0178] Training module: Used to train reinforcement learning agents based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, it returns to the discretization module and performs iterative loops until the reinforcement learning training action space is generated.

[0179] Specifically, the rationality of masking macro action space is verified based on the crash rate of the agent in the maximum number of training rounds. If the crash rate does not exceed the preset value (usually set to 5%), the verification is reasonable; otherwise, the verification is unreasonable.

[0180] Example 4

[0181] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a reinforcement learning action space dimensionality reduction method.

[0182] Example 5

[0183] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a reinforcement learning action space dimensionality reduction method.

[0184] Definitions of abbreviations and key terms

[0185] Real-time power grid dispatch plan: Based on data such as ultra-short-term load forecast, unit generation capacity, and power grid model, and under the premise of meeting safety and power quality requirements, it takes into account constraints such as unit operation constraints, power grid security constraints, and system power balance constraints, and aims to achieve centralized optimization and compilation of power generation plans for the next 5 minutes to 1 hour with the goal of minimizing power generation costs.

[0186] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0187] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

Claims

1. A reinforcement learning action space dimensionality reduction method, characterized in that, Includes the following steps: Taking into account various adjustable resources in the new power system, several meta-action spaces are constructed for reinforcement learning training of real-time power grid dispatching plan arrangement. The meta-action spaces include adjustable resources on the source side and the load side. The adjustable resources on the source side include conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power. The adjustable resources on the load side include distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles. By discretizing the continuous action space for several meta-action spaces, several discretized meta-action spaces are obtained, specifically: The degree of discretization of the action space for adjustable resources on the source side is determined based on the unit's ramp rate, as shown in the following formula: in, Source-side discrete granularity, in MW. This refers to the unit's ramp-up rate, expressed in MW / min. For the source-side empirical coefficients, [] denotes rounding; The source-side discrete granularity satisfies the following constraints: in, This is the upper limit of the unit's output. This represents the lower limit of the unit's output. These are the discrete empirical coefficients on the source side; The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula: in, The discrete particle size on the load side is expressed in MW. The adjustable load response rate is expressed in MW / min. For the load-side empirical coefficient; If the adjustable resources on the load side can only be reduced, the discrete granularity of the load side must satisfy the following constraints: If the adjustable resources on the load side can be adjusted both vertically and horizontally, the discrete granularity of the load side satisfies the following constraints: in, P The current value of the adjustable load. For adjustable maximum load, These are the empirical coefficients for the load-side discrete parameters; Considering the adjustment characteristics of various adjustable resources, the discretized meta-action space is optimized and combined according to the set rules and algorithms to form the macro-action space; By masking illegal action spaces in the macro action space, a masked macro action space is obtained. The reinforcement learning agent is trained based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, the process returns to the continuous action space discretization step and iterates until the reinforcement learning training action space is generated.

2. The reinforcement learning action space dimensionality reduction method according to claim 1, characterized in that, The process involves considering the adjustment characteristics of various adjustable resources, and optimizing and combining the discretized meta-action spaces according to set rules and algorithms to form a macro-action space, specifically: Different macro-action spaces are constructed according to the positive and negative values ​​of the power generation and consumption imbalance. If the power generation and consumption imbalance is positive, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the upper limit of output, pumped storage power stations that are generating power but have not reached the upper limit of output, energy storage loads that have not reached the upper limit of capacity, and electric vehicle charging piles that are charging. If the power generation and consumption imbalance is negative, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging. The formula for calculating the power generation-consumption imbalance is as follows: in, The total load forecast for the next time period. The projected total output of new energy sources for the next time period. This represents the total output of the conventional generating units at this moment. This represents the power generation and consumption imbalance at the next moment.

3. The reinforcement learning action space dimensionality reduction method according to claim 1, characterized in that, The process of masking illegal actions within the macro action space to obtain a masked macro action space is as follows: Illegal actions are spatially shielded based on the response methods of source-side and load-side adjustable resources; The response methods of the source-side adjustable resources include: 1) Conventional thermal power plants can adjust the unit output within the upper and lower limits of the unit output according to the response target; 2) Conventional hydropower can adjust the unit output within the upper and lower limits of the unit output according to the response target; 3) When the pumped storage power station is in pumping mode, the power output of the power station is constant and cannot be adjusted, and there is no pumping-power generation state transition; 4) Centralized wind power and centralized photovoltaic systems can adjust unit output according to response targets; The response methods of the load-side adjustable resources include: 1) Some energy storage loads only suspend their current charging / discharging state and do not switch between charging and discharging states; 2) Distributed wind power and distributed photovoltaic power can reduce the output of some or all units according to the response target; 3) Electric vehicle charging stations only pause the current charging / discharging state and do not switch between charging and discharging states.

4. The reinforcement learning action space dimensionality reduction method according to claim 1, characterized in that, The verification of the rationality of the shielded macro action space specifically includes: The validity of the masked macro action space is verified based on the crash rate of the agent during the maximum number of training rounds. If the crash rate does not exceed the preset value, the verification is valid; otherwise, the verification is invalid.

5. A reinforcement learning action space dimensionality reduction system, characterized in that, include: Meta-action space construction module: This module comprehensively considers various adjustable resources in the new power system and constructs several meta-action spaces for reinforcement learning training of real-time power grid dispatching plans. The meta-action spaces include adjustable resources on the source side and the load side. The adjustable resources on the source side include conventional thermal power, conventional hydropower, pumped storage power stations, centralized wind power, and centralized photovoltaic power. The adjustable resources on the load side include distributed wind power, distributed photovoltaic power, some energy storage loads, and electric vehicle charging piles. Discretization module: Used to discretize the continuous action space for several meta-action spaces, resulting in several discretized meta-action spaces, specifically: The degree of discretization of the action space for adjustable resources on the source side is determined based on the unit's ramp rate, as shown in the following formula: in, Source-side discrete granularity, in MW. This refers to the unit's ramp-up rate, expressed in MW / min. For the source-side empirical coefficients, [] denotes rounding; The source-side discrete granularity satisfies the following constraints: in, This is the upper limit of the unit's output. This represents the lower limit of the unit's output. These are the discrete empirical coefficients on the source side; The degree of discretization of the action space for load-side adjustable resources is determined based on the response characteristics of the adjustable load, as shown in the following formula: in, The discrete particle size on the load side is expressed in MW. The adjustable load response rate is expressed in MW / min. For the load-side empirical coefficient; If the adjustable resources on the load side can only be reduced, the discrete granularity of the load side must satisfy the following constraints: If the adjustable resources on the load side can be adjusted both vertically and horizontally, the discrete granularity of the load side satisfies the following constraints: in, P The current value of the adjustable load. For adjustable maximum load, These are the empirical coefficients for the load-side discrete parameters; Macro Action Space Construction Module: This module takes into account the adjustment characteristics of various adjustable resources and optimizes and combines the discretized meta action spaces according to the set rules and algorithms to form the macro action space. Shielding module: Used to shield illegal actions in the macro action space, resulting in a shielded macro action space; Training module: Used to train reinforcement learning agents based on the masked macro action space. During the training process, the rationality of the masked macro action space is verified. If it is reasonable, the reinforcement learning training action space is generated. Otherwise, it returns to the discretization module and performs iterative loops until the reinforcement learning training action space is generated.

6. The reinforcement learning action space dimensionality reduction system according to claim 5, characterized in that, In the macro-action space construction module, considering the adjustment characteristics of various adjustable resources, the discretized meta-action spaces are optimized and combined according to the set rules and algorithms to form the macro-action space, specifically: Different macro-action spaces are constructed according to the positive and negative values ​​of the power generation and consumption imbalance. If the power generation and consumption imbalance is positive, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the upper limit of output, pumped storage power stations that are generating power but have not reached the upper limit of output, energy storage loads that have not reached the upper limit of capacity, and electric vehicle charging piles that are charging. If the power generation and consumption imbalance is negative, the macro-action space includes conventional thermal power and conventional hydropower that have not reached the lower limit of output, pumped storage power stations that are generating power and have not reached the lower limit of output, centralized wind power, centralized photovoltaic power, distributed wind power, distributed photovoltaic power, energy storage loads that have not reached the lower limit of capacity, and electric vehicle charging piles that are discharging. The formula for calculating the power generation-consumption imbalance is as follows: in, The total load forecast for the next time period. The projected total output of new energy sources for the next time period. This represents the total output of the conventional generating units at this moment. This represents the power generation and consumption imbalance at the next moment.

7. The reinforcement learning action space dimensionality reduction system according to claim 5, characterized in that, In the shielding module, illegal action space is shielded from the macro action space to obtain a shielded macro action space, specifically as follows: Illegal actions are spatially shielded based on the response methods of source-side and load-side adjustable resources; The response methods of the source-side adjustable resources include: 1) Conventional thermal power plants can adjust the unit output within the upper and lower limits of the unit output according to the response target; 2) Conventional hydropower can adjust the unit output within the upper and lower limits of the unit output according to the response target; 3) When the pumped storage power station is in pumping mode, the power output of the power station is constant and cannot be adjusted, and there is no pumping-power generation state transition; 4) Centralized wind power and centralized photovoltaic systems can adjust unit output according to response targets; The response methods of the load-side adjustable resources include: 1) Some energy storage loads only suspend their current charging / discharging state and do not switch between charging and discharging states; 2) Distributed wind power and distributed photovoltaic power can reduce the output of some or all units according to the response target; 3) Electric vehicle charging stations only pause the current charging / discharging state and do not switch between charging and discharging states.

8. The reinforcement learning action space dimensionality reduction system according to claim 5, characterized in that, In the training module, verifying the rationality of the masked macro action space specifically involves: The validity of the masked macro action space is verified based on the crash rate of the agent during the maximum number of training rounds. If the crash rate does not exceed the preset value, the verification is valid; otherwise, the verification is invalid.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning action space dimensionality reduction method as described in any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning action space dimensionality reduction method as described in any one of claims 1 to 4.

Images