A power grid section control method and device based on deep reinforcement learning

Abstract

The present application relates to a method and device for power grid section control based on deep reinforcement learning. The method includes: obtaining the initial power grid state data of the target power grid and the power grid control intelligent model; inputting the initial power grid state data into the power grid control intelligent model, determining the power grid control method and power grid control data for the target power grid; adjusting the power grid parameters of the target power grid according to the power grid control method and the power grid control data, and outputting the control power grid state data; feeding back the control power grid state data to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than the disturbance threshold. The use of this method can realize real-time monitoring and adjustment of the power grid operation status, ensure the stable operation of the power grid, and minimize the need for manual intervention, thereby improving the efficiency and accuracy of power grid control.

Description

Technical Field

[0001] The present application relates to the field of smart grid technology, and in particular to a power grid section control method, apparatus, computer equipment, storage medium and computer program product based on deep reinforcement learning. Background Art

[0002] The power system is a complex nonlinear system. The control strategy for a single section may not be applicable to other sections. Therefore, the multi-section control problem is a difficult problem in the field of power grid automatic control. In traditional production practice, power operators usually modify the sections based on daily dispatching experience. On the one hand, they rely heavily on expert experience, and on the other hand, they are difficult to cope with complex and changeable large-scale power grids, resulting in low control accuracy of the power grid sections. Summary of the invention

[0003] Based on this, it is necessary to provide a power grid section control method, device, computer equipment, computer readable storage medium and computer program product based on deep reinforcement learning, which can improve the control accuracy of the power grid section in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides a power grid section control method based on deep reinforcement learning, comprising:

[0005] Obtaining the initial grid state data of the target grid and the grid control intelligent model;

[0006] Inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid;

[0007] According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output;

[0008] The regulated power grid state data is fed back to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

[0009] In a second aspect, the present application also provides a power grid section control device based on deep reinforcement learning, comprising:

[0010] A power grid data acquisition module is used to obtain the initial power grid state data of the target power grid and the power grid control intelligent model;

[0011] A control mode determination module, used for inputting the initial power grid state data into the power grid control intelligent model, and determining the power grid control mode and power grid control data for the target power grid;

[0012] A control data output module, used to adjust the grid parameters of the target grid according to the grid control mode and the grid control data, and output control grid state data;

[0013] The power grid section control module is used to feed back the regulated power grid state data to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

[0014] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0015] Obtaining the initial grid state data of the target grid and the grid control intelligent model;

[0016] Inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid;

[0017] According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output;

[0018] The regulated power grid state data is fed back to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0020] Obtaining the initial grid state data of the target grid and the grid control intelligent model;

[0021] Inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid;

[0022] According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output;

[0023] The regulated power grid state data is fed back to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

[0024] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0025] Obtaining the initial grid state data of the target grid and the grid control intelligent model;

[0026] Inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid;

[0027] According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output;

[0028] The regulated power grid state data is fed back to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

[0029] The above-mentioned power grid section control method, device, computer equipment, storage medium and computer program product based on deep reinforcement learning obtains the initial power grid state data of the target power grid and the power grid control intelligent model; inputs the initial power grid state data into the power grid control intelligent model to determine the power grid control method and power grid control data for the target power grid; adjusts the power grid parameters of the target power grid according to the power grid control method and the power grid control data, and outputs the controlled power grid state data; and feeds back the controlled power grid state data to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than the disturbance threshold.

[0030] By obtaining the initial state data of the target power grid and the intelligent model of power grid control, the system can fully understand the real-time situation and characteristics of the power grid. This data-driven method makes the power grid control process no longer rely on static rules or manual intervention, but can be dynamically adjusted and optimized according to actual conditions. The initial state data is input into the intelligent model of power grid control. The system can determine the best power grid control method and parameter settings through intelligent analysis and learning of the model. This decision-making ability based on the intelligent model enables the system to make decisions faster and more accurately and adapt to changes in the power grid operation status. According to the control method and data output by the model, the parameters of the target power grid are adjusted to optimize the power grid operation status. The regulated data is fed back to the intelligent model of power grid control. The system can continuously monitor the power grid operation status and further optimize and adjust according to the feedback information to ensure the stable operation of the power grid. This circular feedback mechanism enables the system to continuously learn and improve its own control capabilities, gradually adapt to the complex and changing power grid environment, and improve the efficiency and reliability of power grid control. This process will realize real-time monitoring and adjustment of the power grid operation status, ensure the stable operation of the power grid, and minimize the need for manual intervention, thereby improving the efficiency and accuracy of power grid control. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 This is an application environment diagram of a power grid section control method based on deep reinforcement learning in one embodiment;

[0033] Figure 2 A schematic diagram of a flow chart of a power grid section control method based on deep reinforcement learning in one embodiment;

[0034] Figure 3 A schematic diagram of a flow chart of a power grid control method and a method for determining power grid control data in one embodiment;

[0035] Figure 4 A schematic diagram of a flow chart of a method for checking power grid control data in one embodiment;

[0036] Figure 5 A schematic diagram of a flow chart of a method for controlling the output of power grid status data in one embodiment;

[0037] Figure 6 A schematic diagram of a flow chart of a method for controlling data feedback in one embodiment;

[0038] Figure 7 A schematic diagram of a flow chart of a method for checking power grid section data in one embodiment;

[0039] Figure 8 A system schematic diagram of a power grid section control method based on deep reinforcement learning in one embodiment;

[0040] Fig. 9 It is a structural block diagram of a power grid section control device based on deep reinforcement learning in one embodiment;

[0041] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] The present application provides a power grid section control method based on deep reinforcement learning, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the initial grid state data and the grid control intelligent model of the target grid through the terminal 102; inputs the initial grid state data into the grid control intelligent model to determine the grid control method and grid control data for the target grid; according to the grid control method and the grid control data, adjusts the grid parameters of the target grid, and outputs the control grid state data; feeds back the control grid state data to the grid control intelligent model until the disturbance of the grid section data of the target grid is less than the disturbance threshold. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0044] In an exemplary embodiment, Figure 2 As shown in the figure, a power grid section control method based on deep reinforcement learning is provided. Figure 1 The server in the example is used to illustrate, including the following steps 202 to 208. Among them:

[0045] Step 202: Acquire the initial grid state data of the target grid and the grid control intelligent model.

[0046] The target power grid may be a portion of the power grid that requires power grid section regulation.

[0047] The initial grid state data may be data directly read from the grid, reflecting the current grid operation state, such as voltage, current, power and other information of the node, as well as the load of the line, the state of the transformer, etc. The data may be acquired in real time through sensors, monitoring equipment or an existing grid monitoring system.

[0048] Among them, the intelligent model of power grid regulation can be a neural network dispatching agent obtained through training, which can realize rapid decision-making on real-time dispatching instructions of the power system in a complex power grid environment, taking into account the dual demands of safety and economy of power system operation.

[0049] Specifically, the initial grid status data of the target grid is obtained through the grid monitoring system, including the voltage, current, load and other information of each node, and the grid control intelligent model corresponding to the target grid is also retrieved; wherein, the grid control intelligent model is constructed by using sample data for deep reinforcement learning and other methods. By training the grid control intelligent model, it can make corresponding control actions according to the current grid status to ensure the stable operation of the grid and fault recovery.

[0050] Step 204 , inputting the initial grid state data into the grid control intelligent model, and determining the grid control method and grid control data for the target grid.

[0051] The power grid control method may be a specific method for controlling the parameters of the power grid.

[0052] The power grid control data may be reference data for controlling parameters of the power grid.

[0053] Specifically, the acquired initial grid state data is preprocessed, including data cleaning, removal of outliers, data interpolation and filling, etc., to ensure the accuracy and integrity of the data. Then, the data is normalized, and the value range of each feature is mapped to the same scale to facilitate the training and processing of the grid control intelligent model. The preprocessed grid state data is input into the pre-trained grid control intelligent model. The intelligent model can be a neural network model based on deep learning, or other types of models, such as a reinforcement learning model or a rule engine. The grid control intelligent model will analyze and process the input grid state data to determine the optimal grid control method. Based on the input grid state data, the intelligent model will determine the grid control method for the target grid, which includes determining the node parameters that need to be adjusted, such as generator output power, transformer ratio, etc., and the line switch state that needs to be adjusted, such as closing or disconnecting certain lines, etc. The determination of the grid control method is usually achieved through the policy network or value function inside the model. Once the grid control method is determined, the grid control intelligent model will generate the corresponding grid control data. The grid control data includes the adjusted parameter values ​​and switch status, as well as the corresponding control strategies and suggestions. The grid control data can be directly applied to the actual power grid, and the grid control system will execute the corresponding control actions to achieve stable operation and optimization of the power grid.

[0054] Step 206, adjusting the grid parameters of the target grid according to the grid control method and the grid control data, and outputting the controlled grid state data.

[0055] Among them, the regulated power grid state data is the state data output after regulating the target power grid.

[0056] Specifically, according to the determined grid control method, the grid parameters of the target grid are adjusted accordingly, which involves adjusting the generator output power, transformer ratio, line switch status and other operations. After applying the grid control data, the operation of the grid is simulated using simulation tools or simulation environments. The impact of the control operation on the grid state is evaluated through methods such as power flow calculation to verify the effectiveness and safety of the control. According to the simulation results, the regulated grid state data is generated, including the voltage, current, load and other information of each node after adjustment, as well as the load condition of the line, the state of the transformer, etc. These data can be used for the operation and monitoring of the actual grid.

[0057] Step 208, feeding back the regulated power grid state data to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than the disturbance threshold.

[0058] The disturbance threshold may be data for evaluating the disturbance of the power grid section data.

[0059] Specifically, when the disturbance of the grid section data of the target grid is greater than the disturbance threshold, it indicates that the regulation of the target grid has not reached the preset state, so the regulated grid state data after regulation is fed back to the grid regulation intelligent model, which includes the adjusted voltage, current, load and other information of each node, as well as the load condition of the line, transformer state and the like. The grid regulation intelligent model will use these regulated grid state data, or combine the new initial grid state data at that moment to evaluate the regulation effect and further optimize the regulation strategy. After the grid regulation intelligent model receives the feedback data and re-regulates, the disturbance of the grid section data is evaluated again. The disturbance can be measured by calculating the difference between the grid section data and the target state data, such as the deviation of the node voltage, the change of the line load and the like. The disturbance obtained by the evaluation is compared with the preset disturbance threshold. If the disturbance of the grid section data is less than the disturbance threshold, it means that the grid has reached a stable state and the regulation process ends; otherwise, the regulated grid state data is fed back to the grid regulation intelligent model for the next round of regulation, and the above steps are repeated until the disturbance of the grid section data is less than the disturbance threshold. This means that the grid has reached a stable state and the regulation process ends. Through an iterative approach, it can be ensured that the intelligent model of power grid control can continuously optimize the control strategy so that the power grid can quickly stabilize and maintain a stable state.

[0060] In the above-mentioned power grid section control method based on deep reinforcement learning, the initial power grid state data of the target power grid and the power grid control intelligent model are obtained; the initial power grid state data is input into the power grid control intelligent model to determine the power grid control method and power grid control data for the target power grid; according to the power grid control method and power grid control data, the power grid parameters of the target power grid are adjusted and the control power grid state data is output; the control power grid state data is fed back to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than the disturbance threshold. Figure 8 As shown, this is a schematic diagram of a power grid section control system based on deep reinforcement learning.

[0061] By obtaining the initial state data of the target power grid and the intelligent model of power grid control, the system can fully understand the real-time situation and characteristics of the power grid. This data-driven method makes the power grid control process no longer rely on static rules or manual intervention, but can be dynamically adjusted and optimized according to actual conditions. The initial state data is input into the intelligent model of power grid control. The system can determine the best power grid control method and parameter settings through intelligent analysis and learning of the model. This decision-making ability based on the intelligent model enables the system to make decisions faster and more accurately and adapt to changes in the power grid operation status. According to the control method and data output by the model, the parameters of the target power grid are adjusted to optimize the power grid operation status. The regulated data is fed back to the intelligent model of power grid control. The system can continuously monitor the power grid operation status and further optimize and adjust according to the feedback information to ensure the stable operation of the power grid. This circular feedback mechanism enables the system to continuously learn and improve its own control capabilities, gradually adapt to the complex and changing power grid environment, and improve the efficiency and reliability of power grid control. This process will realize real-time monitoring and adjustment of the power grid operation status, ensure the stable operation of the power grid, and minimize the need for manual intervention, thereby improving the efficiency and accuracy of power grid control.

[0062] In an exemplary embodiment, Figure 3 As shown, the initial grid state data is input into the grid control intelligent model to determine the grid control method and grid control data for the target grid, including steps 302 to 304. Among them:

[0063] Step 302: input the initial grid state data into the grid control intelligent model to determine the continuous control action and discrete control action of the target grid.

[0064] Among them, the continuous control action can be a control instruction for uninterrupted control of the power grid.

[0065] The discrete control action may be a control instruction for performing point-by-point control on the power grid.

[0066] Specifically, the acquired initial power grid state data is preprocessed and normalized, and then input into the power grid control intelligent model. Through the power grid control intelligent model, continuous control actions are determined, such as continuous adjustment of generator output power, transformer ratio adjustment, etc. These actions can usually be adjusted within a continuous range, for example, the generator output power can be increased or decreased within a certain range. At the same time, discrete control actions are also determined through the power grid control intelligent model, such as switching lines, switching circuit breakers, etc. These actions are some discrete operations and can only be selected in a limited number of states. In one case, according to the continuous control action determined by the power grid control intelligent model, the target power grid is directly adjusted accordingly, which involves adjusting the output power of the generator, the ratio of the transformer, etc., to achieve fine control of the power grid state; at the same time, according to the discrete control action determined by the power grid control intelligent model, the target power grid is directly operated accordingly, which includes switching lines, switching circuit breakers, etc., to achieve adjustment of the power grid topology. According to the implemented control action, the regulated power grid state data after control is generated, including the voltage, current, load and other information of each node after adjustment, as well as the load condition of the line, the state of the transformer, etc.

[0067] Step 304: Determine a power grid control method and power grid control data for the target power grid according to the continuous control action and the discrete control action.

[0068] Specifically, according to the determined continuous control actions and discrete control actions, the operation status and control objectives of the power grid are comprehensively considered to determine the power grid control mode for the target power grid, which includes the combined use of continuous actions and discrete actions to achieve comprehensive control and optimization of the power grid state. According to the determined power grid control mode, the corresponding power grid control data are generated, which include information such as node parameters that need to be adjusted and the switch status of the line. The data of the continuous control action can be a series of continuous values, such as the specific value of the generator output power; the data of the discrete control action is a set of discrete states, such as the switch status of the line. The determination of the power grid control mode is usually achieved through the strategy network or value function inside the model. Once the power grid control mode is determined, the power grid control intelligent model will generate corresponding power grid control data according to the continuous control action and the discrete control action.

[0069] In this embodiment, by inputting the initial grid state data into the grid control intelligent model, the system can comprehensively consider the real-time state of the grid and various control methods, so as to determine the continuous control action and discrete control action suitable for the target grid. Continuous control action refers to parameters that can be adjusted smoothly, such as fine-tuning the output power of the generator; while discrete control actions are discrete operations, such as changing the switch state of the line. In this way, the system can flexibly select the appropriate control method according to the actual situation and needs, and generate corresponding grid control data. This makes the grid control more refined and personalized, can better cope with various challenges and changes in the operation of the grid, and improves the stability and reliability of the grid.

[0070] In an exemplary embodiment, Figure 4 As shown, after determining the grid control mode and grid control data for the target grid according to the continuous control action and the discrete control action, the method further includes steps 402 to 406. Among them:

[0071] Step 402, performing power flow calculation according to the power grid control method and power grid control data to obtain power grid control simulation data.

[0072] The power grid control simulation data may be simulation data obtained by simulating using a power grid control method and power grid control data.

[0073] Specifically, according to the predetermined grid control mode and the generated grid control data, including the adjusted node parameters, the switch status of the line, etc., the power system flow calculation tools, such as the Newton-Raphson method or the DC power flow calculation method, are used to calculate the power flow of the grid. In the calculation process, the topological structure of the grid, the voltage and current parameters of each node, and the node parameter adjustment involved in the control data, the line switch status change and other information are considered. According to the power flow calculation results, the grid control simulation data is generated. The grid control simulation data includes the voltage, current, power and other information of each node after adjustment, as well as the load condition of the line, the state of the transformer, etc., reflecting the actual operation state of the grid after the application of the control mode and data. In one of the feasible ways, the grid control simulation data is analyzed, the control effect is evaluated, and the effectiveness and optimization of the grid control are judged by comparing the conformity of the simulation data with the target grid operation requirements, and the changes in the grid parameters before and after the control. If the control effect meets the expected requirements, it means that the grid control mode and data are reasonable and effective; otherwise, it may be necessary to further adjust the control strategy or data for optimization.

[0074] Step 404: Perform a security check based on the power grid control method and the power grid control data to obtain power grid control security data.

[0075] The power grid control safety data may be inspection data indicating the security of the power grid control method and the power grid control data.

[0076] Specifically, a power system safety assessment tool, such as a power flow calculation software, a short circuit analysis tool, etc., is used to perform a safety check on the power grid that uses the power grid control method and the generated power grid control data. During the inspection process, the topological structure of the power grid, the voltage and current of each node, and the node parameter adjustment involved in the control data, the line switch state change and other information are considered. According to the results of the safety check, the power grid control safety data is generated. The power grid control safety data reflects the safety of the power grid after the control method and data are applied, including whether the voltage of each node is within the safety range, whether the line load exceeds the rated value, whether the equipment is overloaded, etc. In an optional embodiment, the power grid control safety data is analyzed to evaluate the safety of the control. The safety and risk level of the control are judged by comparing the compliance of the power grid control safety data with the power grid safety standards, and the changes in the power grid safety situation before and after the control. If the safety of the power grid after the control is guaranteed and meets the expected requirements, it means that the power grid control method and data are reasonable and effective; otherwise, it may be necessary to further adjust the control strategy or data to ensure the safe operation of the power grid.

[0077] Step 406, when the grid control simulation data or the grid control safety data fails to meet the grid control standard of the target grid, return to the step of inputting the initial grid state data into the grid control intelligent model, determining the grid control method and grid control data for the target grid, until the grid control simulation data or the grid control safety data meets the grid control standard.

[0078] Specifically, when the grid control simulation data or the grid control safety data fails to meet the grid control standard of the target grid, return to execute the step of "inputting the initial grid state data into the grid control intelligent model, and determining the grid control method and grid control data for the target grid" until the grid control simulation data or grid control safety data generated by the evaluation meets the grid control standard of the target grid, then the control process is terminated; if the grid control simulation data or grid control safety data generated by the evaluation still does not meet the standard, continue to return to execute the step of "inputting the initial grid state data into the grid control intelligent model, and determining the grid control method and grid control data for the target grid", and re-determine the grid control method and data until the generated grid control simulation data or grid control safety data meet the grid control standard.

[0079] In this embodiment, by performing power flow calculation and safety check, the system can accurately evaluate the impact of power grid control methods and data, obtain power grid control simulation data and safety data, and on this basis, the system will check whether these data meet the preset power grid control standards. If the standards are met, it means that the current power grid control plan is effective and feasible, but if the standards are not met, the system will automatically return to the initial state, re-run the power grid control intelligent model, and determine new power grid control methods and data until the power grid control standards are met. This cyclic process ensures the continuous optimization and self-correction of power grid control, improves the stability and safety of power grid operation, and reduces the need for human intervention, making power grid control more intelligent, efficient and reliable.

[0080] In an exemplary embodiment, Figure 5 As shown, according to the power grid control mode and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output, including steps 502 to 506. Among them:

[0081] Step 502: According to the grid control method and grid control data, the grid parameters of the target grid are adjusted, and the intermediate grid state data is output.

[0082] The intermediate power grid state data may be an intermediate output parameter for regulating the target power grid.

[0083] Specifically, according to the grid control mode and the generated grid control data, the grid parameters of the target grid are adjusted accordingly, which involves adjusting the generator output power, the transformer ratio, the switch state of the line and other operations. The adjusted grid parameters are applied to the target grid and the corresponding adjustment actions are implemented. This includes the actual operation of the grid equipment, such as adjusting the generator output power, switching the line state, etc. The power flow calculation is performed on the adjusted grid to evaluate the operation status of the grid. The power flow calculation can be performed using power system simulation software or tools, taking into account the topological structure, equipment parameters, load conditions, etc. of the grid. According to the power flow calculation results, the intermediate grid state data is generated. The intermediate grid state data reflects the state of the adjusted grid parameters in actual operation, including information such as voltage, current, power, etc. of each node, as well as the load condition of the line, the state of the equipment, etc.

[0084] Step 504, generating control strategy update information and value function update information of the power grid control intelligent model according to the intermediate power grid state data, and outputting the control power grid state data.

[0085] The control strategy update information may be information that guides the direction in which the power grid control intelligent model is adjusted.

[0086] The value function update information may be information obtained by calculating the value function, and is used to guide the specific parameters of the power grid control intelligent model to be adjusted.

[0087] Specifically, the generated intermediate power grid state data is used to generate control strategy update information for the power grid control intelligent model. The control strategy update information includes information such as the selection and execution order of the control action adjusted according to the current power grid state. The update process can be based on the reinforcement learning algorithm, and the policy information in the model is updated by evaluating the value of the current state and the pros and cons of the execution action; wherein, the generation method of the policy update information can be a gradient-based method, such as using Q-learning, Policy Gradient and other algorithms in the reinforcement learning algorithm. At the same time, the value function information in the power grid control intelligent model is generated according to the generated intermediate power grid state data. The value function information reflects the expected benefits or rewards under different states through the value function. By learning and updating the value function, the power grid control intelligent model can better guide the formulation and execution of control decisions. The update process can be carried out through the value function update method in the reinforcement learning algorithm. The power grid parameters of the target power grid are adjusted according to the intermediate power grid state data to generate corresponding control power grid state data.

[0088] Step 506: Feedback the control strategy update information and the value function update information to the power grid control intelligent model.

[0089] Specifically, the acquired control strategy update information and value function update information are fed back to the power grid control intelligent model, which can be achieved by updating the parameters, weights or other related variables of the model. In the power grid control intelligent model, the feedback control strategy update information and value function update information are used to update the adjustment strategy and value function of the power grid control intelligent model, which involves operations such as adjusting the neural network structure of the model and updating the model parameters to improve the accuracy and effect of the model in subsequent decision-making. After updating the power grid control intelligent model, the model needs to be retrained to further improve its performance and adaptability. This can be achieved by using the feedback control strategy update information and value function update information to construct a training sample set, and then using the sample set to retrain the power grid control intelligent model.

[0090] In this embodiment, by adjusting the grid parameters of the target grid according to the grid control mode and data, the system can respond to changes in the grid state in real time, optimize the parameter settings of the grid operation, and thus generate intermediate grid state data. Using these intermediate state data, the system can update the grid control intelligent model, including the update of the control strategy and the value function. These updated information can reflect the actual situation and needs of the current grid operation, so that the grid control intelligent model can more accurately guide the grid control operation, and these control strategy update information and value function update information are fed back to the grid control intelligent model, and the performance and adaptability of the model are continuously optimized. This process realizes the dynamic adjustment and optimization of the grid control strategy, improves the intelligence level and adaptability of the grid control, and thus ensures the stable operation and safety of the grid.

[0091] In an exemplary embodiment, Figure 6 As shown, the control strategy update information and the value function update information are fed back to the power grid control intelligent model, including steps 602 to 606. Among them:

[0092] Step 602 , according to the control strategy update information and the value function update information, random control data is selected from the power grid control intelligent model to update the power grid control intelligent model to obtain an updated power grid control intelligent model.

[0093] The random control data may be a part of the power grid control data stored in the power grid control intelligent model that is arbitrarily selected.

[0094] Specifically, in the original data set of the power grid control intelligent model, a part of the data is randomly selected as random control data, and the random control data includes power grid state data and corresponding control action data, which are used to update the adjustment strategy and value function of the power grid control intelligent model. According to the obtained control strategy update information and value function update information, the parameters of the power grid control intelligent model are updated in combination with the selected random control data, which includes adjusting the model's weights, biases, activation functions and other parameters to make the power grid control intelligent model better adapt to the latest control strategies and value functions. By adopting incremental learning technology, the data set of the updated model is merged with the training data set of the original model, and then the model is retrained to obtain an updated power grid control intelligent model.

[0095] Step 604, optimizing the execution mode and execution efficiency of the updated power grid control intelligent model to obtain an optimized power grid control intelligent model.

[0096] Specifically, model compression and quantization technology are used to reduce the number of parameters and the amount of calculation of the updated power grid control intelligent model, and improve the execution efficiency of the model in the real-time control process. Optimization measures are further used to improve and optimize the updated power grid control intelligent model, and the optimization strategy of execution mode and execution efficiency is comprehensively considered. The structure, parameters and calculation process of the model are adjusted to obtain the optimized power grid control intelligent model. In the optimization process, new actions need to be explored to find better strategies, and known strategies should also be used to maximize efficiency. This can be achieved by using methods such as the ε-greedy strategy, where ε represents the probability of exploration, which can be gradually reduced as the optimization proceeds. The optimization measures include: 1. Using hardware accelerators (such as GPU, TPU, etc.) to accelerate the model reasoning process and improve the execution speed of the model; 2. Through parallel computing technology, the model's calculation process is divided into multiple parallel tasks, which are executed simultaneously on multiple processing units to improve execution efficiency; 3. Using cache technology to reduce the reading time of model parameters and data and improve data access efficiency; 4. According to the characteristics of power grid regulation, optimize the algorithm design of the model, reduce unnecessary calculation and memory overhead, and improve the execution efficiency of the model; 5. Using batch processing technology, multiple regulation tasks are merged into one batch for simultaneous processing, reducing the overhead of model calls and calculations and improving execution efficiency.

[0097] Step 606: Use the optimized power grid control intelligent model as the power grid control intelligent model.

[0098] Specifically, the optimized grid control intelligent model is used as the grid control intelligent model, which is achieved by loading the optimized model file or parameters into the grid control system. The replaced grid control intelligent model is verified to ensure that it matches the original model in terms of function and performance, and can achieve the expected effect in practical applications. The replaced model is tested and tuned in the actual control scenario. The performance and effect of the optimized model are evaluated through simulated or actual grid control tasks, and necessary adjustments and optimizations are made according to the test results. Once it is confirmed that the optimized model can meet the grid control needs and achieve the expected effect, it can be deployed in the actual grid control system and applied to the actual grid control tasks.

[0099] In this embodiment, random control data is selected from the power grid control intelligent model for updating according to the control strategy update information and the value function update information, so that the model can timely reflect the latest control strategy and value function, and the execution mode and execution efficiency of the updated power grid control intelligent model are optimized, including model compression, hardware acceleration, parallel computing and other technologies, so as to improve the running speed and efficiency of the model, and the optimized power grid control intelligent model is applied to the actual power grid control task to replace the original model. Through this process, the power grid control intelligent model can have stronger real-time, accuracy and adaptability, provide more reliable and efficient support for power grid control decision-making, and thus improve the stability and reliability of the power grid.

[0100] In an exemplary embodiment, Figure 7 As shown, the regulated grid state data is fed back to the grid regulation intelligent model until the disturbance of the grid section data of the target grid is less than the disturbance threshold, including steps 702 to 704. Wherein:

[0101] Step 702: When the disturbance of the power grid section data is greater than the disturbance threshold and the adjustment step number of the target power grid is less than the step number threshold, the regulated power grid state data is used as the initial power grid state data.

[0102] Specifically, after each power grid regulation, check whether the disturbance of the power grid section data is greater than the preset disturbance threshold, which is determined by comparing the difference between the current power grid state data and the data at the previous moment; at the same time, check whether the adjustment step number of the target power grid is less than the preset step number threshold, and the adjustment step number refers to the number of times or rounds of power grid regulation. When the disturbance of the power grid section data is greater than the disturbance threshold and the adjustment step number of the target power grid is less than the step number threshold, the current regulated power grid state data is used as the new initial power grid state data to start the next round of power grid regulation process, otherwise, end this regulation process.

[0103] Step 704, returning to the step of inputting the initial grid state data into the grid control intelligent model, and determining the grid control method and grid control data for the target grid, until the disturbance of the grid section data is less than the disturbance threshold.

[0104] Specifically, the regulated grid state data is used as input, and the process of "inputting the initial grid state data into the grid regulation intelligent model, determining the grid regulation method and grid regulation data for the target grid" is re-executed. After each grid regulation, it is checked whether the disturbance of the grid section data is less than the preset disturbance threshold. This can be determined by comparing the difference between the current grid state data and the data at the previous moment. If the disturbance of the grid section data is still greater than the disturbance threshold, the grid regulation operation is continued until the disturbance of the grid section data is less than the disturbance threshold.

[0105] In this embodiment, when the disturbance of the grid section data exceeds the set threshold and the adjustment step number of the target grid has not reached the threshold, the system will regard the current regulated grid state data as the new initial grid state data, and then the system will return to the step of inputting the initial state data into the grid regulation intelligent model, and re-determine the grid regulation method and data for the target grid. This cycle will continue until the disturbance of the grid section data is reduced to below the preset disturbance threshold. This mechanism can respond to abnormal situations in the operation of the grid in a timely manner, ensure the stability and reliability of the grid, and also improve the autonomy and real-time performance of the grid regulation system, so that the grid regulation can respond to various emergencies more flexibly and intelligently, and ensure the safe operation of the grid.

[0106] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides a power grid section control device based on deep reinforcement learning for implementing the power grid section control method based on deep reinforcement learning involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the power grid section control device based on deep reinforcement learning provided below can refer to the limitations of a power grid section control method based on deep reinforcement learning above, and will not be repeated here.

[0108] In an exemplary embodiment, Fig. 9 As shown, a power grid section control device based on deep reinforcement learning is provided, including: a power grid data acquisition module 902, a control mode determination module 904, a control data output module 906 and a power grid section control module 908, wherein:

[0109] A power grid data acquisition module 902 is used to acquire the initial power grid state data of the target power grid and the power grid control intelligent model;

[0110] A control mode determination module 904 is used to input the initial power grid state data into the power grid control intelligent model to determine the power grid control mode and power grid control data for the target power grid;

[0111] The control data output module 906 is used to adjust the grid parameters of the target grid according to the grid control mode and the grid control data, and output the control grid state data;

[0112] The power grid section control module 908 is used to feed back the regulated power grid state data to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than the disturbance threshold.

[0113] In one embodiment, the control mode determination module 904 is also used to input the initial grid state data into the grid control intelligent model to determine the continuous control action and discrete control action of the target grid; based on the continuous control action and the discrete control action, determine the grid control mode and grid control data for the target grid.

[0114] In one embodiment, the control mode determination module 904 is also used to perform power flow calculation according to the power grid control mode and the power grid control data to obtain power grid control simulation data; perform safety checks according to the power grid control mode and the power grid control data to obtain power grid control safety data; when the power grid control simulation data or the power grid control safety data fails to meet the power grid control standard of the target power grid, return to execute the step of inputting the initial power grid state data into the power grid control intelligent model to determine the power grid control mode and power grid control data for the target power grid until the power grid control simulation data or the power grid control safety data meets the power grid control standard.

[0115] In one embodiment, the control data output module 906 is also used to adjust the grid parameters of the target grid according to the grid control method and the grid control data, and output the intermediate grid status data; generate the control strategy update information and the value function update information of the grid control intelligent model according to the intermediate grid status data, and output the control grid status data; and feed back the control strategy update information and the value function update information to the grid control intelligent model.

[0116] In one embodiment, the control data output module 906 is also used to select random control data from the power grid control intelligent model to update the power grid control intelligent model according to the control strategy update information and the value function update information, so as to obtain an updated power grid control intelligent model; optimize the execution mode and execution efficiency of the updated power grid control intelligent model to obtain an optimized power grid control intelligent model; and use the optimized power grid control intelligent model as the power grid control intelligent model.

[0117] In one embodiment, the grid section control module 908 is also used to use the controlled grid state data as the initial grid state data when the disturbance of the grid section data is greater than the disturbance threshold and the adjustment step number of the target grid is less than the step threshold; return to execute the step of inputting the initial grid state data into the grid control intelligent model, determining the grid control method and grid control data for the target grid, until the disturbance of the grid section data is less than the disturbance threshold.

[0118] Each module in the above-mentioned power grid section control device based on deep reinforcement learning can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.10 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store server data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a power grid section control method based on deep reinforcement learning is implemented.

[0120] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0122] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0123] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A power grid section control method based on deep reinforcement learning, characterized in that: The method comprises: Obtaining the initial grid state data of the target grid and the grid control intelligent model; Inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid; According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and the control power grid state data is output; The step of adjusting the grid parameters of the target grid according to the grid control mode and the grid control data and outputting the control grid state data includes: According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and intermediate power grid state data is output; Generate control strategy update information and value function update information of the power grid control intelligent model according to the intermediate power grid state data, and output the control power grid state data; Feeding back the control strategy update information and the value function update information to the power grid control intelligent model; The step of feeding back the control strategy update information and the value function update information to the power grid control intelligent model includes: According to the control strategy update information and the value function update information, random control data is selected from the power grid control intelligent model to update the power grid control intelligent model to obtain an updated power grid control intelligent model; Optimizing the execution mode and execution efficiency of the updated power grid control intelligent model to obtain an optimized power grid control intelligent model; Using the optimized power grid regulation intelligent model as the power grid regulation intelligent model; The regulated power grid state data is fed back to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

2. The method according to claim 1, characterized in that: The step of inputting the initial grid state data into the grid control intelligent model to determine a grid control method and grid control data for the target grid includes: Inputting the initial grid state data into the grid control intelligent model to determine the continuous control action and discrete control action of the target grid; According to the continuous control action and the discrete control action, a power grid control mode and power grid control data for the target power grid are determined.

3. The method according to claim 2, characterized in that After determining the power grid control mode and power grid control data for the target power grid according to the continuous control action and the discrete control action, the method further includes: Perform power flow calculation according to the power grid control method and the power grid control data to obtain power grid control simulation data; Performing a security check according to the power grid control method and the power grid control data to obtain power grid control security data; In the event that the grid control simulation data or the grid control safety data fails to meet the grid control standard of the target grid, the step of inputting the initial grid state data into the grid control intelligent model and determining the grid control method and grid control data for the target grid is returned to the step of until the grid control simulation data or the grid control safety data meets the grid control standard.

4. The method according to claim 1, characterized in that Feeding back the regulated power grid state data to the power grid regulation intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold includes: When the disturbance of the power grid section data is greater than the disturbance threshold and the adjustment step number of the target power grid is less than the step number threshold, using the regulated power grid state data as the initial power grid state data; Return to the step of inputting the initial grid state data into the grid control intelligent model, and determining the grid control method and grid control data for the target grid, until the disturbance of the grid section data is less than the disturbance threshold.

5. A power grid section control device based on deep reinforcement learning, characterized in that: The device comprises: A power grid data acquisition module is used to obtain the initial power grid state data of the target power grid and the power grid control intelligent model; A control mode determination module, used for inputting the initial power grid state data into the power grid control intelligent model, and determining the power grid control mode and power grid control data for the target power grid; A control data output module, used to adjust the grid parameters of the target grid according to the grid control mode and the grid control data, and output control grid state data; The step of adjusting the grid parameters of the target grid according to the grid control mode and the grid control data and outputting the control grid state data includes: According to the power grid control method and the power grid control data, the power grid parameters of the target power grid are adjusted, and intermediate power grid state data is output; Generate control strategy update information and value function update information of the power grid control intelligent model according to the intermediate power grid state data, and output the control power grid state data; Feeding back the control strategy update information and the value function update information to the power grid control intelligent model; The step of feeding back the control strategy update information and the value function update information to the power grid control intelligent model includes: According to the control strategy update information and the value function update information, random control data is selected from the power grid control intelligent model to update the power grid control intelligent model to obtain an updated power grid control intelligent model; Optimizing the execution mode and execution efficiency of the updated power grid control intelligent model to obtain an optimized power grid control intelligent model; Using the optimized power grid regulation intelligent model as the power grid regulation intelligent model; The power grid section control module is used to feed back the regulated power grid state data to the power grid control intelligent model until the disturbance of the power grid section data of the target power grid is less than a disturbance threshold.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images