SCADA load regulation and control method based on deep reinforcement learning

Abstract

The invention discloses an SCADA load regulation and control method based on deep reinforcement learning. The method comprises the following steps: S1, collecting system state data and constructing a state vector; s2, outputting an initial regulation action by using a strategy function; s3, optimizing a strategy function structure and parameters through a whale optimization algorithm; s4, generating a regulation and control instruction based on the optimization strategy; s5, performing fine adjustment on local key node actions by using a cockroach swarm algorithm; s6, executing security constraint judgment, and screening out non-compliance actions; s7, issuing the effective instruction to the SCADA system; and S8, collecting feedback and updating the strategy function. According to the method, the strategy adaptability, the regulation and control precision and the system safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of SCADA load control, and in particular to a SCADA load control method based on deep reinforcement learning. Background Art

[0002] Existing SCADA (Supervisory Control and Data Acquisition) systems are widely used in key infrastructure such as electricity, industry, petrochemicals, and water conservancy to achieve real-time monitoring, scheduling, and control of remote equipment and systems. In power systems or energy networks, one of the core tasks of the SCADA system is to regulate the load, that is, to perform load distribution, peak shaving and valley filling, or emergency adjustments in real time based on information such as the load status, voltage level, current changes, and network topology of each node to ensure safe, stable, and efficient operation of the system. Traditional SCADA load regulation methods are mainly based on manually set regulation rules, linear optimization algorithms, or static expert systems. These methods can achieve the expected results under known and stable operating conditions, but they have obvious shortcomings in the context of highly complex modern power systems, multi-source access, and highly dynamic operating conditions.

[0003] First, traditional rule- or model-driven methods are difficult to adapt to the ever-changing load fluctuations and topology reconstruction in the power system. Once the system state changes nonlinearly, the original control rules may no longer apply, requiring manual intervention or resetting of the rules, resulting in low response efficiency and high operation and maintenance costs. Secondly, many traditional methods lack the ability to learn from historical system operation data, are unable to optimize their own strategies based on feedback, and are unable to respond quickly and effectively in the face of emergencies or abnormal conditions. In addition, there are limitations to the granularity of control. Many methods can only perform centralized control at a coarse level, and cannot take into account the optimization needs of local nodes, making it difficult to achieve optimal overall energy efficiency at the system level.

[0004] In recent years, the development of artificial intelligence (AI), particularly deep reinforcement learning (DL), has provided new insights into autonomous policy optimization in complex systems. By combining deep neural networks with a reinforcement learning framework, DL enables control agents to learn the state-action-reward mapping relationship through continuous interaction with the environment, enabling the self-evolution of optimal control strategies. However, the direct application of DL methods to SCADA systems still faces a series of technical challenges, such as the sensitivity of policy model hyperparameter settings, insufficient local optimization capabilities, and the difficulty of embedding safety constraints into policy outputs. In practical industrial environments, the structural design and training parameters of the policy function have a decisive impact on system performance, but the lack of a systematic search mechanism often leads to unstable policy performance or trapping in local optima. Furthermore, policy functions are typically guided by global objectives, lacking precision in local load control and making them difficult to adapt to the regulation requirements of multiple sub-regions in complex networks.

[0005] In response to the above problems, the present invention proposes a SCADA load control method based on the fusion of deep reinforcement learning and swarm intelligence algorithm, which breaks through multiple limitations in the existing technology. This method not only realizes learning control behavior from operating data through deep reinforcement learning policy function, but also innovatively introduces the whale optimization algorithm to perform a global search on the policy function structure and parameters, thereby improving the stability and performance expression ability of the model; at the same time, the cockroach swarm algorithm is used to optimize local control actions, significantly improving the response accuracy of node-level load adjustment. In addition, this method also realizes real-time screening of control actions by constructing a safety constraint judgment module, ensuring that all output instructions are executed within the system's load capacity, taking into account both intelligence and safety. This integrated, multi-level intelligent control method effectively makes up for the shortcomings of traditional SCADA load control technology in terms of strategy adaptability, self-optimization capability and local precision control, and has good engineering feasibility and promotion value.

[0006] Therefore, how to provide a SCADA load control method based on deep reinforcement learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0007] One objective of the present invention is to propose a SCADA load control method based on deep reinforcement learning and a swarm intelligence optimization algorithm. This method leverages the autonomous learning capabilities of deep reinforcement learning policy functions and the global search and local fine-tuning properties of swarm intelligence algorithms. The method describes in detail the entire process of tuning policy hyperparameters using a whale optimization algorithm, fine-tuning local load actions using a cockroach swarm algorithm, and dynamically filtering control instructions in conjunction with a safety constraint mechanism. This method boasts strong adaptability, high control accuracy, robust policy optimization capabilities, and high operational safety, making it suitable for real-time load scheduling and energy efficiency management in a variety of industrial energy systems.

[0008] A SCADA load control method based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0009] A SCADA load control method based on deep reinforcement learning, characterized by comprising the following steps:

[0010] S1, collect the operating data in the SCADA system and construct the current system state vector;

[0011] S2. Based on the system state vector, a policy network model is constructed using a deep reinforcement learning algorithm;

[0012] S3. Globally optimize the structural hyperparameters of the policy network model using the whale optimization algorithm, with the optimization goal being to maximize the cumulative reward value on the training set;

[0013] S4. Based on the optimized policy network model, a control action vector is generated according to the current system state vector;

[0014] S5. Apply the cockroach swarm algorithm to perform local search optimization in the neighborhood action space for the control action vector to generate an optimized control action vector;

[0015] S6. Input the optimized control action vector into the safety constraint judgment module, filter the actions according to the upper and lower voltage limits, current limit values and topology switching constraints, and eliminate the control actions that do not meet the operation safety conditions;

[0016] S7, sending the filtered control actions to the SCADA system execution layer to control the load equipment to perform corresponding adjustment operations;

[0017] S8. Collect the system feedback status after the control is executed, and use the feedback status and the previous status as input to perform reinforcement learning to update the policy network model.

[0018] Optionally, the S1 specifically includes:

[0019] S11, collecting real-time operating data of multiple controlled nodes from the SCADA system, wherein the operating data includes the voltage value, current value, load power value of each node and the connection relationship information between the nodes;

[0020] S12, performing data preprocessing on the collected voltage value, current value, and load power value, wherein the data preprocessing includes time synchronization processing, invalid value elimination, and dimension unification processing;

[0021] S13. Construct a corresponding node state subvector based on the processed electrical parameters of each node and its connection relationship in the power grid, wherein the node state subvector includes the node number, current voltage value, current current value, current load power value, and a set of associated adjacent node numbers;

[0022] S14. Combine the state sub-vectors of all nodes according to the power grid topology to generate a system state vector, which comprehensively represents the operating state of the SCADA system at the current time step. The system state vector serves as the input basis for the subsequent strategy network model.

[0023] Optionally, the S2 specifically includes:

[0024] S21, the system state vector S to be constructed t Input to the policy network model, which is a function approximation model based on deep reinforcement learning, and is represented by a parameterized structure π θ (S t ), where π θ represents the policy function defined by the parameter set θ;

[0025] S22, constructing an action space A, wherein the action space includes the control target node number, the load adjustment range of the corresponding node, the standby node activation flag and the network structure adjustment instruction, and the action vector is represented by the strategy function π θ (S t ) output, represented as a t =π θ (S t ), where a t ∈A represents the specific control action at time step t;

[0026] S23. Define the reward function R t , evaluate action a t In state S t The impact on the operation effect of the SCADA system under the current environment, the reward function includes energy consumption cost term, security constraint penalty term and load balancing incentive term;

[0027] S24, using a deep reinforcement learning algorithm to train and optimize the policy network model, the algorithm includes a state transfer process, action selection, reward evaluation and network parameter update process, using the historical state transfer sequence (S t ,a t ,R t ,S t+1 ) Perform gradient updates on the policy network and update the policy parameter set θ to maximize the cumulative expected reward;

[0028] S25, after completing the training, the policy network model retains the current optimal parameter set θ *.

[0029] Optionally, the S3 specifically includes:

[0030] S31. Setting the policy function π θ (S t ) hyperparameter search range, the hyperparameters include the number of hidden layers L, the number of neurons in each layer n l , activation function type f l , learning rate α, and entropy regularization term weight coefficient λ;

[0031] S32, initialize the individual population of the whale optimization algorithm {X1,X2,...,X K}, where each individual X k Represents a set of policy functions π θ (S t ) of the hyperparameter combination, the population size is K, and each set of hyperparameters X k Applied to the training strategy function and calculate the corresponding cumulative expected reward value R on the verification data k ;

[0032] S33. In each iteration, the individual X is updated according to the update mechanism of the whale optimization algorithm. k Perform location update, the update formula is:

[0033]

[0034] in, Indicates the maximum cumulative expected reward value R in the current t generation k The optimal individual is A=2a·r1-a, C=2r2, where a is the convergence factor, r1 and r2 are random numbers in the range of [0,1];

[0035] S34, each updated hyperparameter individual Applied to the policy function π θ (S t ) training process and recalculate its corresponding cumulative expected reward value At the same time, update the new optimal individual in this generation

[0036] S35. Repeat the individual update and reward evaluation process until the termination condition of the whale optimization algorithm is met, wherein the termination condition includes the convergence of the cumulative expected reward value or the maximum number of iterations.

[0037] S36, the final optimal hyperparameter combination X * Applied to the subsequent policy function π θ (S t)’s structural configuration and training settings as the basic strategy function for generating control action vectors.

[0038] Optionally, the S4 specifically includes:

[0039] S41, the optimized strategy function π θ (S t ) After configuration is completed, it is used for online reasoning;

[0040] S42, the system state vector S of the current time step t Input to the policy function π θ (S t ), the corresponding control action vector a is output by the strategy function t ;

[0041] S43, the control action vector a t It includes multiple sub-commands, each of which contains the target node number, load adjustment range, standby node enablement status and network structure switching flag.

[0042] Optionally, the S5 specifically includes:

[0043] S51, from the control action vector a t Extract the target node set containing the load adjustment instruction, construct the local control subspace, and generate the local state subvector set corresponding to the target node;

[0044] S52. Initializing a population of individuals of the cockroach swarm algorithm based on the local state subvector, where each individual represents a set of local control action plans, the plans including the node number, load adjustment direction, adjustment amplitude, and load correlation factor with adjacent nodes;

[0045] S53. In each iteration, the fitness values of the individuals in terms of load balance index, voltage stability index and local energy consumption function are evaluated, and the current local optimal individual is selected as the target direction;

[0046] S54: Update individual positions according to the behavior mechanism of the cockroach swarm algorithm, wherein the behavior mechanism includes:

[0047] Neighborhood optimal movement: each individual moves closer to the current local optimal individual;

[0048] Memory retention: retain historical optimal solutions for comparison and update;

[0049] Obstacle avoidance perturbation: introducing small-scale random perturbations to escape local optimality;

[0050] Synchronous propagation: broadcast the updated information of excellent individuals in the population;

[0051] S55. When the local termination conditions are met, including the fitness function convergence or the number of iterations reaching the set upper limit, the current optimal local control action plan is output;

[0052] S56, replacing the control action vector a with the local optimal control action plan t The corresponding sub-vector part in the local optimization is obtained to obtain the control action vector a′ t .

[0053] Optionally, the S6 specifically includes:

[0054] S61, the locally optimized control action vector a′ t Input to the safety constraint judgment module;

[0055] S62. Screening each control instruction based on preset operation safety rules, wherein the safety rules include node voltage limit, current upper limit, line load rate threshold, and topology switchability judgment;

[0056] S63. Eliminate the control instructions that do not meet the safety constraints, retain the actions that meet all the constraints, and form the filtered control action vector for issuance and execution.

[0057] Optionally, the S7 specifically includes:

[0058] S71, sending the control action vector that has passed the security screening as a control instruction to the SCADA system execution layer;

[0059] S72. Control the corresponding load device to perform start / stop, switch, or power adjustment operations according to the node number, load adjustment range, and network structure adjustment information in the control instruction;

[0060] S73. After the operation is completed, the execution result is recorded and the system status is updated synchronously.

[0061] Optionally, the S8 specifically includes:

[0062] S81, collect the system state after the control action is executed, and construct a new system state vector S t+1 ;

[0063] S82, the current state vector S t , control action vector a′ t , reward value R t and the subsequent state vector S t+1 Composition state transition samples;

[0064] S83. Use the state transition sample to update the strategy function π θ (S t) to improve the adaptability and optimality of the control strategy in the future state.

[0065] The beneficial effects of the present invention are:

[0066] (1) This paper constructs a deep reinforcement learning policy function based on state vector input to achieve online learning and self-optimization of the SCADA system control strategy, avoiding the limitations of traditional methods based on artificial rules and static models. Furthermore, by introducing the whale optimization algorithm to perform global search and optimization on the network structure and hyperparameters of the policy function, the control strategy has better generalization ability and stability in different system scenarios, thereby improving the overall performance of the control strategy.

[0067] (2) This invention innovatively introduces a cockroach swarm algorithm to locally optimize the control action vector output by the policy function, and uses the neighborhood search mechanism of swarm intelligence to solve the problem of fine-grained control in local load clusters. This design effectively improves the response speed and control accuracy under grid load reconfiguration, local node overload, or critical operating conditions, compensating for the shortcomings of deep reinforcement learning strategies in local control accuracy.

[0068] (3) After the control action is generated, the present invention introduces a safety constraint judgment module. By performing real-time verification of various operational boundaries such as voltage, current, line capacity, and topology feasibility, it screens out control instructions that do not meet operational safety conditions, ensuring that no new operational risks are introduced during the SCADA system control process. At the same time, the system supports real-time feedback of control effects and iterative strategy updates, enhancing the stability and robustness of the entire control closed loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0070] Figure 1 This is an overall flow chart of a SCADA load control method based on deep reinforcement learning and swarm intelligence algorithm proposed in the present invention;

[0071] Figure 2 This is a schematic diagram of the input and output structure of the control strategy function of the SCADA load control method based on deep reinforcement learning proposed in the present invention;

[0072] Figure 3 This is an iterative flow chart of the optimization strategy parameters of the whale optimization algorithm for the SCADA load control method based on deep reinforcement learning proposed in this invention. DETAILED DESCRIPTION

[0073] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0074] refer to Figure 1-3 , a SCADA load control method based on deep reinforcement learning, comprising the following steps:

[0075] S1, collect the operating data in the SCADA system and construct the current system state vector;

[0076] S2. Based on the system state vector, a policy network model is constructed using a deep reinforcement learning algorithm;

[0077] S3. Globally optimize the structural hyperparameters of the policy network model using the whale optimization algorithm, with the optimization goal being to maximize the cumulative reward value on the training set;

[0078] S4. Based on the optimized policy network model, a control action vector is generated according to the current system state vector;

[0079] S5. Apply the cockroach swarm algorithm to perform local search optimization in the neighborhood action space for the control action vector to generate an optimized control action vector;

[0080] S6. Input the optimized control action vector into the safety constraint judgment module, filter the actions according to the upper and lower voltage limits, current limit values and topology switching constraints, and eliminate the control actions that do not meet the operation safety conditions;

[0081] S7, sending the filtered control actions to the SCADA system execution layer to control the load equipment to perform corresponding adjustment operations;

[0082] S8. Collect the system feedback status after the control is executed, and use the feedback status and the previous status as input to perform reinforcement learning to update the policy network model.

[0083] The proposed integrated SCADA load control method integrates deep reinforcement learning with swarm intelligence optimization algorithms to establish a complete closed-loop control process: state perception – strategy generation – local optimization – safe execution – feedback update. Compared to traditional control methods based on rules or static optimization, this method can adapt to dynamic operating environments, autonomously learn control strategies, and achieve significant improvements in control accuracy and response speed, offering greater flexibility, scalability, and intelligent decision-making capabilities.

[0084] In this embodiment, S1 specifically includes:

[0085] S11, collecting real-time operating data of multiple controlled nodes from the SCADA system, wherein the operating data includes the voltage value, current value, load power value of each node and the connection relationship information between the nodes;

[0086] S12, performing data preprocessing on the collected voltage value, current value, and load power value, wherein the data preprocessing includes time synchronization processing, invalid value elimination, and dimension unification processing;

[0087] S13. Construct a corresponding node state subvector based on the processed electrical parameters of each node and its connection relationship in the power grid, wherein the node state subvector includes the node number, current voltage value, current current value, current load power value, and a set of associated adjacent node numbers;

[0088] S14. Combine the state sub-vectors of all nodes according to the power grid topology to generate a system state vector, which comprehensively represents the operating state of the SCADA system at the current time step. The system state vector serves as the input basis for the subsequent strategy network model.

[0089] This invention uses structured modeling of SCADA system operational data to construct a system state vector, creating a standardized input data interface. This improves the control algorithm's ability to handle high-dimensional, complex inputs. Compared to traditional manual point selection or threshold setting methods, this invention achieves a comprehensive and accurate description of the system state, providing a reliable basis for subsequent policy functions and significantly improving control accuracy and the integration efficiency of upstream and downstream models.

[0090] In this embodiment, S2 specifically includes:

[0091] S21, the system state vector S to be constructed t Input to the policy network model, which is a function approximation model based on deep reinforcement learning, and is represented by a parameterized structure π θ (S t ), where π θ represents the policy function defined by the parameter set θ;

[0092] S22, constructing an action space A, wherein the action space includes the control target node number, the load adjustment range of the corresponding node, the standby node activation flag and the network structure adjustment instruction, and the action vector is represented by the strategy function π θ (S t ) output, represented as a t =π θ (S t ), where a t ∈A represents the specific control action at time step t;

[0093] S23. Define the reward function R t , evaluate action a t In state S t The impact on the operation effect of the SCADA system under the current environment, the reward function includes energy consumption cost term, security constraint penalty term and load balancing incentive term;

[0094] S24, using a deep reinforcement learning algorithm to train and optimize the policy network model, the algorithm includes a state transfer process, action selection, reward evaluation and network parameter update process, using the historical state transfer sequence (S t ,a t ,R t ,S t+1 ) Perform gradient updates on the policy network and update the policy parameter set θ to maximize the cumulative expected reward;

[0095] S25, after completing the training, the policy network model retains the current optimal parameter set θ * .

[0096] This paper designs a systematic update process for the learning mechanism of the control policy function. By introducing a reward function and a state transition mechanism to train the policy function, it enables it to continuously learn from historical experience. Compared with fixed rule strategies, this paper provides the policy with environmental awareness and self-optimization capabilities, effectively improving the decision-making performance of SCADA load control in nonlinear and multi-disturbance environments.

[0097] In this embodiment, S3 specifically includes:

[0098] S31. Setting the policy function π θ (S t ) hyperparameter search range, the hyperparameters include the number of hidden layers L, the number of neurons in each layer n l , activation function type f l , learning rate α, and entropy regularization term weight coefficient λ;

[0099] S32, initialize the individual population of the whale optimization algorithm {X1,X2,...,X K}, where each individual X k Represents a set of policy functions π θ (S t ) of the hyperparameter combination, the population size is K, and each set of hyperparameters X k Applied to the training strategy function and calculate the corresponding cumulative expected reward value R on the verification data k ;

[0100] S33. In each iteration, the individual X is updated according to the update mechanism of the whale optimization algorithm. k Perform location update, the update formula is:

[0101]

[0102] in, Indicates the maximum cumulative expected reward value R in the current t generation k The optimal individual is A=2a·r1-a, C=2r2, where a is the convergence factor, r1 and r2 are random numbers in the range of [0,1];

[0103] S34, each updated hyperparameter individual Applied to the policy function π θ (S t ) training process and recalculate its corresponding cumulative expected reward value At the same time, update the new optimal individual in this generation

[0104] S35. Repeat the individual update and reward evaluation process until the termination condition of the whale optimization algorithm is met, wherein the termination condition includes the convergence of the cumulative expected reward value or the maximum number of iterations.

[0105] S36, the final optimal hyperparameter combination X * Applied to the subsequent policy function π θ (S t )’s structural configuration and training settings as the basic strategy function for generating control action vectors.

[0106] This paper introduces the whale optimization algorithm to perform a global search and optimization of the structural hyperparameters of the policy function, addressing the problem of deep reinforcement learning policy model structure design relying on experience and being difficult to generalize. Compared to traditional methods that use fixed structures or manual trial-and-error parameter adjustment, this paper automatically finds the optimal policy structure, improving the performance stability and generalization ability of the control strategy and reducing model training costs.

[0107] In this embodiment, the S4 specifically includes:

[0108] S41, the optimized strategy function π θ (S t ) After configuration is completed, it is used for online reasoning;

[0109] S42, the system state vector S of the current time step t Input to the policy function π θ (S t ), the corresponding control action vector a is output by the strategy function t ;

[0110] S43, the control action vector a tIt includes multiple sub-commands, each of which contains the target node number, load adjustment range, standby node enablement status and network structure switching flag.

[0111] Based on the optimized policy function, this method dynamically inputs the current system state to generate control actions for real-time scenarios. This method can quickly generate optimal control responses under different load conditions, boasting strong real-time performance and a clear output structure. Compared to traditional methods that rely on offline rule matching, it significantly improves the dynamic regulation capabilities of SCADA systems.

[0112] In this embodiment, the S5 specifically includes:

[0113] S51, from the control action vector a t Extract the target node set containing the load adjustment instruction, construct the local control subspace, and generate the local state subvector set corresponding to the target node;

[0114] S52. Initializing a population of individuals of the cockroach swarm algorithm based on the local state subvector, where each individual represents a set of local control action plans, the plans including the node number, load adjustment direction, adjustment amplitude, and load correlation factor with adjacent nodes;

[0115] S53. In each iteration, the fitness values of the individuals in terms of load balance index, voltage stability index and local energy consumption function are evaluated, and the current local optimal individual is selected as the target direction;

[0116] S54: Update individual positions according to the behavior mechanism of the cockroach swarm algorithm, wherein the behavior mechanism includes:

[0117] Neighborhood optimal movement: each individual moves closer to the current local optimal individual;

[0118] Memory retention: retain historical optimal solutions for comparison and update;

[0119] Obstacle avoidance perturbation: introducing small-scale random perturbations to escape local optimality;

[0120] Synchronous propagation: broadcast the updated information of excellent individuals in the population;

[0121] S55. When the local termination conditions are met, including the fitness function convergence or the number of iterations reaching the set upper limit, the current optimal local control action plan is output;

[0122] S56, replacing the control action vector a with the local optimal control action plan t The corresponding sub-vector part in the local optimization is obtained to obtain the control action vector a′ t .

[0123] This invention innovatively incorporates a cockroach swarm algorithm to perform refined optimization on local nodes involved in control actions. This achieves high precision and adaptability in local load control while maintaining global strategic objectives. Compared to using reinforcement learning alone, this invention exhibits greater local robustness and control accuracy in response to local anomalies such as sudden node loads and boundary disturbances, thereby improving overall system energy efficiency.

[0124] In this embodiment, S6 specifically includes:

[0125] S61, the locally optimized control action vector a′ t Input to the safety constraint judgment module;

[0126] S62. Screening each control instruction based on preset operation safety rules, wherein the safety rules include node voltage limit, current upper limit, line load rate threshold, and topology switchability judgment;

[0127] S63. Eliminate the control instructions that do not meet the safety constraints, retain the actions that meet all the constraints, and form the filtered control action vector for issuance and execution.

[0128] This invention incorporates a safety constraint judgment mechanism for control actions, enabling real-time detection of electrical safety boundary conditions before policy execution, thereby protecting system operation from unauthorized instructions. Compared to unconstrained output or post-correction approaches, this invention incorporates risk control upfront, ensuring both intelligent and secure policies and improving system reliability.

[0129] In this embodiment, the S7 specifically includes:

[0130] S71, sending the control action vector that has passed the security screening as a control instruction to the SCADA system execution layer;

[0131] S72. Control the corresponding load device to perform start / stop, switch, or power adjustment operations according to the node number, load adjustment range, and network structure adjustment information in the control instruction;

[0132] S73. After the operation is completed, the execution result is recorded and the system status is updated synchronously.

[0133] This invention distributes safety-screened control action instructions to the execution layer, ensuring that policy implementation aligns with on-site device behavior, creating a closed-loop path from algorithm generation to physical control. Compared to traditional control methods, which suffer from long policy deployment delays and high execution errors, this invention improves response efficiency and ensures the control system's ability to rapidly respond to complex load conditions.

[0134] In this embodiment, S8 specifically includes:

[0135] S81, collect the system state after the control action is executed, and construct a new system state vector S t+1 ;

[0136] S82, the current state vector S t , control action vector a′ t , reward value R t and the subsequent state vector S t+1 Composition state transition samples;

[0137] S83. Use the state transition sample to update the strategy function π θ (S t ) to improve the adaptability and optimality of the control strategy in the future state.

[0138] By collecting state feedback after control execution and constructing state transition samples to update the policy function, this invention achieves continuous self-optimization and online learning of control strategies. This feedback mechanism effectively avoids the problem of static policies, enabling the system to continuously adapt to environmental changes during long-term operation, and improving the overall level of intelligent and autonomous control.

[0139] Example 1:

[0140] To verify the feasibility of the present invention in practice, the present invention was applied to a regional load control scenario in a certain city's smart grid SCADA system. The area covers 7 medium-voltage distribution substations and connects 54 downstream load nodes. The types of electricity consumption involved include industrial load (38%), commercial load (24%), and residential load (38%). This grid structure has the typical characteristics of rapid dynamic changes, unbalanced loads, and significant peak power consumption. The traditional rule-based SCADA control method has problems with response delays and inaccurate control when dealing with sudden industrial outages during holidays and grid switching surges. In severe cases, it can even cause regional voltage limit violations.

[0141] In this scenario, we deploy the present invention in the control strategy module, connect to the master station data acquisition interface of the SCADA system, obtain node voltage, current, load power and topology information in real time, build state input, and generate real-time control instructions through the policy function model deployed on a dedicated server. The present invention first optimizes the hyperparameters of the deep reinforcement learning policy function offline through the whale optimization algorithm, and obtains a cumulative reward improvement of about 18.7% compared to the manually set parameters on the benchmark verification set. Subsequently, the trained policy function is used for online control, and the control action is generated in combination with the real-time state. The cockroach swarm algorithm is used to perform local control and refinement on 13 key nodes with dense load in the area to improve the control accuracy. All generated control instructions are issued for execution after verification by the security screening module, and are used to update the policy model in the next round of feedback.

[0142] The experimental period was April 20, 2025, to April 27, 2025, during a typical workweek characterized by significant daily load fluctuations and simultaneous industrial and commercial operations. The testing compared key indicators of the proposed method with those of traditional threshold-based SCADA control methods, including control response time, node load balancing, voltage compliance, energy efficiency, and the number of overloaded nodes. The system operated on a test platform at a municipal power grid control center, with the main control server equipped with a 32-core CPU and 128GB of memory. The average model execution latency did not exceed 0.15 seconds.

[0143] The test results show that the present invention significantly improves the adaptability of the SCADA system to abnormal load conditions. The traditional method has an average response time of 6.4 seconds under load mutation conditions, while the solution of the present invention shortens it to 1.1 seconds; the traditional method has multiple continuous overload nodes, while the present invention can predict the load trend in advance and accurately regulate it, and the overload rate is reduced by 73%; after regulation, the voltage qualification rate of the entire network is increased from 95.1% to 98.6%; the load balance is reduced from a standard deviation of 23.7kW to 9.5kW. In addition, under the same load power supply conditions, the control system of the present invention achieved a daily average energy consumption reduction of about 6.2%, showing its obvious advantages in energy scheduling optimization.

[0144] Table 1: Performance comparison between the present invention and traditional SCADA control strategies

[0145]

[0146] The comparative data in Table 1 clearly demonstrates the superior performance of the present method in SCADA load control. First, in terms of average response time, the present method achieves an 82.8% improvement, achieving a response speed of 1.1 seconds, significantly lower than the 6.4 seconds of the traditional method. This demonstrates that the present method can more quickly generate and issue effective control commands under dynamic system changes, effectively resolving the lag in response experienced by traditional systems.

[0147] Secondly, in terms of the number of node overloads, the traditional SCADA method experienced 37 node overloads during the test period, while the method of the present invention reduced this number to 10 times, a reduction of 73.0%, demonstrating that it has greater initiative and foresight in load trend prediction and real-time adjustment capabilities, significantly enhancing the system's operational safety and stability.

[0148] The voltage compliance rate, while maintained at 95.1% by the traditional method, increased to 98.6% by the proposed method. This demonstrates that the combination of the strategy function and the local adjustment mechanism effectively maintains voltage balance and power quality in the distribution network. Regarding load balancing, the traditional method exhibits significant variation between nodes, with a standard deviation of 23.7kW. However, the proposed method limits this to within 9.5kW, resulting in more balanced load distribution and demonstrating the fine-tuning capabilities of the local optimization mechanism.

[0149] In terms of energy conservation, traditional SCADA systems rely primarily on static rules and are unable to achieve real-time energy efficiency optimization. However, the solution presented in this paper achieves a 6.2% daily average energy consumption reduction while maintaining a stable power supply, demonstrating significant energy savings. Regarding the average execution time of policies, the proposed solution achieves an average control delay of only 0.15 seconds, far superior to the 0.41 seconds of traditional methods, ensuring rapid system response and real-time performance.

[0150] Furthermore, regarding the adaptive adjustment capabilities of control strategies, traditional methods rely on static rules and lack a self-updating mechanism. This invention, however, incorporates a continuous learning mechanism based on deep reinforcement learning, enabling the strategy to continuously optimize based on load behavior. Finally, regarding the accuracy of local load regulation, traditional methods typically employ coarse adjustments based on regions. This invention, by incorporating a cockroach swarm algorithm, achieves fine control within a ±2kW range, significantly improving the resolution and accuracy of node-level load control.

[0151] Based on the above data, the present invention demonstrates technical effects that are significantly superior to traditional SCADA systems in terms of intelligent regulation, safety assurance, energy conservation and consumption reduction, and fine control, fully verifying its engineering feasibility and innovative value.

[0152] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A SCADA load control method based on deep reinforcement learning, characterized in that: The steps include: S1, collect the operating data in the SCADA system and construct the current system state vector; S2. Based on the system state vector, a policy network model is constructed using a deep reinforcement learning algorithm; S3. Globally optimize the structural hyperparameters of the policy network model using the whale optimization algorithm, with the optimization goal being to maximize the cumulative reward value on the training set; S4. Based on the optimized policy network model, a control action vector is generated according to the current system state vector; S5. Apply the cockroach swarm algorithm to perform local search optimization in the neighborhood action space for the control action vector to generate an optimized control action vector; S6. Input the optimized control action vector into the safety constraint judgment module, filter the actions according to the upper and lower voltage limits, current limit values and topology switching constraints, and eliminate the control actions that do not meet the operation safety conditions; S7, sending the filtered control actions to the SCADA system execution layer to control the load equipment to perform corresponding adjustment operations; S8. Collect the system feedback status after the control is executed, and use the feedback status and the previous status as input to perform reinforcement learning to update the policy network model.

2. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that, Said S1 specifically includes: S11, collecting real-time operating data of multiple controlled nodes from the SCADA system, wherein the operating data includes the voltage value, current value, load power value of each node and the connection relationship information between the nodes; S12, performing data preprocessing on the collected voltage value, current value, and load power value, wherein the data preprocessing includes time synchronization processing, invalid value elimination, and dimension unification processing; S13. Construct a corresponding node state subvector based on the processed electrical parameters of each node and its connection relationship in the power grid, wherein the node state subvector includes the node number, current voltage value, current current value, current load power value, and a set of associated adjacent node numbers; S14. Combine the state sub-vectors of all nodes according to the power grid topology to generate a system state vector, which comprehensively represents the operating state of the SCADA system at the current time step. The system state vector serves as the input basis for the subsequent strategy network model.

3. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that, The S2 specifically includes: S21, the system state vector S to be constructed t Input to the policy network model, which is a function approximation model based on deep reinforcement learning, and is represented by a parameterized structure π θ (S t ), where π θ represents the policy function defined by the parameter set θ; S22, constructing an action space A, wherein the action space includes the control target node number, the load adjustment range of the corresponding node, the standby node activation flag and the network structure adjustment instruction, and the action vector is represented by the strategy function π θ (S t ) output, represented as a t =π θ (S t ), where a t ∈A represents the specific control action at time step t; S23. Define the reward function R t , evaluate action a t In state S t The impact on the operation effect of the SCADA system under the current environment, the reward function includes energy consumption cost term, security constraint penalty term and load balancing incentive term; S24, using a deep reinforcement learning algorithm to train and optimize the policy network model, the algorithm includes a state transfer process, action selection, reward evaluation and network parameter update process, using the historical state transfer sequence (S t ,a t ,R t ,S t+1 ) Perform gradient updates on the policy network and update the policy parameter set θ to maximize the cumulative expected reward; S25, after completing the training, the policy network model retains the current optimal parameter set θ * .

4. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that, The S3 specifically includes: S31. Setting the policy function π θ (S t ) hyperparameter search range, the hyperparameters include the number of hidden layers L, the number of neurons in each layer n l , activation function type f l , learning rate α, and entropy regularization term weight coefficient λ; S32, initialize the individual population of the whale optimization algorithm {X1,X2,...,X K }, where each individual X k Represents a set of policy functions π θ (S t ) of the hyperparameter combination, the population size is K, and each set of hyperparameters X k Applied to the training strategy function and calculate the corresponding cumulative expected reward value R on the verification data k ; S33. In each iteration, the individual X is updated according to the update mechanism of the whale optimization algorithm. k Perform location update, the update formula is: in, Indicates the maximum cumulative expected reward value R in the current t generation k The optimal individual is A=2a·r1-a, C=2r2, where a is the convergence factor, r1 and r2 are random numbers in the range of [0,1]; S34, each updated hyperparameter individual Applied to the policy function π θ (S t ) training process and recalculate its corresponding cumulative expected reward value At the same time, update the new optimal individual in this generation S35. Repeat the individual update and reward evaluation process until the termination condition of the whale optimization algorithm is met, wherein the termination condition includes the convergence of the cumulative expected reward value or the maximum number of iterations. S36, the final optimal hyperparameter combination X * Applied to the subsequent policy function π θ (S t )’s structural configuration and training settings as the basic strategy function for generating control action vectors.

5. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that, The S4 specifically includes: S41, the optimized strategy function π θ (S t ) After configuration is completed, it is used for online reasoning; S42, the system state vector S of the current time step t Input to the policy function π θ (S t ), the corresponding control action vector a is output by the strategy function t ; S43, the control action vector a t It includes multiple sub-commands, each of which contains the target node number, load adjustment range, standby node enablement status and network structure switching flag.

6. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that, The S5 specifically includes: S51, from the control action vector a t Extract the target node set containing the load adjustment instruction, construct the local control subspace, and generate the local state subvector set corresponding to the target node; S52. Initializing a population of individuals of the cockroach swarm algorithm based on the local state subvector, where each individual represents a set of local control action plans, the plans including the node number, load adjustment direction, adjustment amplitude, and load correlation factor with adjacent nodes; S53. In each iteration, the fitness values of the individuals in terms of load balance index, voltage stability index and local energy consumption function are evaluated, and the current local optimal individual is selected as the target direction; S54: Update individual positions according to the behavior mechanism of the cockroach swarm algorithm, wherein the behavior mechanism includes: Neighborhood optimal movement: each individual moves closer to the current local optimal individual; Memory retention: retain historical optimal solutions for comparison and update; Obstacle avoidance perturbation: introducing small-scale random perturbations to escape local optimality; Synchronous propagation: broadcast the updated information of excellent individuals in the population; S55. When the local termination conditions are met, including the fitness function convergence or the number of iterations reaching the set upper limit, the current optimal local control action plan is output; S56, replacing the control action vector a with the local optimal control action plan t The corresponding sub-vector part in the local optimization is obtained to obtain the control action vector a′ t .

7. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that: The S6 specifically includes: S61, the locally optimized control action vector a′ t Input to the safety constraint judgment module; S62. Screening each control instruction based on preset operation safety rules, wherein the safety rules include node voltage limit, current upper limit, line load rate threshold, and topology switchability judgment; S63. Eliminate the control instructions that do not meet the safety constraints, retain the actions that meet all the constraints, and form the filtered control action vector for issuance and execution.

8. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that: The S7 specifically includes: S71, sending the control action vector that has passed the security screening as a control instruction to the SCADA system execution layer; S72. Control the corresponding load device to perform start / stop, switch, or power adjustment operations according to the node number, load adjustment range, and network structure adjustment information in the control instruction; S73. After the operation is completed, the execution result is recorded and the system status is updated synchronously.

9. A SCADA load control method based on deep reinforcement learning according to claim 1, characterized in that: The S8 specifically includes: S81, collect the system state after the control action is executed, and construct a new system state vector S t+1 ; S82, the current state vector S t , control action vector a′ t , reward value R t and the subsequent state vector S t+1 Composition state transition samples; S83. Use the state transition sample to update the strategy function π θ (S t ) to improve the adaptability and optimality of the control strategy in the future state.

Images