Breaker adaptive protection method based on reinforcement learning

Abstract

The invention discloses a circuit breaker adaptive protection method based on reinforcement learning, and the method comprises the following steps: collecting the original operation data of a circuit breaker, and carrying out the preprocessing; a DreamerV2 model is constructed; carrying out modeling on a state evolution process by utilizing a world model sub-module in the DreamerV2 model; generating observation reconstruction loss and reward estimation loss according to the potential state sequence and the hidden state sequence; making a decision on the current potential state through a strategy optimization module, and outputting a circuit breaker control action; the control action is applied to the circuit breaker, and a reward value composed of an actual execution result and a feedback state is obtained; and constructing a strategy optimization objective function and a joint loss function to update the strategy network and the world model sub-module respectively. According to the method, the DreamV2 reinforcement learning model is introduced, self-adaptive optimization of the circuit breaker control strategy is achieved, and the method has the advantages of being rapid in response and high in strategy precision.

Description

Technical Field

[0001] This invention relates to the field of power system protection, and in particular to an adaptive protection method for circuit breakers based on reinforcement learning. Background Technology

[0002] In existing power systems, circuit breakers, as critical protection devices, typically rely on fixed thresholds or pre-defined rules for their tripping strategies. This makes them unable to dynamically adapt to the complex changes in the power grid's operating state. For example, when the system experiences short-term disturbances or fluctuations, traditional protection logic may misjudge, leading to false tripping or delayed response, thus affecting power supply continuity and equipment safety. Furthermore, existing methods generally lack in-depth mining and modeling of historical circuit breaker operation data, making it difficult to form an effective prediction and evolutionary understanding of the system state. They are unable to make reasonable inferences under unobserved conditions, resulting in insufficient generalization ability of protection strategies.

[0003] Especially in modern multi-source heterogeneous complex power grids, the protected objects of circuit breakers have nonlinear, non-stationary, and high-dimensional characteristics. Traditional control strategies based on thresholds or empirical rules cannot effectively model high-frequency state transitions and time-series characteristics, and lack a real-time feedback learning mechanism for the effects of circuit breaker control actions. Therefore, existing methods still have significant limitations in dealing with dynamic fault response and adaptive protection strategy optimization, and it is urgent to introduce reinforcement learning methods with state modeling and strategy updating capabilities to improve them. Summary of the Invention

[0004] One objective of this invention is to propose an adaptive protection method for circuit breakers based on reinforcement learning. This invention introduces the DreamerV2 reinforcement learning model and constructs a modular architecture including observation coding, state modeling, reward prediction, and policy optimization. It dynamically senses the evolution trend of the power system state and realizes intelligent decision-making and feedback correction of circuit breaker control actions. This method not only improves the real-time performance and accuracy of fault response but also has good generalization ability and environmental adaptability. It can flexibly adjust and efficiently protect the circuit breaker operating state under complex conditions and has the advantages of rapid response, high policy accuracy, and strong self-learning ability.

[0005] An adaptive protection method for circuit breakers based on reinforcement learning according to an embodiment of the present invention includes the following steps: Collect raw operating data of the circuit breaker, preprocess the raw operating data, and construct standardized observation vectors; Construct the DreamerV2 model, which includes a world model submodule and a policy optimization module; The standardized observation vectors are input into the world model submodule of the DreamerV2 model, and the world model submodule is used to model the state evolution process to generate potential state sequences and hidden state sequences. Generate observation reconstruction loss and reward estimation loss based on the latent state sequence and hidden state sequence; The strategy optimization module makes decisions about the current potential state, the strategy network outputs the circuit breaker control action, and the value function network outputs the state value estimate. Apply control actions to the circuit breaker and obtain a reward value consisting of the actual execution result and the feedback status; The policy network is updated by constructing a policy optimization objective function based on the reward value, the estimated state value, and the uncertainty measure of the potential state. A joint loss function is constructed based on the observation reconstruction loss, the reward estimation loss, and the uncertainty measure of the potential state to perform posterior correction on the world model submodule, thereby performing adaptive optimization of the circuit breaker protection policy.

[0006] Optionally, the raw operating data includes current, voltage, frequency, phase, and circuit breaker status information. The preprocessing includes missing value completion, outlier removal, one-hot encoding, and numerical normalization. The circuit breaker status information represents the on / off state of the circuit breaker at the current sampling time, with 0 indicating that the circuit breaker is in the on state and 1 indicating that the circuit breaker is in the off state.

[0007] Optionally, the world model submodule includes an observation encoding module, a cyclic state space modeling module, a decoder module, and a reward prediction module; The cyclic state-space modeling module includes a gated cyclic unit, a prior network, and a posterior network. The observation coding module performs feature embedding on the standardized observation vector; The cyclic state-space modeling module combines the circuit breaker's historical potential states, historical control actions, and current observation information to generate the potential state at the current time step, simulating the dynamic changes in the circuit breaker's operating environment. The decoder module maps the latent state to the observation reconstruction vector of the circuit breaker at the current moment, which is used to evaluate the expressive power of the latent space and the consistency of the model. The reward prediction module outputs the corresponding instant reward value based on the current potential state and control action, providing expected feedback signals for strategy optimization; The policy optimization module includes a policy network and a value function network; The policy network is used to generate the action policy output of the circuit breaker based on the current potential state; The value function network is used to evaluate the expected long-term return corresponding to the current potential state, and to assist in the behavioral evaluation during the policy training process. The world model submodule and policy optimization module in the DreamerV2 model constitute a collaborative architectural unit. The world model submodule provides the ability to represent the latent environment, while the policy optimization module performs action learning and policy updates based on the latent space.

[0008] Optionally, generating the latent state sequence and the hidden state sequence includes: The preprocessed, standardized observation vectors are input into the world model submodule; The standardized observation vector is embedded with features using an observation coding module to obtain embedded features. The observation coding module is a multilayer perceptron structure. The potential state of the previous time step Controlling actions Hidden state The input vector is concatenated with the currently embedded features and fed into the cyclic state space modeling module, where the current hidden state is updated using a gated cyclic unit. The current hidden state is input into the prior network, which is a two-layer fully connected structure. The current hidden state is concatenated with the embedded features and then input into the posterior network, which is a two-layer fully connected network. Sample the current latent state from the posterior distribution of the latent state; Repeat the above process to obtain the potential and deterministic states of all time steps in sequence, forming a potential state sequence and a hidden state sequence; The potential state sequence and the hidden state sequence are input into the strategy optimization module of the DreamerV2 model, and used as the basic input for control strategy generation and circuit breaker protection response adjustment.

[0009] Optionally, the generation observation reconstruction loss and reward estimation loss include: The latent state sequence and the hidden state sequence are concatenated into a state combination vector, and the state combination vector is passed as input to the decoder module and the reward prediction module. The state combination vector is input into the decoder module to generate the observation reconstruction vector. The decoder module is a multi-layer fully connected network. The decoder module output is compared with the standardized observation vector to calculate the observation reconstruction loss; Simultaneously, the state combination vector is input into the reward prediction module, which outputs the instantaneous reward value corresponding to the current state. The reward prediction module is a neural network composed of two fully connected layers. The instant reward value is compared with the actual reward generated by the feedback of the circuit breaker's action to calculate the estimated loss.

[0010] Optionally, the decision-making process for the current potential state through the strategy optimization module includes: The state combination vector is input into the policy network, and the policy network outputs the probability distribution of control actions. The control action is selected based on the probability distribution of the control action, and the control action includes immediate tripping, delayed tripping, and keeping the circuit open. The immediate tripping corresponds to the emergency disconnection state of the circuit breaker; The delayed trip corresponds to disconnecting the circuit after a set delay time. The "keep connected" option corresponds to maintaining the current electrical connection state unchanged. Simultaneously, the state combination vector is input into the value function network to calculate the state value estimate, which is used for target guidance during the policy optimization process.

[0011] Optionally, obtaining the reward value composed of the actual execution result and the feedback status includes: Based on the control actions generated by the strategy optimization module, execute the corresponding trip, delayed trip, or keep-on operation; After the control action is completed, feedback information generated by the circuit breaker is collected. The feedback information includes the action response delay time, the trip status protection trigger flag, and the observation vector at the next moment. A reward function is constructed based on the feedback information to evaluate the effectiveness and timeliness of the current control action; The reward function is evaluated based on the effectiveness and timeliness of the circuit breaker protection behavior, including: If the control action successfully completes the trip under fault conditions, a positive reward of 1 is given. If the control action mistakenly triggers a trip under fault-free system conditions, a negative penalty will be imposed, and the reward value will be 0. If there is a significant delay in the trip response, a corresponding penalty will be imposed based on the degree of delay. The reward value is in the range of [0,1]. The higher the degree of delay, the lower the reward value. The reward value calculated by the reward function, along with the state combination vector and control action corresponding to the current time step, is fed back to the policy optimization module and the world model submodule for dynamic adjustment and optimization of the circuit breaker control policy during training.

[0012] Optionally, the adaptive optimization of the circuit breaker protection strategy includes: Based on the posterior and prior networks, a measure of uncertainty in the latent state is constructed: ; in, Representing the potential state Uncertainty measure This represents the distribution of latent states generated by the posterior network. This represents the distribution of latent states generated by the prior network. This represents the Kullback-Leibler divergence function, which measures the difference between the previous and subsequent distributions; Construct a policy optimization objective function that incorporates uncertainty, and update the policy network parameters by minimizing the policy optimization objective function: ; in, This represents the policy optimization objective function of the policy network. This represents the probability distribution of control actions, describing the situation in the latent state. Take action below The probability, This represents the uncertainty adjustment coefficient. The value function network represents the current potential state. State value estimate, The parameters of the policy network are represented. Represents the mathematical expectation. This represents the reward value generated based on the feedback from the circuit breaker's actions. A joint loss function is constructed, and the network parameter set in the world model submodule is updated by minimizing the joint loss function. The joint loss function includes reward prediction error, observation reconstruction error, and posterior bias constraint. The optimized policy network and world model submodule are synchronously passed to the model input cache of the next time step, forming a continuous optimization process of circuit breaker control policy driven by potential states, and realizing the closed-loop fusion of fault response and policy evolution.

[0013] The beneficial effects of this invention are: This invention proposes an adaptive protection method for circuit breakers based on reinforcement learning, which can overcome the problems of hysteresis and high misjudgment rate of traditional protection strategies in response to dynamic changes in power grid state. By introducing the DreamerV2 model, a world model submodule with potential state expression capability and a strategy optimization module with strategy decision-making capability are constructed. This invention realizes state-driven and dynamic updating of circuit breaker control logic. This method can not only perform long-term state prediction and control action simulation in complex nonlinear environments, but also perform closed-loop feedback learning based on actual execution results, effectively enhancing the timeliness and robustness of circuit breaker strategy.

[0014] Furthermore, by embedding features into the observation data, modeling state evolution, generating action strategies, and fusing reward feedback, this invention enables circuit breakers to accurately judge and adaptively adjust tripping behavior when facing complex operating conditions and sudden faults. The joint loss function mechanism collaboratively optimizes observation reconstruction errors, reward prediction errors, and potential state uncertainties during the training process, improving the model's expressive power and generalization performance, thereby ensuring the intelligent operation and safety protection of circuit breakers in the power system. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0016] Figure 1 This is a flowchart of a circuit breaker adaptive protection method based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of the DreamerV2 model, which is a circuit breaker adaptive protection method based on reinforcement learning proposed in this invention. Figure 3 This is a structural diagram of the world model submodule of a circuit breaker adaptive protection method based on reinforcement learning proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figure 1-3 An adaptive protection method for circuit breakers based on reinforcement learning includes the following steps: Collect raw operating data of the circuit breaker, preprocess the raw operating data, and construct standardized observation vectors; Construct the DreamerV2 model, which includes a world model submodule and a policy optimization module; The standardized observation vectors are input into the world model submodule of the DreamerV2 model, and the world model submodule is used to model the state evolution process to generate potential state sequences and hidden state sequences. Generate observation reconstruction loss and reward estimation loss based on the latent state sequence and hidden state sequence; The strategy optimization module makes decisions about the current potential state, the strategy network outputs the circuit breaker control action, and the value function network outputs the state value estimate. Apply control actions to the circuit breaker and obtain a reward value consisting of the actual execution result and the feedback status; The policy network is updated by constructing a policy optimization objective function based on the reward value, the estimated state value, and the uncertainty measure of the potential state. A joint loss function is constructed based on the observation reconstruction loss, the reward estimation loss, and the uncertainty measure of the potential state to perform posterior correction on the world model submodule, thereby performing adaptive optimization of the circuit breaker protection policy.

[0019] This invention proposes an adaptive protection method for circuit breakers based on reinforcement learning. It integrates the DreamerV2 model to construct a latent environment representation and a strategy optimization mechanism, enabling continuous learning and dynamic updating of the circuit breaker control strategy. By modeling the operating state of the power system, this method can effectively identify abnormal situations under complex operating conditions and adaptively generate optimal control actions, improving the response accuracy and protection capability of the circuit breaker, enhancing the system's intelligence, adaptability, and robustness, and providing an innovative solution for the intelligent control of power equipment.

[0020] In this embodiment, the original operating data includes current, voltage, frequency, phase, and circuit breaker status information. The preprocessing includes missing value completion, outlier removal, one-hot encoding, and numerical normalization. The circuit breaker status information represents the on / off state of the circuit breaker at the current sampling time, where 0 indicates that the circuit breaker is in the on state and 1 indicates that the circuit breaker is in the off state.

[0021] This invention fully reflects the multidimensional physical state of the power system in which the circuit breaker is located by collecting raw operational data, including current, voltage, frequency, phase, and circuit breaker status information. At the same time, it effectively improves data integrity and robustness by completing missing values ​​and removing outliers. It uses one-hot encoding to process discrete variables and numerical normalization to unify feature dimensions, making the data structure more standardized and facilitating efficient training and inference of subsequent models. In particular, the circuit breaker status information uses 0 and 1 to represent its on / off state, providing a precise label of the current equipment behavior for the reinforcement learning model, ensuring the semantic clarity of input observations and the consistency of control command response, and improving the input quality of circuit breaker control strategy optimization.

[0022] In this embodiment, the world model submodule includes an observation encoding module, a cyclic state space modeling module, a decoder module, and a reward prediction module; The cyclic state-space modeling module includes a gated cyclic unit, a prior network, and a posterior network. The observation coding module performs feature embedding on the standardized observation vector; The cyclic state-space modeling module combines the circuit breaker's historical potential states, historical control actions, and current observation information to generate the potential state at the current time step, simulating the dynamic changes in the circuit breaker's operating environment. The decoder module maps the latent state to the observation reconstruction vector of the circuit breaker at the current moment, which is used to evaluate the expressive power of the latent space and the consistency of the model. The reward prediction module outputs the corresponding instant reward value based on the current potential state and control action, providing expected feedback signals for strategy optimization; The policy optimization module includes a policy network and a value function network; The policy network is used to generate the action policy output of the circuit breaker based on the current potential state; The value function network is used to evaluate the expected long-term return corresponding to the current potential state, and to assist in the behavioral evaluation during the policy training process. The world model submodule and policy optimization module in the DreamerV2 model constitute a collaborative architectural unit. The world model submodule provides the ability to represent the latent environment, while the policy optimization module performs action learning and policy updates based on the latent space.

[0023] In constructing the DreamerV2 model, this invention refines the design of a world model sub-module, including observation encoding, cyclic state modeling, decoder, and reward prediction. This sub-module, along with a policy network and value function network, forms a policy optimization module. This enables the model to achieve accurate modeling and action decision-making in the latent state space. Through modular modeling, it enhances the dynamic modeling capability of the circuit breaker's operating environment and improves the trainability and generalization performance of the control strategy. This achieves synergistic coupling between world modeling and policy optimization, thereby improving the intelligence level of the circuit breaker protection strategy.

[0024] In this embodiment, generating the latent state sequence and the hidden state sequence includes: The preprocessed, standardized observation vectors are input into the world model submodule; The standardized observation vector is embedded with features using an observation coding module to obtain embedded features. The observation coding module is a multilayer perceptron structure. The potential state of the previous time step Controlling actions Hidden state The input vector is concatenated with the currently embedded features and fed into the cyclic state space modeling module, where the current hidden state is updated using a gated cyclic unit. The current hidden state is input into the prior network, which is a two-layer fully connected structure. The current hidden state is concatenated with the embedded features and then input into the posterior network, which is a two-layer fully connected network. Sample the current latent state from the posterior distribution of the latent state; Repeat the above process to obtain the potential and deterministic states of all time steps in sequence, forming a potential state sequence and a hidden state sequence; The potential state sequence and the hidden state sequence are input into the strategy optimization module of the DreamerV2 model, and used as the basic input for control strategy generation and circuit breaker protection response adjustment.

[0025] This invention constructs a complete potential state generation process, utilizing an observation coding module to extract features, a gated recurrent unit to capture time dependencies, and prior and posterior networks to model the potential state distribution, thereby achieving dynamic modeling of the circuit breaker's potential state in a time series. This process integrates historical states, current observations, and control action information, effectively enhancing the model's ability to grasp the evolution trend of the power system, providing a continuous and interpretable potential space expression for subsequent strategy optimization, and improving the accuracy and foresight of the circuit breaker protection strategy.

[0026] In this embodiment, the generation observation reconstruction loss and reward estimation loss include: The latent state sequence and the hidden state sequence are concatenated into a state combination vector, and the state combination vector is passed as input to the decoder module and the reward prediction module. The state combination vector is input into the decoder module to generate the observation reconstruction vector. The decoder module is a multi-layer fully connected network. The decoder module output is compared with the standardized observation vector to calculate the observation reconstruction loss; Simultaneously, the state combination vector is input into the reward prediction module, which outputs the instantaneous reward value corresponding to the current state. The reward prediction module is a neural network composed of two fully connected layers. The instant reward value is compared with the actual reward generated by the feedback of the circuit breaker's action to calculate the estimated loss.

[0027] This invention reconstructs observed features by reverse mapping the latent state sequence through a decoder module and models the expected reward value using a reward prediction module. It calculates the reconstruction loss and the reward estimation loss separately, thereby strengthening the model's self-supervised constraint on the accuracy of state representation. This not only improves the world model's ability to fit real observed data but also optimizes the expected feedback support for the policy optimization module. Furthermore, it achieves tight coupling between circuit breaker behavior prediction and control policy learning, enhancing the system's learning adaptability and convergence speed.

[0028] In this embodiment, the decision-making process for the current potential state through the strategy optimization module includes: The state combination vector is input into the policy network, and the policy network outputs the probability distribution of control actions. The control action is selected based on the probability distribution of the control action, and the control action includes immediate tripping, delayed tripping, and keeping the circuit open. The immediate tripping corresponds to the emergency disconnection state of the circuit breaker; The delayed trip corresponds to disconnecting the circuit after a set delay time. The "keep connected" option corresponds to maintaining the current electrical connection state unchanged. Simultaneously, the state combination vector is input into the value function network to calculate the state value estimate, which is used for target guidance during the policy optimization process.

[0029] This invention generates the probability distribution of circuit breaker control actions based on potential states through a policy network, and estimates the long-term value of the current state using a value function network. This effectively improves the targeting and expected benefits of circuit breaker control actions. The policy optimization structure makes decision outputs based on the potential state space generated by the world model, which strengthens the state dependence and action rationality of the control strategy, improves the protection response accuracy and behavior selection capability of the circuit breaker in complex operating scenarios, and enhances the dynamics and intelligence of system decision-making.

[0030] In this embodiment, obtaining the reward value, which is composed of the actual execution result and the feedback status, includes: Based on the control actions generated by the strategy optimization module, execute the corresponding trip, delayed trip, or keep-on operation; After the control action is completed, feedback information generated by the circuit breaker is collected. The feedback information includes the action response delay time, the trip status protection trigger flag, and the observation vector at the next moment. A reward function is constructed based on the feedback information to evaluate the effectiveness and timeliness of the current control action; The reward function is evaluated based on the effectiveness and timeliness of the circuit breaker protection behavior, including: If the control action successfully completes the trip under fault conditions, a positive reward of 1 is given. If the control action mistakenly triggers a trip under fault-free system conditions, a negative penalty will be imposed, and the reward value will be 0. If there is a significant delay in the trip response, a corresponding penalty will be imposed based on the degree of delay. The reward value is in the range of [0,1]. The higher the degree of delay, the lower the reward value. The reward value calculated by the reward function, along with the state combination vector and control action corresponding to the current time step, is fed back to the policy optimization module and the world model submodule for dynamic adjustment and optimization of the circuit breaker control policy during training.

[0031] This invention deploys control actions generated by a strategy network to a circuit breaker execution system. By collecting reward feedback information from indicators such as action response delay and tripping status feedback, a reward function tailored to power protection needs is constructed. The reward value reflects the rationality and timeliness of the circuit breaker action, which helps to reinforce the learning strategy to positively incentivize effective behaviors and suppress ineffective behaviors. This enhances the system's ability to respond quickly and adaptively to sudden power failures, thereby increasing the practicality and engineering deployment value of the circuit breaker protection strategy.

[0032] In this embodiment, the adaptive optimization of the circuit breaker protection strategy includes: Based on the posterior and prior networks, a measure of uncertainty in the latent state is constructed: ; in, Representing the potential state Uncertainty measure This represents the distribution of latent states generated by the posterior network. This represents the distribution of latent states generated by the prior network. This represents the Kullback-Leibler divergence function, which measures the difference between the previous and subsequent distributions; Construct a policy optimization objective function that incorporates uncertainty, and update the policy network parameters by minimizing the policy optimization objective function: ; in, This represents the policy optimization objective function of the policy network. This represents the probability distribution of control actions, describing the situation in the latent state. Take action below The probability, This represents the uncertainty adjustment coefficient. The value function network represents the current potential state. State value estimate, The parameters of the policy network are represented. Represents the mathematical expectation. This represents the reward value generated based on the feedback from the circuit breaker's actions. A joint loss function is constructed, and the network parameter set in the world model submodule is updated by minimizing the joint loss function. The joint loss function includes reward prediction error, observation reconstruction error, and posterior bias constraint. The optimized policy network and world model submodule are synchronously passed to the model input cache of the next time step, forming a continuous optimization process of circuit breaker control policy driven by potential states, and realizing the closed-loop fusion of fault response and policy evolution.

[0033] This invention constructs a policy optimization objective function and a joint loss function that integrate latent state uncertainty measurement, thereby realizing a dual optimization and update mechanism for the policy network and the world model sub-module. This mechanism comprehensively considers observation reconstruction error, reward prediction error and distribution differences, enhances the model's robustness to data bias and environmental changes, improves the adaptability and robustness of the circuit breaker policy in complex power scenarios, and ensures the continuous optimization and dynamic evolution capability of the adaptive protection policy in real operation.

[0034] Example 1: To verify the feasibility of this invention in practice, it was applied to a 110kV substation power distribution system. This system adopts a bus protection structure composed of multiple circuit breakers. The operating environment is complex and frequently affected by current and frequency fluctuations and load switching. Traditional circuit breaker protection strategies are difficult to respond flexibly according to the real-time environment, resulting in problems such as high false trip rate, large action delay, and delayed system fault response. They cannot achieve rapid closed-loop response and adaptive processing for sudden faults.

[0035] In the aforementioned substation scenario, by deploying an intelligent module with edge computing capabilities at the circuit breaker control terminal, the circuit breaker adaptive protection method based on reinforcement learning proposed in this invention is invoked to achieve real-time acquisition, preprocessing, modeling, decision-making, and control of operating data. The system continuously collects current, voltage, frequency, phase, and breaking status information of the circuit to which the circuit breaker belongs. After missing value completion, outlier removal, and normalization, a standardized observation vector is constructed. This vector is fed into the pre-deployed DreamerV2 model as input. Within the world model submodule, features are first extracted through the observation encoding module, and a latent state sequence is constructed by combining a gated loop structure. The latent state modeling is then completed with the help of prior and posterior networks.

[0036] After the potential state sequence is generated, the observation information is reconstructed by the decoder module, the reward prediction module estimates the feedback of the action, and the policy network makes a decision on the current potential state, outputting the circuit breaker control action and automatically selecting the "immediate trip," "delayed trip," or "keep on" operation command. The value function network simultaneously provides a value estimate of the current state for the evaluation and iteration of the optimization strategy. In the actual control process, the circuit breaker execution module implements the protection action according to the action command output by the policy network. The system transmits back feedback information such as the circuit breaker state change, response delay, and electrical indicators after the action to construct the reward function and correct the model.

[0037] After running for a period of time, through backtracking analysis of the model strategy optimization module and comparison of the recorded results of circuit breaker protection behavior, it was found that the present invention can respond quickly under abnormal conditions such as load change, short-term overcurrent and unplanned voltage drop, and the control action is more selective and timely. Especially when encountering the same type of fault multiple times, the system shows good strategy migration capability and fault identification sensitivity. Compared with the traditional static protection setting method, the potential state-driven control mechanism adopted by the present invention makes the circuit breaker protection action more environmentally adaptable, effectively suppresses the false tripping rate under multi-source disturbance conditions, and shortens the fault location and response delay.

[0038] To verify the performance of the present invention in practice, it was compared with traditional methods, and the results are shown in Table 1.

[0039] Table 1. Comparison of Key Performance of Reinforcement Learning-Based Circuit Breaker Protection Method and Traditional Method

[0040] As shown in Table 1, the present invention has achieved significant optimization in core protection performance. In terms of response time, it has been reduced from 230ms to 105ms, enabling the system to handle sudden faults more promptly and reducing the risk of power transmission interruption. In terms of false tripping rate, the present invention is only 1.2%, which is more than 70% lower than the traditional method. This indicates that it has higher discrimination ability in identifying normal operating conditions, avoids unnecessary tripping operations, and improves the continuity of power grid operation.

[0041] In terms of fault identification accuracy, the traditional method achieves 85.2%, while this invention improves it to 92.4%. Although the increase is only 8.5%, it is still of great value in high-risk circuit breaker protection tasks. The improved accuracy stems from the DreamerV2 model's ability to model potential states and the continuous optimization mechanism of the policy network, which ensures the model's generalization ability in changing environments.

[0042] This invention, through its strategy of automatically iterating and updating twice daily, possesses continuous learning and adaptive capabilities that traditional methods lack. It can continuously adjust the control logic based on the real-time status of the power grid. Furthermore, the system recovery time is significantly reduced to 7 seconds, indicating that the strategy convergence speed and fault self-healing capability are enhanced.

[0043] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for adaptive protection of a circuit breaker based on reinforcement learning, characterized in that, The method comprises the following steps: Collecting original operation data of the circuit breaker and preprocessing the original operation data to construct a standardized observation vector; Building a DreamerV2 model, which comprises a world model submodule and a policy optimization module; Inputting the standardized observation vector into the world model submodule in the DreamerV2 model, modeling the state evolution process by using the world model submodule, and generating a latent state sequence and a hidden state sequence; Generating an observation reconstruction loss and a reward estimation loss according to the latent state sequence and the hidden state sequence; Making a decision on the current latent state by the policy optimization module, outputting a circuit breaker control action by a policy network, and outputting a state value estimate by a value function network; Applying the control action to the circuit breaker and obtaining a reward value composed of an actual execution result and a feedback state; Updating the policy network according to a policy optimization objective function constructed based on the reward value, the state value estimate and an uncertainty measure of the latent state, and performing posterior correction on the world model submodule according to a joint loss function constructed based on the observation reconstruction loss, the reward estimation loss and the uncertainty measure of the latent state, to perform adaptive optimization of the circuit breaker protection strategy.

2. The circuit breaker adaptive protection method based on reinforcement learning according to claim 1, characterized in that, The original operation data comprises current, voltage, frequency, phase and circuit breaker state information, the preprocessing comprises missing value completion, outlier removal, one-hot encoding and numerical normalization, and the circuit breaker state information represents the on-off state of the circuit breaker at the current sampling time, 0 representing that the circuit breaker is in the on state and 1 representing that the circuit breaker is in the off state.

3. The circuit breaker adaptive protection method based on reinforcement learning according to claim 1, characterized in that, The world model submodule comprises an observation encoding module, a recurrent state space modeling module, a decoder module and a reward prediction module; The recurrent state space modeling module comprises a gated recurrent unit, a prior network and a posterior network; The observation encoding module performs feature embedding on the standardized observation vector; The recurrent state space modeling module generates the latent state of the current time step by combining the historical latent state, the historical control action and the current observation information of the circuit breaker, and simulates the dynamic change process of the circuit breaker operating environment; The decoder module maps the latent state to an observation reconstruction vector of the circuit breaker at the current time, which is used to evaluate the expression ability of the latent space and the model consistency; The reward prediction module outputs the corresponding immediate reward value according to the current latent state and the control action, which provides an expected feedback signal for policy optimization; The policy optimization module comprises a policy network and a value function network; The policy network is used to generate an action policy output of the circuit breaker according to the current latent state; The value function network is used to evaluate the long-term expected return of the current latent state, which assists the behavior evaluation in the policy training process; The world model submodule and the policy optimization module in the DreamerV2 model constitute mutually cooperative architecture units, the world model submodule provides the latent environment expression ability, and the policy optimization module performs action learning and policy update based on the latent space.

4. The circuit breaker adaptive protection method based on reinforcement learning according to claim 1, characterized in that, The generation of the latent state sequence and the hidden state sequence comprises: Inputting the preprocessed standardized observation vector into the world model submodule; The observation encoding module is a multi-layer perceptron structure; the latent state of the previous time step , control action , hidden state concatenate with the current embedding feature to form an input vector, input into the recurrent state space modeling module, and update the current hidden state using the gated recurrent unit The current hidden state is input into a prior network, which is a two-layer fully connected structure; The current hidden state and the embedded features are spliced and input into a posterior network, which is a two-layer fully connected network; The current latent state is sampled from the latent state posterior distribution; The above process is repeated to obtain the latent state and deterministic state of all time steps in turn, forming a latent state sequence and a hidden state sequence; The latent state sequence and the hidden state sequence are input into the policy optimization module of the DreamerV2 model as the basis for generating the control policy and adjusting the breaker protection response.

5. The method of claim 1, wherein, The generation of the observation reconstruction loss and the reward estimation loss includes: The latent state sequence and the hidden state sequence are spliced into a state combination vector, and the state combination vector is input into the decoder module and the reward prediction module; The state combination vector is input into the decoder module to generate an observation reconstruction vector, and the decoder module is a multi-layer fully connected network; The decoder module output is compared with the normalized observation vector to calculate the observation reconstruction loss; At the same time, the state combination vector is input into the reward prediction module to output the immediate reward value corresponding to the current state, and the reward prediction module is a neural network composed of two-layer fully connected structure; The immediate reward value is compared with the actual reward generated by the feedback of the breaker execution action to calculate the reward estimation loss.

6. The method of claim 1, wherein, The decision-making of the current latent state by the policy optimization module includes: The state combination vector is input into the policy network to output the control action probability distribution; According to the control action probability distribution, the control action is selected by the maximum probability, which includes immediate tripping, delayed tripping and maintaining connection; The immediate tripping corresponds to the emergency shutdown state of the circuit breaker; The delayed tripping corresponds to the circuit shutdown after a delay setting time; The maintaining connection corresponds to maintaining the current electrical connection state unchanged; At the same time, the state combination vector is input into the value function network to calculate the state value estimate, which is used for target guidance in the policy optimization process.

7. The method of claim 1, wherein the method is based on reinforcement learning. The reward value composed of the actual execution result and the feedback state includes: According to the control action generated by the policy optimization module, the corresponding tripping, delayed tripping or maintaining connection operation is performed; After the control action is executed, the feedback information generated by the circuit breaker is collected, including the action response delay time, the tripping state protection trigger flag and the observation vector at the next time; According to the feedback information, a reward function is constructed to evaluate the effectiveness and timeliness of the current control action; The reward function evaluates the effectiveness and timeliness of the circuit breaker protection behavior, including: If the control action successfully completes the tripping under the fault condition, a positive reward is given, and the reward value is 1; If the control action misfires the tripping under the system fault-free condition, a negative penalty is given, and the reward value is 0; If there is a significant delay in the tripping response, a corresponding penalty is given according to the delay degree, and the reward value is in the range of [0, 1], the higher the delay degree, the lower the reward value. The reward value calculated by the reward function, the state combination vector corresponding to the current time step and the control action are fed back to the policy optimization module and the world model submodule to dynamically adjust and optimize the circuit breaker control policy during the training process.

8. The method of claim 1, wherein, The adaptive optimization of the circuit breaker protection strategy comprises: Based on the posterior network and the prior network, an uncertainty measure of the latent state is constructed: ； wherein, represents a measure of uncertainty of the latent state represents a posterior network-generated latent state distribution, represents a prior network-generated latent state distribution, represents the Kullback-Leibler divergence function, which measures the difference between two distributions.​ An uncertainty-fused policy optimization objective function is constructed, and the policy network parameters are updated by minimizing the policy optimization objective function: ； wherein, represents a policy optimization objective function of the policy network, represents a control action probability distribution, describing a probability of taking an action under a latent state , represents an uncertainty adjustment coefficient, represents a state value estimation value of the value function network for a current latent state , represents a parameter of the policy network, represents a mathematical expectation, represents a reward value generated according to the feedback of executing the action by the circuit breaker; A joint loss function is constructed, and the network parameter set in the world model submodule is updated by minimizing the joint loss function, wherein the joint loss function comprises a reward prediction error, an observation reconstruction error and a posterior bias constraint; The optimized policy network and the world model submodule are synchronously transmitted to the model input buffer of the next time step to form a continuous optimization process of the circuit breaker control policy driven by the latent state, and the closed-loop fusion of fault response and policy evolution is realized.

Images