Element reliability parameter correction method and system based on deep reinforcement learning
Through deep reinforcement learning methods, the reliability assessment inverse problem is constructed in the power system, and the problem of inaccurate component reliability parameters is solved, the evaluation accuracy and calculation efficiency are improved, and scientific risk management basis is provided.
Patent Information
- Application Number
- CN202410926600.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-26
AI Technical Summary
In power systems, data input errors, missing data or changes in equipment reliability parameters lead to the inaccurate acquisition of component reliability parameters, affecting the reliability evaluation accuracy and risk management decisions of the power system.
Using a method based on deep reinforcement learning, the inverse problem is evaluated by building the reliability of the power distribution system, and using indicators such as the system's average power supply availability rate, average interrupt frequency index, system average interrupt duration index and average power supply shortage, etc., it is converted into Marl. The Koff decision-making process optimizes the calculation to obtain the optimal component reliability parameter correction strategy.
It improves the accuracy and evaluation accuracy of component reliability parameters, significantly improves computing efficiency, and provides a more scientific basis for power system optimization and risk management.
Smart Images

Figure CN119026448A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of component reliability parameter correction, and in particular to a component reliability parameter correction method and system based on deep reinforcement learning. Background Art
[0002] The reliability of the power system is a key factor in ensuring the continuity and stability of power supply. It involves the ability of the power system to continue to supply power under normal or abnormal conditions. In the design and operation of the power system, the reliability of the system is usually evaluated based on the reliability parameters of its components, which include the failure rate and repair time of the components. The failure rate refers to the frequency of component failures per unit time, while the repair time refers to the time required to restore the component from a faulty state to a normal working state. Due to various reasons such as data input errors, data missing, or changes in equipment reliability parameters, it may not be possible to accurately obtain the reliability parameters of certain components during the reliability evaluation process. These uncertainties will affect the accuracy of the reliability evaluation, and thus affect the optimization and risk management decisions of the entire power system. In order to solve this problem, the inverse problem of reliability evaluation (IPRE) of power system came into being.
[0003] The inverse problem of power system reliability assessment aims to use known system reliability indicators to infer unknown or inaccurate component reliability parameters. This is an important research direction in the field of power system reliability assessment. It not only helps to reveal the true operating status of the existing system, but also can be used to guide system planning and improve the stability and security of power supply. Reliability indicators of distribution systems, such as system average interruption frequency index (SAIFI), system average interruption duration index (SAIDI) and average energy notsupplied (AENS), are usually calculated by power grid companies based on the statistical number of interruptions, interruption duration and load interruption at each load point. By performing inverse reliability assessment on available reliability indicators, unknown or erroneous component reliability parameters can be quantified and corrected. Summary of the invention
[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0005] In view of the above existing problems, the present invention is proposed.
[0006] Therefore, the present invention provides a component reliability parameter correction method and system based on deep reinforcement learning, which can solve the problems mentioned in the background technology.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0008] In a first aspect, the present invention provides a component reliability parameter correction method based on deep reinforcement learning, comprising:
[0009] Constructing an inverse reliability assessment problem of the distribution system according to the system reliability index of the distribution system, wherein the system reliability index includes the system average power supply availability index, the average interruption frequency index, the system average interruption duration index and the average power outage shortage;
[0010] A state space, an action space and a reward function are preset to transform the inverse problem of reliability assessment of the power distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of the system reliability assessment after the parameter adjustment;
[0011] According to the Markov decision process transformed from the inverse problem of distribution system reliability assessment based on the preset deep deterministic policy gradient algorithm, optimization calculation is performed to obtain the optimal component reliability parameter correction strategy.
[0012] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, the inverse problem of reliability assessment of the power distribution system constructed according to the system reliability index of the power distribution system includes:
[0013] The calculation formula of the average power supply availability index is as follows:
[0014]
[0015] Among them, λ i and r i is the failure rate and repair time of component i, N c is the number of components in the power distribution system, N load is the number of load points in the distribution system, N iis the number of users who shed load when the i-th component fails, N j is the number of users connected to the jth load point.
[0016] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, wherein: the inverse problem of reliability assessment of the power distribution system constructed according to the system reliability index of the power distribution system also includes:
[0017] The calculation formula for the average power outage and power shortage is as follows:
[0018]
[0019] Where Pi is the average load at load point i;
[0020] The calculation formula of the system average interruption frequency index is as follows:
[0021]
[0022] The calculation formula for the average power outage frequency index of users is as follows:
[0023]
[0024] Among them, M j is the number of load shedding users connected to load point j.
[0025] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, wherein: the preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process includes:
[0026] The modified reliability parameters are used to form the state space of the agent, that is, s = {λ, μcondition y}, s represents the state space, λ represents the component failure rate, μ represents the component repair rate, represents the reliability index calculated based on parameters λ and μ in year y, and They represent the average power generation and average load demand of each node in year y, which are used as inputs for reliability assessment. y is a factor that is strongly correlated with component reliability parameters in the yth year, including at least meteorological conditions and equipment aging degree. It is only used as the input of the deep neural network and does not change with the change of λ and μ.
[0027] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, wherein: the preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process also includes:
[0028] The action space of the agent is constructed by using the correction value of the reliability parameter, that is, a = {Δλ, Δμ}, Δλ represents the correction value of the component failure rate, Δμ represents the correction value of the component repair rate, Δλ and Δμ are determined by engineering experience, and the corrected reliability parameters should satisfy the following relationship:
[0029]
[0030] Among them, λ UB , LB and μ UB , μ LB are the upper and lower limits of the failure rate and repair rate respectively.
[0031] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, wherein: the preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process also includes:
[0032] The reward function is as follows:
[0033] reward[(s -1 ,a)→s]=(accuracy(s)-accuracy(s -1 )) / ||ss -1 ||
[0034]
[0035] Among them, reward[(s -1 ,a)→s] means the agent is in state s -1 The reward obtained by taking action a to transfer to state s, M is the number of reliability indicators, and R i are the reliability indicators obtained from the i-th evaluation and statistics respectively, and Y represents the year.
[0036] As a preferred solution of the component reliability parameter correction method based on deep reinforcement learning described in the present invention, wherein: the Markov decision process transformed from the inverse problem of distribution system reliability assessment according to the preset deep deterministic policy gradient algorithm is optimized and calculated to obtain the optimal component reliability parameter correction strategy, including:
[0037] The deep deterministic policy gradient algorithm solves the inverse reliability problem of the distribution system. The algorithm includes an Actor policy network and a Critic evaluation network. The Actor policy network is used to explore the environment and output action decisions, and the Critic evaluation network is used to judge the pros and cons of each action, thereby guiding the direction of the Actor policy network gradient update.
[0038] In a second aspect, the present invention provides a component reliability parameter correction system based on deep reinforcement learning, comprising:
[0039] A problem construction module is used to construct an inverse reliability assessment problem of a distribution system according to system reliability indicators of the distribution system, wherein the system reliability indicators include an average power supply availability index of the system, an average interruption frequency index, an average interruption duration index of the system, and an average power outage shortage;
[0040] A conversion module, which is used to preset a state space, an action space and a reward function, and convert the inverse problem of the reliability assessment of the power distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of the component reliability parameters, and the reward function includes the improvement degree of the system reliability assessment after the parameter adjustment;
[0041] The solution correction module is used to optimize the Markov decision process transformed according to the preset deep deterministic policy gradient algorithm distribution system reliability assessment inverse problem to obtain the optimal component reliability parameter correction strategy.
[0042] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when executing the computer program.
[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described above when executed by a processor.
[0044] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a component reliability parameter correction method and system based on deep reinforcement learning, constructs an inverse problem of reliability assessment of a distribution system according to the system reliability index of the distribution system, wherein the system reliability index includes the system average power supply availability index, the average interruption frequency index, the system average interruption duration index, and the average power outage and power shortage; presets a state space, an action space, and a reward function, and transforms the inverse problem of reliability assessment of the distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of the system reliability assessment after the parameter adjustment; optimizes and calculates the Markov decision process transformed from the inverse problem of reliability assessment of the distribution system according to the preset deep deterministic policy gradient algorithm, and obtains the optimal component reliability parameter correction strategy. The proposed method can accurately correct component reliability parameters, improve the accuracy of reliability assessment, and has achieved remarkable results in improving computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:
[0046] Figure 1 A method flow chart of a component reliability parameter correction method and system based on deep reinforcement learning provided by one embodiment of the present invention;
[0047] Figure 2 A schematic diagram of the interaction between a component reliability parameter correction method based on deep reinforcement learning and an intelligent agent of the system and the environment provided by an embodiment of the present invention;
[0048] Figure 3 A schematic diagram of a Markov decision process of a component reliability parameter correction method and system based on deep reinforcement learning provided by an embodiment of the present invention;
[0049] Figure 4 A schematic diagram of an Actor-Critic algorithm of a component reliability parameter correction method and system based on deep reinforcement learning provided by an embodiment of the present invention;
[0050] Figure 5 A reliability inverse problem solving flow chart based on DDPG of a component reliability parameter correction method and system based on deep reinforcement learning provided by an embodiment of the present invention;
[0051] Figure 6A deep reinforcement learning convergence curve diagram of a component reliability parameter correction method and system based on deep reinforcement learning provided by one embodiment of the present invention;
[0052] Figure 7 An internal structure diagram of a computer device of a component reliability parameter correction method and system based on deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0054] Example 1
[0055] Reference Figure 1-Figure 7 , is the first embodiment of the present invention, which provides a component reliability parameter correction method and system based on deep reinforcement learning, including a component reliability parameter correction method based on deep reinforcement learning and a component reliability parameter correction system based on deep reinforcement learning, wherein a component reliability parameter correction method based on deep reinforcement learning includes:
[0056] Before describing the embodiments of the present application in detail, some related concepts are first explained for the sake of clarity.
[0057] State space: refers to the set of all possible system states. In the power system, the state space can include the reliability parameters of each component in the system, the operating status of the system, external environmental conditions, etc. The setting of the state space determines the range of perception and response of the intelligent agent, which is the basis for the intelligent agent's decision-making.
[0058] Action space: refers to the set of all possible actions that an agent can perform. In the power system reliability parameter correction problem, the action space may include fine-tuning the component failure rate or repair time parameters. The agent attempts to improve the system reliability by selecting specific actions in the action space.
[0059] Reward function: defines the immediate feedback obtained by the agent after performing a specific action in a specific state, which guides the agent to learn which behaviors are beneficial. In the context of power system component reliability parameter correction, the reward function may be designed based on the degree of improvement in system reliability assessment, encouraging the agent to adopt parameter correction strategies that can improve system reliability.
[0060] Markov decision process: is a mathematical framework used to describe how agents make decisions in uncertain environments. In MDP, the agent is in a series of states, it can perform actions to change the state, and evaluate the effect of the action based on the state transition probability and reward function. MDP assumes that the current state depends only on the previous state and the current action, but not on the past history, that is, it satisfies the Markov property. In the problem of power system reliability parameter correction, MDP is used to model how the agent optimizes the system reliability parameters through a series of actions in order to maximize the accuracy of system reliability assessment.
[0061] In related technologies, when the number of reliability parameters to be solved is greater than the number of known reliability assessment indicators (the number of variables is greater than the number of equations), existing methods often solve the inverse reliability assessment problem by constructing an optimization problem. This process is subjective and may lead to inaccurate results. Existing studies often use heuristic algorithms to solve the inverse reliability assessment problem, which takes a long time to solve and cannot meet the actual needs of engineering.
[0062] The present application provides a method that can effectively solve the above-mentioned problems. Next, multiple embodiments will be combined to explain in detail how to implement the component reliability parameter correction method based on deep reinforcement learning;
[0063] Figure 1 A method flow chart of a component reliability parameter correction method and system based on deep reinforcement learning is shown, including:
[0064] S101, constructing an inverse reliability assessment problem of the distribution system according to system reliability indicators of the distribution system, wherein the system reliability indicators include a system average power supply availability index, an average interruption frequency index, a system average interruption duration index, and an average power outage shortage;
[0065] Among them, the inverse problem of reliability assessment of the distribution system is constructed according to the system reliability index of the distribution system, including:
[0066] Specifically, the calculation formula of the average power supply availability index is as follows:
[0067]
[0068] Among them, λ i and r i is the failure rate and repair time of component i, N c is the number of components in the power distribution system, N load is the number of load points in the distribution system, N i is the number of users who shed load when the i-th component fails, N j is the number of users connected to the jth load point.
[0069] In an optional embodiment, substituting the known CRP value, we can obtain:
[0070]
[0071] Among them, a 1,i and b1 are constants simplified from the average power supply availability index.
[0072] Furthermore, the inverse problem of reliability assessment of the distribution system constructed according to the system reliability index of the distribution system also includes:
[0073] The calculation formula for the average power outage and power shortage is as follows:
[0074]
[0075] Where Pi is the average load at load point i;
[0076] The calculation formula of the system average interruption frequency index is as follows:
[0077]
[0078] The calculation formula for the average power outage frequency index of users is as follows:
[0079]
[0080] Among them, M j is the number of load shedding users connected to load point j.
[0081] It should be noted that by using the system reliability indicators of the distribution system, such as the average power supply availability, the average interruption frequency index, the system average interruption duration index, and the average power outage power shortage, an inverse reliability assessment problem is constructed. This method combines the known reliability data with the operating characteristics of the distribution system, provides a theoretical basis for subsequent parameter correction, and ensures the objectivity and accuracy of the problem.
[0082] S102, presetting a state space, an action space and a reward function, converting the inverse problem of reliability assessment of the power distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes measuring the degree of improvement of the system reliability assessment after the parameter adjustment;
[0083] It should be noted that the subject of reinforcement learning is an agent, which is responsible for taking actions and making decisions. The object that interacts with the agent is called the environment, which can be abstractly understood as the rules or mechanisms in the interaction process.
[0084] The mathematical foundation and modeling tool of reinforcement learning is Markov decision process (MDP). An MDP consists of a state space Action Space It consists of state transfer function P(s'|s,a), reward function r(s,a), discount factor γ∈[0,1), etc.
[0085] The state refers to the summary of the current environment. The state space refers to the set of all possible states, denoted as The state space can be discrete or continuous. The state space can be a finite set or an infinite countable set.
[0086] An action is a decision made by an agent based on its current state. The selection of an action can be deterministic or random. The action space is the set of all possible actions, denoted by
[0087] Reward refers to a value that the environment returns to the agent after the agent performs an action. Rewards often need to be designed by yourself, and the quality of reward design is crucial to the effectiveness of reinforcement learning. Reward is a function of the current state s and the current action a, denoted as r(s,a).
[0088] Reinforcement learning usually assumes that state transitions are random and the randomness comes from the environment. The state transition probability function is used to quantify the randomness of state transitions.
[0089] The value of this formula represents the probability that when the agent performs action a in the current state s, the state of the environment becomes s′.
[0090] Usually, the state transition has the Markov property, which can be expressed as This formula indicates that the state S at the next moment t+1 Depends only on the current state S t and action A t , and does not depend on past states and actions.
[0091] The return is the sum of all rewards from the current moment to the end of this round, so the return is also called cumulative future reward. Let the return at time t be the random variable U t . Define the return as:
[0092] Ut =R t +R t+1 +R t+2 +R t+3 +…+R n
[0093] Among them, R t is the reward value at time t, R t+1 is the reward value at time t+1, and so on.
[0094] The goal of reinforcement learning is to find a strategy that maximizes the expected return. This strategy is called the optimal policy.
[0095] In MDP, a discounted return is often used to discount future rewards. The definition of discounted return is:
[0096] U t =R t +γ·R t+1 +γ 2 ·R t+2 +γ 3 ·R t+3 +…
[0097] Among them, γ∈[0,1) refers to the discount rate, which discounts future rewards.
[0098] Specifically, the inverse problem of distribution system reliability is transformed into a Markov decision process, which includes defining agents, environments, states, actions, and rewards, and the distribution system fault recovery problem can be transformed into a Markov decision process. Among them, the agent is the corrector of CRP, and the environment is the evaluator of distribution system reliability.
[0099] In the embodiment of the present application, the state space, action space and reward function are preset, and the inverse problem of distribution system reliability assessment is converted into a Markov decision process, including:
[0100] The modified reliability parameters are used to form the state space of the agent, that is, s = {λ, μcondition y}, s represents the state space, λ represents the component failure rate, μ represents the component repair rate, represents the reliability index calculated based on parameters λ and μ in year y, and They represent the average power generation and average load demand of each node in year y, which are used as inputs for reliability assessment. yis a factor that is strongly correlated with component reliability parameters in the yth year, including at least meteorological conditions and equipment aging degree. It is only used as the input of the deep neural network and does not change with the change of λ and μ.
[0101] In the embodiment of the present application, the state space, action space and reward function are preset, and the inverse problem of reliability assessment of the power distribution system is converted into a Markov decision process, which further includes:
[0102] The action space of the agent is constructed by using the correction value of the reliability parameter, that is, a = {Δλ, Δμ}, Δλ represents the correction value of the component failure rate, Δμ represents the correction value of the component repair rate, Δλ and Δμ are determined by engineering experience, and the corrected reliability parameters should satisfy the following relationship:
[0103]
[0104] Among them, λ UB , LB and μ UB , μ LB are the upper and lower limits of the failure rate and repair rate respectively.
[0105] In the embodiment of the present application, the state space, action space and reward function are preset, and the inverse problem of reliability assessment of the power distribution system is converted into a Markov decision process, which further includes:
[0106] The reward function is as follows:
[0107] reward[(s -1 ,a)→s]=(accuracy(s)-accuracy(s -1 )) / ||ss -1 ||
[0108]
[0109] Among them, reward[(s -1 ,a)→s] means the agent is in state s -1 The reward obtained by taking action a to transfer to state s, M is the number of reliability indicators, and R i are the reliability indicators obtained from the i-th evaluation and statistics respectively, and Y represents the year.
[0110] It should be noted that the objective function aims to ensure the efficiency of the agent in correcting the reliability parameters, that is, to achieve the greatest possible improvement in the reliability assessment accuracy with the smallest possible reliability parameter correction step.
[0111] It should be noted that the distribution system fault recovery problem is modeled as a Markov decision process, which makes the problem easier to solve by defining states, actions, and reward mechanisms. The definition of the state space takes into account the failure rate, repair rate, and various influencing factors of the components, and fully reflects the operating status of the distribution system. The action space is limited to the correction amount of these parameters, ensuring the rationality and feasibility of parameter adjustment.
[0112] The design of the reward function is key, as it directly reflects the effectiveness of the agent's decision. In this case, the reward is not only related to the improvement of the reliability index, but also takes into account the size of the correction step, aiming to balance the magnitude of the improvement and the magnitude of the adjustment to optimize the overall recovery strategy. This design encourages the agent to find a more sophisticated and efficient fault recovery strategy while avoiding the instability caused by over-adjustment.
[0113] In addition, by setting the upper and lower limits of the failure rate and repair rate, we can prevent the parameter adjustment from exceeding the allowable range of the actual project, ensuring the physical meaning of the correction and the stability of the system. This conversion method transforms the originally complex inverse problem of distribution system reliability assessment into an optimizable mathematical problem, providing a clear path for solving the problem.
[0114] S103, performing optimization calculations according to the Markov decision process transformed from the inverse problem of distribution system reliability assessment using a preset deep deterministic policy gradient algorithm to obtain an optimal component reliability parameter correction strategy.
[0115] Among them, the Markov decision process transformed from the inverse problem of distribution system reliability assessment according to the preset deep deterministic policy gradient algorithm is optimized and calculated to obtain the optimal component reliability parameter correction strategy, including: the deep deterministic policy gradient algorithm solves the inverse problem of distribution system reliability. The algorithm contains an Actor policy network and a Critic evaluation network. The Actor policy network is used to explore the environment and output action decisions. The Critic evaluation network is used to judge the pros and cons of each action, thereby guiding the direction of the Actor policy network gradient update.
[0116] It should be noted that deep reinforcement learning refers to the addition of deep neural networks on the basis of reinforcement learning. The present invention adopts the deep deterministic policy gradient algorithm (DDPG) algorithm to solve the inverse reliability problem of the distribution system. The algorithm combines value learning with policy learning and is suitable for solving reinforcement learning problems with continuous action space. It includes an Actor policy network and a Critic evaluation network. The Actor policy network is used to explore the environment and output action decisions, and the Critic evaluation network is used to judge the pros and cons of each action, thereby guiding the direction of the Actor policy network gradient update.
[0117] It should be noted that value learning and strategy learning are two types of thinking in reinforcement learning. The idea of value learning can be simply understood as obtaining a "prophet" who can predict the future through training. This prophet must be able to foresee the possible results of taking all actions and choose actions based on the foreseen results. In reinforcement learning, the prophet is the action-value function, also known as the Q function, as shown in formula (18).
[0118]
[0119] Among them, the action value function Q π (s t ,a t ) is the value of the current state s t Next, we select action a based on strategy π t Return U t expected value.
[0120] It should be noted that the action-value function is generally represented by a neural network. At the beginning, the parameters of the network are random, which means that the prophet knows nothing. However, as the intelligent agent interacts with the environment and accumulates enough experience, it continuously optimizes the action-value function and eventually obtains the optimal action-value function.
[0121]
[0122] Among them, the optimal action value function The value of the action value function Qπ(s t ,a t ) is the maximum value of .
[0123] It should be noted that in this process, the "wisdom" of the intelligent agent is not ideal at the beginning, but through training, it can eventually reach the level of a "prophet".
[0124] Ultimately, the agent selects an action based on the optimal action-value function, and its strategy is to select an action with the largest output result in the action-value function.
[0125] It should be noted that the value learning method mainly learns the action-value function and then derives a strategy based on the action-value function. There is no explicit strategy in the learning process; while the policy-based method directly and explicitly learns a target strategy.
[0126]
[0127] It should be noted that the input of the policy function π is state s and action a, and the output is a probability value between 0 and 1.
[0128] Whenever a state s is observed, the policy function is used to calculate the probability of each action, and then random sampling is performed to obtain an action a for the agent to execute. In this way, the agent can be controlled by the policy function. The policy function can also be represented by a neural network as π(a|s;θ). At the beginning, θ is randomly initialized, and then the collected states, actions, and rewards are used to update θ. Let the strategy become better and better.
[0129] It should be noted that a state-value function is needed to evaluate whether a strategy is good or bad.
[0130]
[0131] Among them, V π (s t ) represents the state value function, Q π (s t ,a t ) represents the action-value function, Indicates action a t The state value function is obtained by the action value function Q π (s t ,a t )About a t Find the expectation to quantify the current state s t the good or bad.
[0132]
[0133] It should be noted that the state value function is expected with respect to the state S, and the J(θ) function is obtained. The better the strategy π is (that is, the better the policy network parameter θ is), the larger J(θ) is. In this way, J(θ) can be used as the objective function of policy learning. By updating the policy network parameter θ, the objective function J(θ) becomes larger and larger, which means that the policy network is getting better and better.
[0134] It should be noted that the policy network π(a|s; θ) is equivalent to the actor, which makes action a based on state s. The value network q(s,a; ω) is equivalent to the critic, which scores the actor's performance and evaluates the quality of action a under state s. The relationship between the policy network (actor) and the value network (critic) is as follows: Figure 4 shown.
[0135] It should be noted that the training policy network (actor) is: after the actor makes an action a, the critic will give a score The score is fed back to the actor to help him make improvements. The actor uses the current state s, his own action a, and the judges' scores. By calculating the gradient of parameter θ using equation (14), the parameter θ is updated (equivalent to changing one’s own performance technique). In this way, the actor’s performance is increasingly praised by the judges, so the actor’s score is Getting higher and higher.
[0136] It should be noted that when training the value network (critic): Initially, the parameter ω of the value network is random, which means that the critic's score is completely random. We can observe a reward r from the environment, regard r as the true value, use r to calibrate the judge's score, and update the parameter ω.
[0137] It should be noted that, in addition, the Actor network and the Critic network have two very important modules respectively: experience replay and target network, which can help DDPG achieve stable and excellent performance.
[0138] In the embodiment of the present application, the process of the DDPG algorithm is as follows:
[0139] (1) Using random network parameters θ Q and θ μ Initialize the Critic network and the Actor network respectively, and use the parameter θ Q′ ←θ Q and θ μ′ ←θ μ Initialize its corresponding target network;
[0140] (2) Initialize the experience replay pool, set e = 1 (in the DDPG algorithm decision process, perform action a in the environment t , get the return reward value and the new observation state, and store the data in the experience pool. When the data in the experience pool reaches a certain size, N samples are randomly selected from it for updating the network parameters);
[0141] (3) Extract the initial state of the environment s1 and set t = 0;
[0142] (4) Let t = t + 1, and select action a according to the current state t =μ(s t |θ μ )+N t , where a t is the current observation state, N t To explore noise;
[0143] (5) Execute action a t , get reward r t , the environment state becomes s t+1 ;
[0144] (6)Tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience pool;
[0145] (7) Extract N tuples from R {(s i ,a i ,r i ,s i+1 )} i=1,2,...,N ;
[0146] (8) For each tuple, calculate y i =r i +γQ′(s i+1 ,μ′(s i+1 )θ Q′ )(γ is the discount rate), and the Critic online network is updated by minimizing the TD deviation, that is,
[0147] (9) Calculate policy gradient (Chain derivation rule, first derive the action, then derive the parameter), update the Actor online network;
[0148] (10) Update the target network
[0149]
[0150] Where τ is the soft update constant.
[0151] (11) If t = T, set e = e + 1 and go to step (12); otherwise, go to step (4);
[0152] (12) If e=E, the algorithm ends, otherwise go to step (3).
[0153] It should be noted that episode is a term that simulates a complete decision-making process or game round. In the context of reinforcement learning or deep reinforcement learning, an episode usually starts from the initial state and eventually reaches the terminal state through the continuous decision-making of the agent and the feedback of the environment. Each episode generates a series of data, which is crucial for training the policy network and the value network.
[0154] In the DDPG algorithm of the embodiment of the present application, episode is used to define a complete training cycle of the algorithm. Specifically, `t=T` in step (11) means that the current episode has completed T time steps or decision steps, and the algorithm will perform some additional operations, such as adding exploration noise e (`e=e+1` in step (12)), and then prepare to start the next episode (`Go to step (3)` in step (12)).
[0155] It is worth noting that exploration noise e plays an important role in reinforcement learning. Due to the uncertainty of the environment, the agent may not be able to find the optimal decision directly. Therefore, in the early stage of training, the agent will add a certain amount of randomness (i.e., exploration noise) to select actions in order to better explore the environment and find potential better strategies. As the training progresses, the exploration noise will gradually decrease, and the agent will rely more and more on the policy network to make decisions.
[0156] In addition, `e=E` in step (12) means that the algorithm has completed E episodes of training. When this condition is met, the algorithm will end the training and output the trained policy network and value network. These two networks can be used in actual decision-making tasks to output the optimal action based on the current state to achieve the maximum cumulative reward.
[0157] In an optional embodiment, the following steps may also be present after step (12):
[0158] (13) At the end of each episode, the target networks of the Critic network and the Actor network are updated at a certain frequency based on the samples in the experience replay pool to keep the target network synchronized with the online network, while introducing a certain degree of stability to prevent excessive fluctuations.
[0159] (14) When updating the target network, a soft update strategy is adopted, that is, the parameters of the target network are a linear combination of the online network parameters. By adjusting the soft update constant τ, the update step size between the target network and the online network can be controlled to balance the relationship between exploration and utilization.
[0160] (15) When calculating policy gradients, the experience replay pool plays an important role. It can break the strong correlation of the environment sequence, making the training samples more independent and diverse, thereby improving the generalization ability of the network.
[0161] (16) At the same time, by continuously storing new observations (states, actions, rewards) into the experience replay pool, the continuous updating of training data is ensured, preventing the network from falling into local optimality too early.
[0162] (17) When the number of samples in the experience replay pool reaches the preset maximum value E, it means that an episode ends. At this time, the experience pool contains rich environmental interaction information and can provide sufficient training data for network parameter optimization.
[0163] (18) In each episode, the algorithm continuously iterates the Actor network and the Critic network, and updates the target network in a timely manner, so that the policy network can gradually learn a better strategy, and the value network can more accurately evaluate the value of actions in each state. The two promote each other and jointly improve the decision-making performance of the intelligent agent in the environment.
[0164] (19) When the preset number of episodes is reached or other stopping conditions are met, the training process of the DDPG algorithm ends. At this time, the policy network and value network have the ability to make efficient and stable decisions in unknown environments.
[0165] It should be noted that after the training is completed, the optimal component reliability parameter correction strategy is obtained.
[0166] It should be noted that the concept of the target network is introduced, which is equivalent to introducing a stable reference point in the training process. The parameters of the target network change slowly, providing a stable prediction target for the policy network and the value network, reducing the instability in the training process. This "progressive update" method helps to smooth the learning curve and improve the convergence speed and final performance of the algorithm.
[0167] In addition, the role of the experience replay pool cannot be ignored. It stores historical data of environmental interactions and learns by randomly extracting samples, effectively breaking the correlation of data sequences and increasing the diversity and generalization of training samples. This helps prevent the model from overfitting to a specific environmental sequence, allowing the DDPG algorithm to show good adaptability and stability in various complex environments.
[0168] Therefore, after sufficient training, the Actor network will learn a strategy that outputs the optimal action in a given state, that is, the optimal component reliability parameter correction strategy. The DDPG algorithm combines the interactive learning of the policy network and the value network, the stable guidance of the target network, and the sample diversity of experience playback to build an efficient, stable and adaptable deep reinforcement learning framework.
[0169] In the embodiment of the present application, the method of the present application first constructs an inverse reliability assessment problem based on the system reliability indicators of the power distribution system. These indicators include the system average power supply availability, the average interruption frequency index, the system average interruption duration index, and the average power outage power shortage. Through mathematical models, these indicators are closely related to the failure rate and repair time of the components, reflecting the reliability and stability of the system as a whole.
[0170] Subsequently, the method transforms the inverse reliability assessment problem into a Markov decision process (MDP). In this process, the state space consists of the current state of the distribution system and the reliability parameters of the components, including failure rate and repair time. The action space refers to the possibility of adjusting the reliability parameters of the components, such as increasing or decreasing the failure rate. The reward function is designed to measure the degree of improvement in the system reliability assessment after the parameter adjustment, and motivate the agent to learn in the direction of improving the system reliability.
[0171] In the state space, the reliability index calculated based on the revised reliability parameters is included, as well as factors closely related to the component reliability parameters, such as meteorological conditions and equipment aging. The action space is composed of the corrections of the component failure rate and repair rate to ensure that the adjusted parameters are within a reasonable range and meet the actual engineering experience and constraints.
[0172] A properly designed reward function can accurately reflect the benefits of an agent after taking an action in a specific state. It is calculated based on the improvement of the reliability index. By comparing the reliability index obtained through evaluation and statistics, the reward function guides the agent to make the best decision and improve the overall reliability of the system.
[0173] Finally, the deep deterministic policy gradient (DDPG) algorithm is used to solve the transformed Markov decision process to obtain the optimal component reliability parameter correction strategy. The DDPG algorithm uses the Actor-Critic framework to allow the Actor policy network to explore the environment and make decisions, while the Critic evaluation network evaluates the quality of these decisions, jointly promoting the optimization of the Actor policy network, thereby minimizing the reliability parameter correction step size while maximizing the improvement of reliability evaluation accuracy.
[0174] In summary, this method can quickly correct component reliability parameters online through a deep reinforcement learning framework, improve the accuracy of system reliability assessment, and ensure the rationality and effectiveness of the correction strategy. Through continuous learning and adaptation, the intelligent agent can gradually master the optimal component reliability parameter correction strategy, effectively deal with the uncertainty of component reliability parameters in the power system, and provide a scientific basis for the optimization and risk management of the power system.
[0175] In summary, the present invention proposes a component reliability parameter correction method based on deep reinforcement learning, constructs an inverse problem of reliability assessment of a distribution system according to the system reliability index of the distribution system, wherein the system reliability index includes the system average power supply availability index, the average interruption frequency index, the system average interruption duration index, and the average power outage and power shortage; presets a state space, an action space, and a reward function, and transforms the inverse problem of reliability assessment of the distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of the system reliability assessment after the parameter adjustment; optimizes and calculates the Markov decision process transformed from the inverse problem of reliability assessment of the distribution system according to the preset deep deterministic policy gradient algorithm, and obtains the optimal component reliability parameter correction strategy. The proposed method can accurately correct component reliability parameters, improve the accuracy of reliability assessment, and has achieved remarkable results in improving computational efficiency.
[0176] This embodiment also provides a component reliability parameter correction system based on deep reinforcement learning, including:
[0177] A problem construction module is used to construct an inverse reliability assessment problem of a distribution system according to system reliability indicators of the distribution system. The system reliability indicators include an average power supply availability index, an average interruption frequency index, an average interruption duration index, and an average power outage shortage;
[0178] A conversion module is used to preset the state space, action space and reward function, and convert the inverse problem of distribution system reliability assessment into a Markov decision process. The state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the improvement degree of system reliability assessment after measuring the parameter adjustment;
[0179] The solution correction module is used to optimize the Markov decision process transformed according to the preset deep deterministic policy gradient algorithm distribution system reliability assessment inverse problem to obtain the optimal component reliability parameter correction strategy.
[0180] The above-mentioned unit modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above-mentioned modules.
[0181] This embodiment also provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a component reliability parameter correction method based on deep reinforcement learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0182] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:
[0183] The inverse reliability assessment problem of the distribution system is constructed according to the system reliability index of the distribution system. The system reliability index includes the system average power supply availability index, the average interruption frequency index, the system average interruption duration index and the average power outage shortage.
[0184] The state space, action space and reward function are preset to transform the inverse problem of distribution system reliability assessment into a Markov decision process. The state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of system reliability assessment after measuring the parameter adjustment.
[0185] According to the Markov decision process transformed from the inverse problem of distribution system reliability assessment based on the preset deep deterministic policy gradient algorithm, optimization calculation is performed to obtain the optimal component reliability parameter correction strategy.
[0186] Example 2
[0187] Reference Figure 2-Figure 6 , which is an embodiment of the present invention, provides a component reliability parameter correction method and system based on deep reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.
[0188] The example in this invention takes some CRP component reliability parameters as the parameters to be determined to verify the effectiveness of the model algorithm proposed in this invention. T1 , L7 , L23 , respectively representing the failure rate of the transformer connected to node 1, the failure rate of branch 7, and the failure rate of branch 23. Referring to the test system data of relevant literature and combining with actual engineering experience, the value interval of the CRP to be determined is set to 0.5-2 times the true value. The present invention uses the statistical value of the reliability index in the past two years as the true value of the reliability index (the variable is greater than the number of indicators, and there are infinite solutions).
[0189] The convergence curve of the method proposed in the present invention is as follows: Figure 6 The solution results of the algorithm proposed in the present invention are shown in Table 1.
[0190] Table 1: Solution results of the proposed algorithm.
[0191]
[0192] As can be seen from Table 1, the algorithm proposed in the present invention can obtain a better reliability parameter correction strategy in a very short time. The accuracy and calculation time of the proposed algorithm are better than those of the heuristic algorithm, among which the accuracy is improved by 1.3%, and the calculation time is reduced by nearly 3700 times (excluding offline training time) (Note: the trained model can be used online and continuously updated to correct the reliability parameters obtained by annual statistics). Although the reliability assessment accuracy of the heuristic algorithm is as high as 98.2%, its corrected reliability parameters obviously deviate from the statistical values and true values, and the correction strategy is unreasonable, which is only one of the infinite quasi-optimal solutions. However, the method proposed in the present invention can make the corrected reliability parameters as close to the statistical values and true values as possible while ensuring the accuracy of the reliability assessment, and the correction strategy is more reasonable.
[0193] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
[0194] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0195] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0198] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0199] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A component reliability parameter correction method based on deep reinforcement learning, characterized in that: include: Constructing an inverse reliability assessment problem of the distribution system according to the system reliability index of the distribution system, wherein the system reliability index includes the system average power supply availability index, the average interruption frequency index, the system average interruption duration index and the average power outage shortage; A state space, an action space and a reward function are preset to transform the inverse problem of reliability assessment of the power distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of the system reliability assessment after the parameter adjustment; According to the Markov decision process transformed from the inverse problem of distribution system reliability assessment based on the preset deep deterministic policy gradient algorithm, optimization calculation is performed to obtain the optimal component reliability parameter correction strategy.
2. The component reliability parameter correction method based on deep reinforcement learning according to claim 1, characterized in that: The inverse problem of reliability assessment of the power distribution system constructed according to the system reliability index of the power distribution system includes: The calculation formula of the average power supply availability index is as follows: Among them, λ i and r i is the failure rate and repair time of component i, N c is the number of components in the power distribution system, N load is the number of load points in the distribution system, N i is the number of users who shed load when the i-th component fails, N j is the number of users connected to the jth load point.
3. The component reliability parameter correction method based on deep reinforcement learning according to claim 2, characterized in that: The inverse problem of reliability assessment of the power distribution system constructed according to the system reliability index of the power distribution system also includes: The calculation formula for the average power outage and power shortage is as follows: Where Pi is the average load at load point i; The calculation formula of the system average interruption frequency index is as follows: The calculation formula for the average power outage frequency index of users is as follows: Among them, M j is the number of load shedding users connected to load point j.
4. The component reliability parameter correction method based on deep reinforcement learning according to claim 3 is characterized in that: The preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process includes: The modified reliability parameters are used to form the state space of the agent, that is, s = {λ, μcondition y }, s represents the state space, λ represents the component failure rate, μ represents the component repair rate, represents the reliability index calculated based on parameters λ and μ in year y, and They represent the average power generation and average load demand of each node in year y, which are used as inputs for reliability assessment. y is a factor that is strongly correlated with component reliability parameters in the yth year, including at least meteorological conditions and equipment aging degree. It is only used as the input of the deep neural network and does not change with the change of λ and μ.
5. The component reliability parameter correction method based on deep reinforcement learning according to claim 4 is characterized in that: The preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process also includes: The action space of the agent is constructed by using the correction value of the reliability parameter, that is, a = {Δλ, Δμ}, Δλ represents the correction value of the component failure rate, Δμ represents the correction value of the component repair rate, Δλ and Δμ are determined by engineering experience, and the corrected reliability parameters should satisfy the following relationship: Among them, λ UB , LB and μ UB , μ LB are the upper and lower limits of the failure rate and repair rate respectively.
6. The component reliability parameter correction method based on deep reinforcement learning according to claim 5, characterized in that: The preset state space, action space and reward function, converting the inverse problem of distribution system reliability assessment into a Markov decision process also includes: The reward function is as follows: reward[(s -1 ,a)→s]=(accuracy(s)-accuracy(s -1 )) / ||s-s -1 || Among them, reward[(s -1 ,a)→s] means the agent is in state s -1 The reward obtained by taking action a to transfer to state s, M is the number of reliability indicators, and R i are the reliability indicators obtained from the i-th evaluation and statistics respectively, and Y represents the year.
7. The component reliability parameter correction method based on deep reinforcement learning according to claim 6, characterized in that: The Markov decision process transformed from the inverse problem of power distribution system reliability assessment according to the preset deep deterministic policy gradient algorithm is optimized and calculated to obtain the optimal component reliability parameter correction strategy, which includes: The deep deterministic policy gradient algorithm solves the inverse reliability problem of the distribution system. The algorithm includes an Actor policy network and a Critic evaluation network. The Actor policy network is used to explore the environment and output action decisions, and the Critic evaluation network is used to judge the pros and cons of each action, thereby guiding the direction of the Actor policy network gradient update.
8. A component reliability parameter correction system based on deep reinforcement learning, characterized in that: include: A problem construction module is used to construct an inverse reliability assessment problem of a distribution system according to system reliability indicators of the distribution system, wherein the system reliability indicators include an average power supply availability index of the system, an average interruption frequency index, an average interruption duration index of the system, and an average power outage shortage; A conversion module, used for presetting a state space, an action space and a reward function, and converting the inverse problem of reliability assessment of the power distribution system into a Markov decision process, wherein the state space includes the current system state, the action space includes the adjustment of component reliability parameters, and the reward function includes the degree of improvement of the system reliability assessment after the parameter adjustment; The solution correction module is used to optimize the Markov decision process transformed according to the preset deep deterministic policy gradient algorithm distribution system reliability assessment inverse problem to obtain the optimal component reliability parameter correction strategy.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Transformer state prediction and adaptive control method based on digital twinning
CN119596704A
