Distribution network defect management method and device based on depth deterministic strategy gradient

Abstract

The invention relates to the technical field of electric power, and provides a distribution network defect management method and device based on a depth deterministic strategy gradient, and the method comprises the steps: building a power grid environment database which comprises a defect record database, a weather state database, a line database and a human resource database; training a DDPG model through interactive learning of an intelligent agent and the power grid environment database; noise in the DDPG model is optimized through a quasi-random fractal search algorithm, and a QRFS-DDPG model is obtained; and inputting the obtained power grid environment information into the QRFS-DDPG model, and determining the defect elimination plan arrangement of the distribution network defects. According to the invention, the depth deterministic strategy gradient algorithm optimized by the quasi-random fractal search algorithm can be used to realize automatic identification and classification of power grid defects and intelligent scheduling of defect processing, the efficiency and quality of defect processing are improved, stable and safe operation of the power grid is ensured, and the operation efficiency and safety of the power grid are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of electric power technology, and in particular to a distribution network defect management method and device based on deep deterministic policy gradient. Background Art

[0002] In the stable operation of the power system, grid defect management plays a vital role. However, the current defect management methods face many challenges, mainly reflected in the following aspects: (1) The existing grid defect management relies heavily on manual inspection and recording, which is not only inefficient, but also limited by the number of personnel and the subjective experience level of personnel, making it difficult to ensure the timeliness and accuracy of defect handling. (2) If new defects cannot be included in the schedule in time, it will affect the operating status of the equipment and may even cause power safety accidents, posing a threat to the stability and safety of the power grid. (3) The shortage of construction forces and the unreasonable scheduling of fault elimination will make it difficult to effectively carry out the fault elimination progress, especially when it is necessary to cooperate with the power outage plan for fault elimination operations, the lack of resource allocation is more prominent. (4) The imperfections of manual inspection and data collection lead to data omissions, and traditional methods are difficult to cope with the rapid processing and analysis of a large amount of defect data, which greatly increases the complexity and uncertainty of grid defect management; (5) The impact of severe weather conditions on the construction progress also greatly increases the complexity and uncertainty of grid defect management. Summary of the invention

[0003] The purpose of the embodiments of the present disclosure is to provide a distribution network defect management method and device based on deep deterministic policy gradient, so as to at least solve the technical problems in the prior art that the complexity and uncertainty of power grid defect management are large, which affects the stability and safe operation of the power grid.

[0004] In order to solve the above technical problems, the embodiments of the present disclosure adopt the following technical solutions:

[0005] In a first aspect, an embodiment of the present disclosure provides a distribution network defect management method based on an improved deep deterministic strategy, including:

[0006] Establishing a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resources database;

[0007] Training the DDPG model through interactive learning between the intelligent agent and the power grid environment database;

[0008] The noise in the DDPG model is optimized by using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model;

[0009] The acquired power grid environment information is input into the QRFS-DDPG model to determine the defect elimination plan for the distribution network.

[0010] In some embodiments, the DDPG model is trained through interactive learning between the agent and the power grid environment database, including:

[0011] Constructing a policy network and a value network, wherein the policy network outputs a dispatch work order decision according to the current power grid state, and the value network evaluates the expected return of the dispatch work order decision;

[0012] The policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy.

[0013] In some embodiments, constructing a policy network and a value network includes:

[0014] Determine the policy function and value function , where the strategy function , which means that in a given state s, according to the parameter Determine the strategy distribution, output action a, the value function Predict that in state s, action a is performed and policy is followed The expected return of the agent is s, the state s is the power grid environment where the agent is located, and the action a is the defect work order sent by the agent. The defect work order includes the allocation to the preset defect team and the expected time for defect elimination. represents the policy network parameters, represents the value network parameters;

[0015] Determine the objective function , where the objective function is defined as starting from the current state s and following the strategy The sum of the discounted returns obtained is expressed as:

[0016] ,

[0017] In the formula, is the discount factor for the time step, is the immediate reward obtained at time t;

[0018] The policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy, including:

[0019] Updating the policy network parameters through policy gradient to optimize the objective function and learn the optimal policy;

[0020] The value network parameters are updated by minimizing the prediction error of the Q value to learn the value of the state-action pair and obtain the optimal value.

[0021] In some embodiments, updating the policy network parameters by policy gradient includes:

[0022] The policy network parameters are optimized by gradient ascent, expressed as:

[0023] ,

[0024] in, It is the action output by the policy network according to the current state s.

[0025] In some embodiments, updating the value network parameters by minimizing the prediction error of the Q value includes:

[0026] The loss function for determining the update of the value network is:

[0027] ,

[0028] Wherein, y is the target Q value, and the calculation formula of the target Q value is:

[0029] ,

[0030] In the formula, It’s an instant reward. is the discount factor and Q' is the output of the target value network.

[0031] In some embodiments, the method further comprises:

[0032] Acquire distribution network experience data, wherein the distribution network experience data includes at least one of a defect status, an execution status of a work order, and a difference between an actual defect elimination time and a planned defect elimination time;

[0033] Updating strategy estimates and value estimates based on the distribution network experience data;

[0034] The DDPG model is trained using the updated policy estimate and value estimate.

[0035] In some embodiments, the acquired power grid environment information is input into the QRFS-DDPG model to determine a defect elimination plan for the distribution network defect, including:

[0036] Using the QRFS-DDPG model, scoring the importance of eliminating distribution network defects in the power grid environment information to obtain a scoring result;

[0037] The processing of the distribution network defects is scheduled and a work order is dispatched according to the scoring results.

[0038] In some embodiments, the noise in the DDPG model is optimized using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model, including:

[0039] Iteratively search for the optimal noise under the preset power grid environment by gradually narrowing the search space;

[0040] The optimal noise is added to the action output by the policy network, expressed as:

[0041] ,

[0042] in, is the action chosen at time step t, is the action output by the Actor network, is the noise term extracted from the Ornstein-Uhlenbeck process at each step.

[0043] In some embodiments, before determining the policy function and the value function, the method further includes:

[0044] Initialize the policy network parameters, value network parameters, and corresponding target network parameters through the agent.

[0045] In a second aspect, the embodiment of the present disclosure provides a distribution network defect management device based on an improved deep deterministic strategy, including:

[0046] A database establishment module, configured to establish a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resources database;

[0047] A model training module, configured to train the DDPG model through interactive learning between the intelligent agent and the power grid environment database;

[0048] A model optimization module is configured to optimize the noise in the DDPG model using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model;

[0049] The defect management module is configured to input the acquired power grid environment information into the QRFS-DDPG model to determine the defect elimination plan for the distribution network.

[0050] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned distribution network defect management method based on the improved deep deterministic strategy when executing the computer program on the memory.

[0051] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned distribution network defect management method based on the improved deep deterministic strategy is implemented.

[0052] The present disclosure provides a distribution network defect management method and device based on deep deterministic policy gradient. The method and device establish a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resource database; a DDPG model is trained through interactive learning between an intelligent agent and the power grid environment database; the noise in the DDPG model is optimized using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model; the acquired power grid environment information is input into the QRFS-DDPG model to determine the defect elimination plan for distribution network defects, and the method can use a reinforcement learning algorithm to realize automatic identification, classification and intelligent scheduling of defect processing of power grid defects, reduce dependence on manual experience, improve the intelligence level of power grid defect management, improve the accuracy and response speed of defect processing, improve the efficiency and quality of defect processing, and ensure the stable and safe operation of the power grid; at the same time, the defect elimination plan of the power grid defects can be optimized, and a scientific and reasonable defect elimination plan can be arranged for the power grid defects to ensure efficient allocation and use of resources, and improve the operation efficiency and safety of the power grid. For example, human resources can be better coordinated, defect elimination plans can be optimized, power outage time can be reduced, and the overall reliability of power grid operation can be improved. In addition, since the advantages of the DDPG algorithm lie in its stability and adaptability to high-dimensional continuous action spaces, and distribution network defect management decisions are easily affected by the environment, in this embodiment, the deep deterministic policy gradient method in reinforcement learning is improved to obtain a distribution network defect management method based on the optimized DDPG network, which can enhance the adaptability and flexibility of power grid defect management. Through the adaptability of the algorithm, the defect management system can adapt to various complex environments and emergencies, ensuring that the power grid can maintain stable operation in the face of uncertainty. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 It is a flow chart of a distribution network defect management method based on an improved deep deterministic strategy according to an embodiment of the present disclosure;

[0055] Figure 2 The interaction mode between the intelligent agent and the environment of the embodiment of the present disclosure;

[0056] Figure 3 It is the QRFS algorithm framework of the embodiment of the present disclosure;

[0057] Figure 4 It is a structural schematic diagram of a distribution network defect management device based on an improved deep deterministic strategy according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] Various aspects and features of the present disclosure are described herein with reference to the drawings.

[0059] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0060] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0061] These and other characteristics of the present disclosure will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the attached drawings.

[0062] It should also be understood that, although the present disclosure has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present disclosure that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0063] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0064] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments claimed are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0065] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0066] First, the terms that may be involved in the following text are explained. Table 1 shows the proper terms that may be involved and their definitions.

[0067] Table 1 Proper nouns and their meanings that may be involved

[0068]

[0069] Embodiment 1

[0070] Figure 1 A flow chart of a distribution network defect management method based on an improved deep deterministic strategy according to an embodiment of the present disclosure is shown. Figure 1 As shown, a distribution network defect management method based on an improved deep deterministic strategy provided by an embodiment of the present disclosure includes:

[0071] S101: Establish a power grid environment database.

[0072] Power grid data includes defect record database, weather status database, line database and human resources database.

[0073] The status in the defect record library includes: 1. Defect category; 2. Defect severity; 3. Defect urgency. The status in the weather status library includes: 1. Defect elimination date; 2. Highest and lowest temperatures of the day; 3. Maximum wind force level; 4. Humidity. The status in the line database includes: 1. Line operation years; 2. Line tripping records. The status in the human resources database includes: 1. The roster of available personnel for each team; 2. Information on the number of operators at different levels.

[0074] The above four databases together constitute an environment for intelligent agents to learn and interact.

[0075] S102: Training the DDPG model through interactive learning between the intelligent agent and the power grid environment database.

[0076] Deep Deterministic Policy Gradient (DDPG) is an advanced deep reinforcement learning algorithm specifically designed to handle problems in continuous action spaces. It combines the Actor-Critic framework with deep learning techniques and uses deterministic policy gradients for optimization, effectively learning the best action to take in a given state. The core idea of ​​the DDPG algorithm is to solve reinforcement learning problems in continuous action spaces by combining deterministic policy gradient methods with deep learning techniques.

[0077] like Figure 2 As shown, in this step, after the power grid environment database is established, the intelligent agent interacts with the environment to learn and train the DDPG model to obtain a strategy model for managing distribution network defects.

[0078] Step S102 specifically includes:

[0079] S1021: constructing a policy network (Actor network) and a value network (Critic network), wherein the policy network outputs a dispatch work order decision according to the current power grid state, and the value network evaluates the expected return of the dispatch work order decision;

[0080] S1022: The policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy.

[0081] The agent interacts with the power grid environment by executing actions (dispatching defect work orders) at each time step, and collects the resulting state transition and reward information to obtain the distribution network defect management strategy. The above process is iterative, and the agent adjusts its strategy according to the latest state transition and reward information in each iteration to obtain the optimal strategy, that is, the best defect management model. The Actor network and the Critic network collaborate with each other, the Actor network generates actions, and the Critic network provides feedback, jointly driving the strategy towards higher returns.

[0082] S103: Optimize the noise in the DDPG model using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model.

[0083] There is a certain amount of noise in the DDPG model, which affects the accuracy of the model. Therefore, in power grid defect management, by optimizing the noise parameters, the adaptability and flexibility of the intelligent agent in a complex environment can be improved, thereby achieving a better defect handling strategy.

[0084] S104: Input the acquired power grid environment information into the QRFS-DDPG model to determine a plan for eliminating distribution network defects.

[0085] After the QRFS-DDPG model is constructed, the acquired power grid environment information can be input into the QRFS-DDPG model. The QRFS-DDPG model processes the power grid environment information, identifies the power grid defects therein, and classifies the power grid defects. According to the identified power grid defects or the classification results of the power grid defects, a better processing plan is set for the power grid defects, and the processing of each defect is intelligently scheduled to arrange the best defect elimination plan.

[0086] Step S104 specifically includes:

[0087] S1041: Score the importance of eliminating distribution network defects in the power grid environment information by using the QRFS-DDPG model to obtain a scoring result;

[0088] S1042: Scheduling the processing of the distribution network defect and dispatching a work order according to the scoring result.

[0089] When the power grid environment information contains multiple defects, the QRFS-DDPG model can score the importance of eliminating each defect, schedule the processing of distribution network defects and dispatch work orders according to the importance from high to low, and obtain the optimal defect elimination plan.

[0090] The distribution network defect management method based on the improved deep deterministic strategy provided by the embodiment of the present disclosure establishes a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resource database; a DDPG model is trained through interactive learning between an intelligent agent and the power grid environment database; the noise in the DDPG model is optimized using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model; the acquired power grid environment information is input into the QRFS-DDPG model to determine the defect elimination plan for the distribution network defects, and the automatic identification, classification and intelligent scheduling of defect processing of power grid defects can be realized by using a reinforcement learning algorithm, etc., to reduce dependence on manual experience, improve the intelligence level of power grid defect management, improve the accuracy and response speed of defect processing, improve the efficiency and quality of defect processing, and ensure the stable and safe operation of the power grid; at the same time, the defect elimination plan of the power grid defects can be optimized, and a scientific and reasonable defect elimination plan can be arranged for the power grid defects to ensure the efficient allocation and use of resources, and improve the operation efficiency and safety of the power grid. For example, human resources can be better coordinated, the defect elimination plan can be optimized, the power outage time can be reduced, and the overall reliability of the power grid operation can be improved.

[0091] In addition, since the advantages of the DDPG algorithm lie in its stability and adaptability to high-dimensional continuous action spaces, and distribution network defect management decisions are easily affected by the environment, in this embodiment, the deep deterministic policy gradient method in reinforcement learning is improved to obtain a distribution network defect management method based on the optimized DDPG network, which can enhance the adaptability and flexibility of power grid defect management. Through the adaptability of the algorithm, the defect management system can adapt to various complex environments and emergencies, ensuring that the power grid can maintain stable operation in the face of uncertainty.

[0092] In some embodiments, in step S1021, constructing a policy network and a value network includes:

[0093] S201: Determine the strategy function and value function .

[0094] Among them, the strategy function , which means that in a given state s, according to the parameter Determine the strategy distribution, output action a, the value function Predict that in state s, action a is performed and policy is followed The expected return of the agent is s, the state s is the power grid environment where the agent is located, and the action a is the defect work order sent by the agent. The defect work order includes the allocation to the preset defect team and the expected time for defect elimination. represents the policy network parameters, represents the value network parameters;

[0095] S202: Determine the objective function .

[0096] The objective function specifies the learning goal of the agent, which is to maximize the expected return. The sum of the discounted returns obtained is expressed as:

[0097] (1)

[0098] In formula (1), is the discount factor for the time step, is the immediate reward obtained at time t.

[0099] Before building the policy network and value network, it is necessary to initialize the policy network parameters, value network parameters, and corresponding target network parameters through the agent.

[0100] Furthermore, in step S1022, the policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy, including:

[0101] S301: updating the policy network parameters through policy gradient to optimize the objective function and learn the optimal policy;

[0102] S302: Update the value network parameters by minimizing the prediction error of the Q value to learn the value of the state-action pair and obtain the optimal value.

[0103] Policy gradient uses the policy gradient theorem to deduce how to update policy parameters to optimize the objective function. For deterministic policies, according to the policy gradient theorem, the policy network parameters can be optimized by gradient ascent. :

[0104] (2)

[0105] in, are the Actor network parameters, Critic network parameters, Actor network parameters Updates are made via policy gradients to learn the optimal policy. It is the action output by the Actor network according to the current state s. Critic network parameters The value of a state-action pair is learned by updating it by minimizing the prediction error of the Q value.

[0106] In order to stabilize the learning process, the target network is used and , which are slowly updated copies of the main network parameters. The Actor target network is used to provide the policy for the next state. The Critic target network is used to evaluate the policy for the next state.

[0107] The loss function of the Critic network is the mean square error between the predicted Q value and the target Q value. The loss function measures the difference between the Critic network's current prediction of the action value and the actual reward obtained (estimated by the target network). It follows the following formula:

[0108] (3)

[0109] Among them, y is the target Q value.

[0110] Calculating the target Q value is a key intermediate step required to update the Critic network. The calculation method is as follows:

[0111] (4)

[0112] Formula (4) uses the Bellman equation, where It’s an instant reward. is the discount factor, and Q' is the output of the target Critic network.

[0113] After executing the action, both the Actor network and the Critic network need to update their own parameters. The gradient ascent method and policy gradient are used to update the Critic network and the Actor network respectively, following the formulas as follows:

[0114] (5)

[0115] (6)

[0116] In addition, the target network parameters also need to be soft-updated to ensure that the target network smoothly follows the main network, thereby maintaining the stability of the learning process. The update follows the following formula:

[0117] (7)

[0118] in, It is a positive number close to 0, which is used to control the speed of updating the target network parameters.

[0119] In some embodiments, the method further comprises:

[0120] S301: Acquire distribution network experience data, wherein the distribution network experience data includes at least one of a defect status, an execution status of a work order, and a difference between an actual defect elimination time and a planned defect elimination time;

[0121] S302: updating strategy estimation and value estimation according to the distribution network experience data;

[0122] S303: Train the DDPG model using the updated strategy estimate and value estimate.

[0123] In this embodiment, an experience playback mechanism is introduced, and the agent will collect the experience data. Store to playback buffer In , these experiences are used for subsequent batch learning and strategy evaluation, updating its strategy and value estimation, and reducing the correlation between samples through experience data. It is based on the difference between the actual removal time of the defect and the planned removal time. If the actual removal time of the defect is earlier than or equal to the planned time, a positive reward is given; if it is later than the planned time, a negative reward is given. This simple binary reward function can be adjusted as needed to reflect the reward intensity in different situations.

[0124] The agent uses a reward or penalty signal (the signal is based on the comparison between the actual defect elimination time and the planned elimination time). In this embodiment, this instant, performance-based reward mechanism is used to motivate the agent to optimize the generation of work orders, which can reduce defect processing time and improve the efficiency of power grid operation, and enhance the ability to intelligently, reasonably, and efficiently schedule distribution network defects.

[0125] In some embodiments, in step S103, the noise in the DDPG model is optimized using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model, including:

[0126] S1031: Iteratively searching for the optimal noise under a preset power grid environment by gradually narrowing the search space;

[0127] S1032: Add the optimal noise to the action output by the strategy network.

[0128] In the DDPG algorithm, noise is added to the action space through the Ornstein-Uhlenbeck process to increase the agent's exploratory behavior during training. The Ornstein-Uhlenbeck process is a time-correlated random process that can generate continuous, biased noise. However, noise itself also has some disadvantages: (1) The noise level is difficult to determine: if the noise is too large, it may cause the agent to take invalid or harmful actions; if the noise is too small, it may not be enough to encourage sufficient exploration. (2) Noise may affect learning stability: in some cases, adding noise may make the learning process unstable, especially in complex or high-risk environments. (3) Noise parameter adjustment may be complex: finding the right noise level may require a lot of adjustment and experimentation.

[0129] To solve the above problems, the quasi-random fractal search algorithm (QRFS) is introduced in the embodiment of the present disclosure to optimize the noise in DDPG. The QRFS algorithm can help automatically adjust the noise parameters to achieve better strategy performance. Through the iterative search process of the algorithm, the optimal exploration noise level in a specific environment can be found, thereby guiding the intelligent agent to conduct effective exploration, accelerate the learning process, and improve the intelligence level of power grid defect management.

[0130] QRFS is a new metaheuristic algorithm that uses fractal geometry, low-discrepancy sequences, and intelligent search space partitioning techniques to solve complex global optimization problems. The algorithm is based on the generation of fractals and initializes the population by carefully selecting low-discrepancy sequences to seek the global minimum of the objective function. This process strategically narrows the fractal search space, directing each population to the most promising area. The key operator of the algorithm is the initialization of different fractal populations. Compared with random initialization, these populations are initialized with low-discrepancy sequences, which comprehensively enhances the algorithm's exploration of the search space. This favorable initialization allows the algorithm to identify promising areas early in the iterative process. At the end of the iteration, the focus shifts to refining the best area found. This refinement is facilitated by the dynamic nature of the population controlled by the sigmoid function. The sigmoid function coordinates the gradual reduction of the population. Therefore, the QRFS algorithm focuses on exploring the optimal solutions in the initial stage and continuously improves these solutions until the end of the iterative process. The flowchart of QRFS is shown in Figure 2 shown.

[0131] QRFS includes population management methods, where population initialization is coordinated by low-discrepancy sequences, called quasi-random or quasi-random number sequences. Unlike purely random sequences, these deterministic sequences provide a more uniform spatial distribution, which helps to better cover the search space or data. In QRFS, the population of each fractal consists of:

[0132] (8)

[0133] Among them, npop represents the population size that decreases according to the sigmoid function, that is, the population size. npop simulates the population size that gradually decreases during the entire iteration process; b and u b represents lower and upper bounds that change dynamically during iterations to adapt to the changing problem space; the dimensionality of the problem is captured by dims; sequence represents one of four randomly selected low variances.

[0134] The QRFS algorithm manages the overall reduction of each fractal using the sigmida function contained in equation (9) below. This equation coordinates the reduction of the population size from the initial value to the final value, with the parameter adjusting the rate of this reduction.

[0135] (9)

[0136] In equation (9), npop represents the population size, and represent the initial and final population sizes, respectively. represents the total number of iterations, and I represents the current number of iterations.

[0137] Initialization is one of the key operations of the QRFS algorithm and is crucial in exploration and exploitation. As the iterative process gradually narrows the search space and reduces the number of algorithms, the starting phase becomes the main driver of these fundamental aspects. The algorithm is initialized by creating a set of named populations (nfractals), where each population represents a fractal and is selected through a random low-discrepancy sequence. The fitness of each solution of the population of all fractals is then determined, including identifying the best and worst solutions, followed by the calculation of the best solution. and the worst-positioned solution The Euclidean distance (ED) between the two solutions is then added and subtracted from the optimal solution to redefine the boundaries of the fractal. The Euclidean distance is calculated as:

[0138] (10)

[0139] The above process transforms the best solution into a centroid, and the two new solutions obtained by adding and subtracting the Euclidean distance become the updated upper and lower limits of the fractal. The reason for redefining the boundaries of the fractal using the Euclidean distance is to ensure that the new limits do not exceed the search space defined by the function being evaluated.

[0140] The iterative process of the QRFS algorithm consists of gradually reducing the search space by reducing the lower and upper limits of each fractal and reducing the algorithm population at the same time. Initially, the population of nfractals is established according to their respective new limits, i.e., determined by the initialization process defined in equation (8) above.

[0141] After the population is initialized, the fitness of the solutions within each fractal population is evaluated to determine the best and worst solutions. The best and worst solutions are crucial to determine the centroid of the fractal.

[0142] In each iteration, the centroid with the best fit is designated as the overall best centroid, and a new centroid for each fractal is calculated using Equation (11).

[0143] (11)

[0144] In formula (11), represents the location of the best solution for a given fractal, represents the best solution among all fractals, and j ranges from 1 to nfractals.

[0145] Then, the distance between the worst solution of each fractal and the new centroid is calculated using formula (12):

[0146] (12)

[0147] In formula (12), represents the location of the worst solution for a given fractal, represents the position of the new centroid of the fractal. Then, from the new centroid ( ) minus this distance (ED Ncj ), ensuring that the resulting solutions become the new upper and lower limits of the fractal. It can be understood that the above solutions remain within the search space defined by the function being evaluated. This process is repeated until the iterative cycle is completed.

[0148] Using the quasi-random fractal search algorithm, the search space is gradually narrowed down and the optimal noise under the preset power grid environment is iteratively searched. In the implementation of the DDPG algorithm, the optimized optimal noise is added to the action output by the Actor network. The formula can be expressed as:

[0149] (13)

[0150] in, is the action chosen at time step t, It is the action output by the Actor network is the noise term extracted from the Ornstein-Uhlenbeck process at each step. This noise term is optimized using the quasi-random fractal search algorithm described above.

[0151] The Ornstein-Uhlenbeck process is a time-dependent stochastic process, and its update rule can be expressed as:

[0152] (14)

[0153] in, is the decay rate, is the mean of the noise, is the standard deviation of the noise, is a random sample drawn from a standard normal distribution, is the time step.

[0154] In power grid defect management, optimizing noise parameters through the above-mentioned quasi-random fractal search algorithm can improve the adaptability and flexibility of the intelligent agent in complex environments, thereby achieving a better defect handling strategy.

[0155] It should be noted that the swarm intelligence optimization (population management) algorithm used when optimizing noise parameters may include a variety of algorithms, such as particle swarm optimization algorithm, ant colony optimization algorithm, etc., and other optimization algorithm methods can also optimize noise parameters. In addition, the disclosed embodiment can adopt a multi-agent reinforcement learning algorithm by simulating the collaborative work of multiple agents (such as different fault elimination teams) in the power grid. Each agent can learn the optimal fault elimination strategy based on its own task allocation, resource status and environmental feedback, so as to obtain the best fault elimination arrangement for different fault elimination teams and improve the scope of application of the defect management solution.

[0156] In summary, the distribution network defect management method based on the improved deep deterministic strategy provided by the embodiment of the present disclosure collects power grid environmental information including a defect record library, a weather status library, a line data library, and a human resources library, and uses a deep deterministic policy gradient algorithm optimized by a quasi-random fractal search algorithm to learn information such as defect data, weather conditions, lines, and human resources. The difference between the actual defect elimination time and the planned elimination time is used as a reward or penalty signal. A more comprehensive and reasonable defect elimination strategy can be formulated, which effectively improves the automatic management capability of distribution network defects and ensures the stable operation of the power grid and the reliability of power supply.

[0157] Embodiment 2

[0158] Figure 4 A schematic diagram of the structure of a distribution network defect management device based on an improved deep deterministic strategy according to an embodiment of the present disclosure is shown. Figure 4As shown, the embodiment of the present disclosure provides a distribution network defect management device based on an improved deep deterministic strategy, including:

[0159] The model building module 10 is configured as a database building module, configured to build a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resources database;

[0160] A model training module 20, configured to train the DDPG model through interactive learning between the intelligent agent and the power grid environment database;

[0161] A model optimization module 30 is configured to optimize the noise in the DDPG model using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model;

[0162] The defect management module 40 is configured to input the acquired power grid environment information into the QRFS-DDPG model to determine a defect elimination plan for the distribution network defects.

[0163] In some embodiments, the model training module 20 includes:

[0164] A construction unit configured to construct a strategy network and a value network, wherein the strategy network outputs a dispatch work order decision according to a current power grid state, and the value network evaluates an expected return of the dispatch work order decision;

[0165] The training unit is configured to train the policy network and the value network through a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy.

[0166] In some embodiments, the building block is further configured to:

[0167] Determine the policy function and value function , where the strategy function , which means that in a given state s, according to the parameter Determine the strategy distribution, output action a, the value function Predict that in state s, action a is performed and policy is followed The expected return of the agent is s, the state s is the power grid environment where the agent is located, and the action a is the defect work order sent by the agent. The defect work order includes the allocation to the preset defect team and the expected time for defect elimination. represents the policy network parameters, represents the value network parameters;

[0168] Determine the objective function , where the objective function is defined as starting from the current state s and following the strategy The sum of the discounted returns obtained is expressed as:

[0169] ,

[0170] In the formula, is the discount factor for the time step, is the immediate reward obtained at time t;

[0171] The training unit is further configured as:

[0172] Updating the policy network parameters through policy gradient to optimize the objective function and learn the optimal policy;

[0173] The value network parameters are updated by minimizing the prediction error of the Q value to learn the value of the state-action pair and obtain the optimal value.

[0174] In some embodiments, the training unit is further configured to:

[0175] The policy network parameters are optimized by gradient ascent, expressed as:

[0176] ,

[0177] in, It is the action output by the policy network according to the current state s.

[0178] In some embodiments, the training unit is further configured to:

[0179] The loss function for determining the update of the value network is:

[0180] ,

[0181] Wherein, y is the target Q value, and the calculation formula of the target Q value is:

[0182] ,

[0183] In the formula, It’s an instant reward. is the discount factor and Q' is the output of the target value network.

[0184] In some embodiments, the model training module 20 is further configured to:

[0185] Acquire distribution network experience data, wherein the distribution network experience data includes at least one of a defect status, an execution status of a work order, and a difference between an actual defect elimination time and a planned defect elimination time;

[0186] Updating strategy estimates and value estimates based on the distribution network experience data;

[0187] The DDPG model is trained using the updated policy estimate and value estimate.

[0188] In some embodiments, the defect management module 40 is further configured to:

[0189] Using the QRFS-DDPG model, scoring the importance of eliminating distribution network defects in the power grid environment information to obtain a scoring result;

[0190] The processing of the distribution network defects is scheduled and a work order is dispatched according to the scoring results.

[0191] In some embodiments, the model optimization module 30 is further configured to:

[0192] Iteratively search for the optimal noise under the preset power grid environment by gradually narrowing the search space;

[0193] The optimal noise is added to the action output by the policy network, expressed as:

[0194] ,

[0195] in, is the action chosen at time step t, is the action output by the Actor network, is the noise term extracted from the Ornstein-Uhlenbeck process at each step.

[0196] In some embodiments, the construction unit is further configured to: before determining the policy function and the value function, initialize the policy network parameters and the value network parameters, as well as the corresponding target network parameters through the agent.

[0197] The distribution network defect management device based on the improved deep deterministic strategy provided in the embodiment of the present disclosure corresponds to the distribution network defect management method based on the improved deep deterministic strategy in the above-mentioned embodiment. Any optional items in the embodiment of the distribution network defect management method based on the improved deep deterministic strategy are also applicable to the embodiment of the distribution network defect management device based on the improved deep deterministic strategy, which will not be repeated here.

[0198] Embodiment 3

[0199] An embodiment of the present disclosure also provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned distribution network defect management method based on the improved deep deterministic strategy when executing the computer program on the memory.

[0200] In some embodiments, the processor executing the computer program may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0201] The memory may be a read-only memory (ROM), random access memory (RAM), phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), other types of random access memory (RAM), flash disks or other forms of flash memory, cache, registers, static memory, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, cassettes or other magnetic storage devices, or any other possible non-transitory medium used to store information or instructions that can be accessed by a computer device.

[0202] The electronic devices of the embodiments of the present disclosure may include but are not limited to fixed terminal devices such as servers, desktop computers, digital TVs, etc., and mobile terminal devices such as vehicle-mounted devices (such as head-up display devices), handheld devices (such as mobile phones, tablet computers, etc.), and wearable devices (such as smart watches, smart bracelets, etc.).

[0203] Embodiment 4

[0204] The embodiment of the present disclosure also provides a computer-readable storage medium, wherein the computer-readable medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned distribution network defect management method based on the improved deep deterministic strategy is implemented.

[0205] The computer-readable storage medium of the embodiments of the present disclosure may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device, for example, the above-mentioned memory.

[0206] The computer programs of the embodiments of the present disclosure may be organized into one or more computer executable components or modules. Any number and combination of such components or modules may be used to implement the various aspects of the present disclosure. For example, the various aspects of the present disclosure are not limited to the specific computer executable instructions or specific components or modules shown in the drawings and described herein. Other embodiments may include different computer executable instructions or components with more or less functions than those shown and described herein.

[0207] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A distribution network defect management method based on an improved deep deterministic strategy, characterized in that: include: Establishing a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resources database; Training the DDPG model through interactive learning between the intelligent agent and the power grid environment database; The noise in the DDPG model is optimized by using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model; The acquired power grid environment information is input into the QRFS-DDPG model to determine the defect elimination plan for the distribution network.

2. The method according to claim 1, characterized in that: The DDPG model is trained through interactive learning between the intelligent agent and the power grid environment database, including: Constructing a policy network and a value network, wherein the policy network outputs a dispatch work order decision according to the current power grid state, and the value network evaluates the expected return of the dispatch work order decision; The policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy.

3. The method according to claim 2, characterized in that Build a strategy network and a value network, including: Determine the policy function and value function , where the strategy function , which means that in a given state s, according to the parameter Determine the strategy distribution, output action a, the value function Predict that in state s, action a is performed and policy is followed The expected return of the agent is s, the state s is the power grid environment where the agent is located, and the action a is the defect work order sent by the agent. The defect work order includes the allocation to the preset defect team and the estimated defect elimination time. represents the policy network parameters, represents the value network parameters; Determine the objective function , where the objective function is defined as starting from the current state s and following the strategy The sum of the discounted returns obtained is expressed as: ， In the formula, is the discount factor for the time step, is the immediate reward obtained at time t; The policy network and the value network are trained by a deterministic policy gradient algorithm and deep learning to obtain a distribution network defect management strategy, including: Updating the policy network parameters through policy gradient to optimize the objective function and learn the optimal policy; The value network parameters are updated by minimizing the prediction error of the Q value to learn the value of the state-action pair and obtain the optimal value.

4. The method according to claim 3, characterized in that Updating the policy network parameters by policy gradient includes: The policy network parameters are optimized by gradient ascent, expressed as: ， in, It is the action output by the policy network according to the current state s.

5. The method according to claim 3, characterized in that: The value network parameters are updated by minimizing the prediction error of the Q value, including: The loss function for determining the update of the value network is: ， Wherein, y is the target Q value, and the calculation formula of the target Q value is: ， In the formula, It’s an instant reward. is the discount factor and Q' is the output of the target value network.

6. The method according to claim 2, characterized in that The method further comprises: Acquire distribution network experience data, wherein the distribution network experience data includes at least one of a defect status, an execution status of a work order, and a difference between an actual defect elimination time and a planned defect elimination time; Updating strategy estimates and value estimates based on the distribution network experience data; The DDPG model is trained using the updated policy estimate and value estimate.

7. The method according to claim 1, characterized in that The acquired power grid environment information is input into the QRFS-DDPG model to determine the defect elimination plan for the distribution network, including: Using the QRFS-DDPG model, scoring the importance of eliminating distribution network defects in the power grid environment information to obtain a scoring result; The processing of the distribution network defects is scheduled and a work order is dispatched according to the scoring results.

8. The method according to claim 2, characterized in that: The noise in the DDPG model is optimized using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model, including: Iteratively search for the optimal noise under the preset power grid environment by gradually narrowing the search space; The optimal noise is added to the action output by the policy network, expressed as: ， in, is the action chosen at time step t, is the action output by the Actor network, is the noise term extracted from the Ornstein-Uhlenbeck process at each step.

9. The method according to claim 3, characterized in that: Before determining the strategy function and the value function, the method further includes: Initialize the policy network parameters, value network parameters, and corresponding target network parameters through the agent.

10. A distribution network defect management device based on an improved deep deterministic strategy, characterized in that: include: A database establishment module, configured to establish a power grid environment database, wherein the power grid environment database includes a defect record database, a weather status database, a line database and a human resources database; A model training module, configured to train the DDPG model through interactive learning between the intelligent agent and the power grid environment database; A model optimization module is configured to optimize the noise in the DDPG model using a quasi-random fractal search algorithm to obtain a QRFS-DDPG model; The defect management module is configured to input the acquired power grid environment information into the QRFS-DDPG model to determine the defect elimination plan for the distribution network.

Images