A substation drone inspection trajectory planning method
The three-dimensional grid map is constructed through DE algorithm and NOA algorithm, and combined with the D3QN algorithm to optimize the track, it solves the accuracy and adaptability of the drone's patrol path in complex substation environments, improves the patrol efficiency and quality, and reduces the cost.
Patent Information
- Application Number
- CN202410462044.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-04-17
AI Technical Summary
The existing drone track planning methods cannot effectively deal with the complex environment of substations, resulting in energy waste and inspection omissions, affecting the quality of inspection.
The DE algorithm is used to build a three-dimensional grid map, combine the NOA algorithm to plan multiple tracks globally, and local track optimization is performed through the D3QN algorithm to ensure the accuracy and adaptability of the drone patrol path.
It improves the efficiency and quality of substation inspections of drones, reduces operating costs, and realizes the importance of automated substation inspections of drones.
Smart Images

Figure CN118310530B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for planning a track of an unmanned aerial vehicle (UAV), and in particular to a method for planning a track of an unmanned aerial vehicle (UAV) inspection of a transformer substation. Background Art
[0002] The stable operation of substations is crucial to maintaining the normal operation of the power system. For a long time, substation inspections have relied on manual work, which is time-consuming, labor-intensive and involves safety risks. In order to ensure personnel safety and improve inspection efficiency, drones are used in substation inspections, but achieving accurate and efficient trajectory planning has always been a technical difficulty. Traditional drone trajectory planning relies on manual planning and cannot cope with the complex and changeable environmental conditions of substations. In addition, the manually planned path is difficult to guarantee to be the optimal solution, which will cause energy waste of the drone, inspection omissions, and affect the overall inspection quality. Summary of the invention
[0003] Purpose of the invention: The purpose of the present invention is to provide a method for realizing optimal inspection track planning during substation UAV inspection, thereby improving the inspection efficiency and quality of substations by UAVs.
[0004] Technical solution: The substation drone inspection trajectory planning method of the present invention comprises:
[0005] (1) Construct a three-dimensional grid map of the substation environment;
[0006] (2) Global multi-track planning on the three-dimensional grid map is performed using the DE algorithm;
[0007] (3) The optimal track of multiple global tracks is solved through the NOA algorithm to obtain the global optimal inspection track.
[0008] Further, step (1) includes: collecting substation environmental data by using a UAV-mounted laser radar;
[0009] The substation area to be measured is divided into m×n×s l×l rectangular grids, and each rectangular grid is assigned a unique row and column number and corresponding angle information in the sensor coordinate system;
[0010] The positions of the N data points detected by the laser radar are set to: p i =(x i ,y i ,z i ), i∈[1,N], projecting it into m×n×s rectangular grids; p i Represents the location information of the i-th data point; x i ,y i 、z i Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th data point in turn;
[0011] Calculate the probability P of each small grid being occupied i , the formula is as follows:
[0012]
[0013] The distance r from the sensor's coordinate origin to the center of each small grid is calculated as follows:
[0014]
[0015] Where l represents the length of a grid; multiple small grid information stacks form a three-dimensional grid map;
[0016] The grid information of each substation is fused to obtain a complete three-dimensional grid map of the substation environment.
[0017] Furthermore, the grid status is determined based on the following:
[0018] ① when , the grid is determined to be in an occupied state, and p i =1;
[0019] ② when , the grid is determined to be unoccupied, and p i =0.
[0020] Further, step (2) comprises:
[0021] (201) Initialize the population
[0022] Assume that NP initial drone individuals x are uniformly and randomly generated in a D-dimensional problem solution space. j,i,0 , the formula is as follows:
[0023] x j,i,0 =x j,min +rand[0,1]×(x j,max ,x j,min ),
[0024] i=1,2,…,NP,j=1,2,…,D
[0025] Among them, x j,i,0 is the initial individual in the 0th generation population; x j,min is the minimum value of the j-th dimension component of the drone individual x; rand[0,1] is a random number uniformly generated in the interval [0,1]; x j,max is the maximum value of the j-th dimension component of the drone individual x; NP is the size of the population; D does not exceed 3;
[0026] (202) Mutation Operation
[0027] Generate variant individuals V through differential strategy i,G , the formula is as follows:
[0028]
[0029] Among them, V i,G is the mutant individual of the G generation population; r1 i ,r2 i ,r3 i is the individual number randomly selected; F is the weight factor; and is the individual taken from the G-th generation population as the target individual X;
[0030] (203) Crossover Operation
[0031] The mutant individual and the target individual are randomly combined to generate the crossover individual u j,i,G , the formula is as follows:
[0032]
[0033] Among them, u j,i,G is the j-th dimension component of the crossover individuals in the G-th generation population; v j,i,G is the j-th dimension component of the mutant individuals in the G-th generation population; x j,i,G is the j-th dimension component of the target individual in the G-th generation population; CR is the crossover probability, ranging from [0,1]; j rand is a random number between [1,2,…,NP];
[0034] (204)Select operation
[0035] Select the best individuals who are more adapted to the environment to enter the next generation and continue to reproduce. The formula is as follows:
[0036]
[0037] Among them, X i,G+1 is the target individual position of the G+1 generation population; U i,G is the crossover individual position of the G generation population; X i,G is the target individual position of the G-generation population;
[0038] (205) Repeat the mutation, crossover, and selection operations until the preset number of iterations or the number of UAV tracks is reached, and multiple global tracks are obtained.
[0039] Further, step (3) comprises:
[0040] (301) Population Initialization
[0041] The global multiple tracks obtained by the DE algorithm are used as the problem search space solution set of the NOA algorithm, and each individual Represents a global track path; the population initialization formula is as follows:
[0042]
[0043] in, represents the j-th dimension variable of individual i; t represents the number of iterations; represents the upper bound of the j-th dimension variable; represents the lower bound of the j-th dimension variable; is a random vector between [0,1]; N is the size of the population; D is the dimension of the problem search space, which does not exceed 3 dimensions;
[0044] (302) Foraging
[0045] Find the relatively optimal solution in the problem search space. The position update formula of the individual nutcracker in the optimization process is as follows:
[0046]
[0047] in, is the position of the ith nutcracker at the t+1th iteration; is the jth position of the i-th nutcracker at the current t-th iteration; is the mean of all solutions of the jth dimension in the current population in the tth iteration; γ is a random number generated according to the Levy flight; A, B, C are three random numbers; U j is the upper bound of the j-th dimension of the optimization problem; L j is the lower bound of the j-th dimension of the optimization problem; τ1, τ2, r and r1 are random real numbers between [0,1]; T max represents the maximum number of iterations; δ represents the probability estimate of the area that can be covered in the problem search space;
[0048] μ is based on the normal distribution (τ4), the Levy flight (τ5), and a randomly generated number between 0 and 1 (τ3), as shown in the following formula:
[0049]
[0050] Among them, r2 and r3 are random real numbers between [0,1];
[0051] (303) Food Storage
[0052] After finding the relatively optimal track, the individual nutcracker saves the optimal track and transfers it to the set storage point. The formula is as follows:
[0053]
[0054] in, It is the new location of the current Nutcracker individual storage area; is the current t-th iteration position of the i-th nutcracker; is the best position solution of the jth dimension in the tth iteration obtained so far; and is the position of the randomly selected individual nutcracker in the tth iteration; λ is a random number generated according to the Levy flight; τ3 is a random number between [0,1]; l is a linear decreasing factor from 1 to 0;
[0055] (304) Second stage of foraging
[0056] During the optimal trajectory solution process, the NOA algorithm selects appropriate reference points to update the storage location and applies the corresponding exploration mechanism to search for the most promising areas, which may contain a near-optimal solution;
[0057] The position update formula is as follows:
[0058]
[0059] in, is the new position of the ith nutcracker at the t+1th iteration; is the current position of the ith nutcracker at the tth iteration; It is the first reference point;
[0060] (305) Second phase development
[0061] When searching for a save point, the Starling may encounter two possibilities:
[0062] The first possibility is that the nutcracker can use the first reference point to remember the location of the storage point; if the location of the storage point is remembered, there are two possibilities as follows: the first is that food exists, and the second is that food does not exist; the mathematical expression of this behavior is as follows:
[0063]
[0064] in, is the updated position; j is the dimension; is the best position for the current iteration; is the first reference point of the current position of the i-th nutcracker; r1, r2, τ3 and τ4 are random numbers between [0,1]; C refers to the index of a solution randomly selected from the population;
[0065] The second possibility is that the Nutcracker fails to find the storage point using the first reference point, and then continues searching using the second reference point. The specific formula is as follows:
[0066]
[0067] in, is the second reference point;
[0068] If we assume that the nutcracker finds its storage point when using the second reference point, the update formula for the second reference point is:
[0069]
[0070] Among them, r1, r2, τ5 and τ6 are random numbers between [0,1].
[0071] Furthermore, the position calculation formula of the reference point is as follows:
[0072] The position update formula of the first reference point is as follows:
[0073]
[0074] The calculation formula for the second reference point is as follows:
[0075]
[0076]
[0077] in, and Indicates the location of the i-th Nudibranch storage point in the current t-th iteration Two reference points, where is the first reference point, is the second reference point; α is a linear decreasing factor from 1 to 0; θ is a random number between [0, π]; is the storage point location of the A-th nutcracker in the current t-th iteration; The storage point location of the Bth nutcracker in the current tth iteration; It is the upper bound of the D-dimensional optimization problem; is the lower bound of the D-dimensional optimization problem; τ3 is a random number between [0,1]; Refers to a vector of randomly generated numbers between 0 and 1; P rp is a parameter used to determine the probability that the nutcracker will globally explore other areas within the search space.
[0078] Furthermore, the calculation formula of the linear decreasing factor α from 1 to 0 is as follows:
[0079]
[0080] Where t represents the number of current iterations; T max Represents the maximum number of iterations.
[0081] Furthermore, the substation drone inspection trajectory planning method also includes: (4) local trajectory optimization of the global optimal inspection trajectory through an improved D3QN algorithm, and the improvement method of the D3QN algorithm is: the D3QN algorithm obtains the maximum Q value through the ALS method.
[0082] The present invention completes the local track planning of the UAV during the inspection process through the improved D3QN algorithm to improve the local obstacle avoidance ability of the UAV during the inspection process. At the same time, in view of the slow convergence speed of the D3QN algorithm itself, it is proposed to use the ALS alternating least squares iteration method to obtain the maximum Q value in the D3QN algorithm, and substitute it into the Bellman equation to obtain the target Q value, so as to accelerate the convergence of the D3QN algorithm.
[0083] Further, step (4) comprises:
[0084] Given a policy π, in state s t The action a of the drone takes based on its current state t ; After taking action, the drone enters a new state s t+1 , and receives a reward value r from the environment t ;
[0085] Set the optimal movement strategy to π * , maximize the cumulative reward value Among them, r τ represents the reward value obtained from the environment at time τ; T is the time at the end; γ∈[0,1] is the discount factor;
[0086] The calculation formula of the inspection drone motion system structure is as follows:
[0087] Given a policy π:s t →a t , state s t and state-action pairs (s t ,a t ) is defined as follows:
[0088] Qπ(s t ,a t )=Eπ[R π |s t ,a t ]
[0089]
[0090] Among them, Qπ (s t ,a t ) is the state-action value function, which means that under a given strategy π, the drone is in state s t Action taken at a t The cumulative reward value; Vπ(s t ) is the state value function, indicating that under the strategy π, the drone is in state s t The cumulative reward value at; E represents the expected value; R π Indicates that under a given strategy π, the drone is in state s t Action taken at a t The immediate reward value of
[0091] The D3QN algorithm trains the neural network by minimizing the square error between the target Q value and the predicted Q value. Its loss function is:
[0092]
[0093]
[0094] Among them, θ i is the weight vector of the neural network; is the target value of the Q function; r t is the reward value; γ∈[0,1] is the discount factor;
[0095] Collect experience tuples during the experiment (s t ,a t ,r t ,s t+1 ) and store it in the replay memory D; during the optimization process, randomly sample small batches of empirical values and input them into the network to solve the correlation problem between continuous sample data;
[0096] The D3QN algorithm obtains the maximum Q value through the ALS method and substitutes it into the Bellman equation to obtain the target Q value;
[0097] Set up matrix R as evaluation matrix, influence matrix P and target value matrix Q; decompose the evaluation matrix R into the multiplication of influence matrix P and target value matrix Q to obtain the screening of target Q value;
[0098]
[0099] Among them, R u,i represents the preference of target u for value i; M is the number of latent factors; P u,m is the preference degree of target u for the mth latent factor; Q i,m is the weight of value i in the mth latent factor;
[0100] The mean square error (RMSE) is used to train the P and Q matrices, and the loss function is:
[0101]
[0102] Among them, r ui represents the scoring result of target u for value i, p u represents the implicit feature factor vector of target u, q i is the implicit feature factor vector of value i; λ represents the regularization term coefficient used to prevent overfitting of the model; ||p u || 2 and||q i || 2 Respectively represent p u and q i The norm of; k represents the set of all (u,i) elements;
[0103] Use alternating least squares to improve the loss function. Fix one of the matrices, first fix Q, and calculate the loss function L(p,q) for p u The partial derivative is equal to 0, and the formula is as follows:
[0104] p u =(Q T Q+λE) -1 +Q T r u
[0105] Similarly, fix P and continue to calculate L(p,q) versus q i The partial derivative is equal to 0, and the formula is as follows:
[0106] q i =(P T P+λE) -1 +P T r i
[0107] Where E is the identity matrix of f×f; r u represents the scoring result of target u; r i Represents the scoring result of value i;
[0108] The DDQN algorithm is used to decouple the selection and calculation of the target Q value, and then the action operation value and the next state value are replaced into the target network to calculate the target Q value;
[0109] Value function V(s t ; ω, β) is only related to the state value but not to the action operation value. The advantage function A(s) is related to both the state value and the action operation value. t ,a t ;ω,α);
[0110] The Q function calculation formula is as follows:
[0111]
[0112] Among them, ω is the common parameter of the two sub-networks; α is the parameter of the dominant function sub-network, and β is the parameter of the advantage function sub-network. is the average value of the advantage function;
[0113] The drone motion control task is decomposed into multiple subtasks, each of which is identified by a different reward function; at time t, the total reward value is equal to the sum of the reward values of all subtasks, which is calculated using the following formula:
[0114]
[0115] Furthermore, the UAV motion control task is decomposed into speed control and direction control;
[0116] The reward function of the speed control branch can be defined as follows:
[0117]
[0118] Among them, r is a parameter used to adjust the reward size; v is the current speed of the drone; v max is the maximum speed of the drone; v min is the minimum speed of the drone;
[0119] The reward function for direction control can be expressed as:
[0120]
[0121] The speed control reward function and the direction control function are used to control the normal movement of the drone. When a collision occurs, a relatively large feedback should be given to the drone to ensure its safety during movement.
[0122] The collision reward function is defined as:
[0123]
[0124] Where, d is the minimum distance to surrounding objects; D max and D min are two distance thresholds;
[0125] The inspection drone avoids obstacles by controlling speed and direction, thereby establishing a collision reward function. The calculation formula of the collision reward function is:
[0126]
[0127] r tk (s t ,a t ,s t+1 )=R v (s t ,a t ,s t+1 )+R o (s t ,a t ,s t+1 )
[0128] Among them, r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; It is the cumulative sum of the speed control reward value and the direction control value;
[0129] The reward function of each Q-network branch has a corresponding Q-value function; Q MDA Combine them according to their respective weights to select the action operation of the drone;
[0130]
[0131] The loss function can be updated as:
[0132]
[0133]
[0134] Among them, L i represents the loss function of the i-th action; θ i is the weight vector of the neural network; Indicates that in state s t , take action a t , get reward r t and the next state s t+1 Expected value under is the target value of the Q function; r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; γ∈[0,1] is the discount factor; y k,i represents the i-th target Q value in the k-th subtask.
[0135] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: the present invention adopts the DE algorithm to complete the global track planning of UAVs during the inspection of substations, selects the possible tracks of all UAV inspection tracks, and retains the global search capability of the DE algorithm; at the same time, the NOA algorithm is combined to complete the optimal track selection of the possible tracks of multiple UAV inspection tracks, which has the advantages of high track planning accuracy and adaptability, can improve the efficiency and quality of UAV inspections of substations, and reduce the operating costs of inspections, which is of great significance for realizing the automated inspection of substations by UAVs. BRIEF DESCRIPTION OF THE DRAWINGS
[0136] Figure 1 This is a flowchart of a substation drone inspection trajectory planning method provided by an embodiment of the present invention;
[0137] Figure 2 It is a block diagram of the DE-NOA algorithm in an embodiment of the present invention;
[0138] Figure 3 4 is a block diagram of an improved D3QN algorithm in an embodiment of the present invention. DETAILED DESCRIPTION
[0139] The present invention will be further described below in conjunction with the accompanying drawings.
[0140] like Figure 1 As shown, an embodiment of the present invention provides a substation drone inspection trajectory planning method, which specifically includes the following steps:
[0141] (1) Construct a three-dimensional grid map of the substation environment;
[0142] The substation environment data is collected by drone-mounted laser radar. After the inspection drone completes the scanning of the entire environment space, it records the information of all equipment and buildings in detail;
[0143] When constructing a detailed 3D grid map of the substation, the substation area to be measured is divided into m×n×s l×l rectangular grids according to specific requirements. Each rectangular grid is assigned a unique row and column number and corresponding angle information in the sensor coordinate system (initially, the unexplored area is marked in gray on the map);
[0144] The specific information of the three-dimensional environment grid map collected by the lidar is as follows:
[0145] The positions of the N data points detected by the laser radar are set to: p i =(x i ,y i ,z i ), i∈[1,N], and project it into m×n×s rectangular grids to accurately reflect the substation environment; p iRepresents the location information of the i-th data point; x i ,y i 、z i Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th data point in turn;
[0146] Calculate the probability P of each small grid being occupied i , the formula is as follows:
[0147]
[0148] The distance r from the sensor's coordinate origin to the center of each small grid is calculated as follows:
[0149]
[0150] Where l represents the length of a grid; multiple small grid information stacks form a three-dimensional grid map;
[0151] The grid status is determined based on the following:
[0152] ① when , the grid is determined to be in an occupied state, and p i =1;
[0153] ② when , the grid is determined to be unoccupied, and p i =0.
[0154] The grid information of each substation is fused to obtain a complete three-dimensional grid map of the substation environment.
[0155] (2) Perform global multi-track planning on the three-dimensional grid map through the DE algorithm (differential evolution algorithm);
[0156] Differential evolution (DE) is a heuristic random search method that relies on population diversity. It optimizes a given objective function by iteratively searching for the optimal solution.
[0157] (201) Initialize the population
[0158] In the initialization phase, NP initial individuals are uniformly and randomly generated in a D-dimensional problem solution space to meet specific constraints. These individuals constitute the 0th generation population.
[0159] Assume that NP initial drone individuals x are uniformly and randomly generated in a D-dimensional problem solution space. j,i,0 , the formula is as follows:
[0160] x j,i,0 =x j,min+rand[0,1]×(x j,max ,x j,min ),
[0161] i=1,2,…,NP,j=1,2,…,D
[0162] Among them, x j,i,0 is the initial individual in the 0th generation population; x j,min is the minimum value of the j-th dimension component of the drone individual x; rand[0,1] is a random number uniformly generated in the interval [0,1]; x j,max is the maximum value of the j-th dimension component of the drone individual x; NP is the size of the population; D does not exceed 3;
[0163] (202) Mutation Operation
[0164] In the standard differential evolution algorithm (DE), the generation of mutant individuals is mainly through a differential strategy method. That is, firstly, several individual vectors in the population are randomly selected, and then for the selected vector, the difference between the other two vectors is calculated and multiplied by a preset weight factor, and the weighted difference vector obtained is added to the selected vector to generate the mutant individual V. i,G , the formula is as follows:
[0165]
[0166] Among them, V i,G is the mutant individual of the G generation population; r1 i ,r2 i ,r3 i is the individual number randomly selected; F is the weight factor, which is a constant in the range [0,2] and is used to control the degree of disturbance of the difference vector; and is the individual taken from the G-th generation population as the target individual X;
[0167] (203) Crossover Operation
[0168] The mutant individual and the target individual are randomly combined to generate the crossover individual u j,i,G , in order to improve population diversity. The crossover operation of the standard differential evolution algorithm usually adopts binomial crossover, that is, the number of dimensions selected for crossover conforms to the binomial distribution rule. The calculation formula of the crossover operation is as follows:
[0169]
[0170] Among them, u j,i,G is the j-th dimension component of the crossover individuals in the G-th generation population; v j,i,G is the j-th dimension component of the mutant individuals in the G-th generation population; x j,i,Gis the j-th dimension component of the target individual in the G-th generation population; CR is the crossover probability, ranging from [0,1]; j rand is a random number between [1,2,…,NP], which is used to ensure that at least one dimension of the crossover individual is different from the target individual;
[0171] (204)Select operation
[0172] The selection operation is to select from the target individuals and crossover individuals, and through the competition law of "survival of the fittest", select the best individuals that are more adapted to the environment to enter the offspring and continue to reproduce. The calculation formula for the selection operation is as follows:
[0173]
[0174] Among them, X i,G+1 is the target individual position of the G+1 generation population; U i,G is the crossover individual position of the G generation population; X i,G is the target individual position of the G-generation population;
[0175] (205) Repeat the mutation, crossover, and selection operations until the preset number of iterations or the number of UAV tracks is reached, and multiple global tracks are obtained.
[0176] Finally, through the differential evolution algorithm (DE), all possible tracks of drone inspections were screened out, and several potential inspection routes were preliminarily selected.
[0177] (3) The optimal trajectory of multiple global trajectories is solved by the NOA algorithm (Nutcracker Optimization Algorithm) to obtain the global optimal inspection trajectory.
[0178] When formulating the global trajectory planning of UAV inspection of substations, it is necessary to consider that the equipment to be inspected in different inspection tasks may change, so the inspection nodes are also adjusted according to the task requirements. The trajectory generated by the differential evolution algorithm (DE) alone may not always guarantee the global optimal solution in a complex substation environment. To solve this problem, the NOA algorithm is introduced, which optimizes the multiple initial trajectories output by the differential evolution algorithm to ensure that the global trajectory path of the UAV inspection is the optimal solution.
[0179] (301) Population Initialization
[0180] The global multiple tracks obtained by the DE algorithm are used as the problem search space solution set of the NOA algorithm, and each individual Represents a global track path;
[0181] The Nutcracker Optimization Algorithm (NOA) needs to randomly generate a series of initial solution points, i.e., nutcracker individuals, in the problem search space. If the size of the population is set to N and the dimension of the problem search space is set to D (not more than 3 dimensions), the population initialization can be performed using the following formula:
[0182]
[0183] in, represents the j-th dimension variable of individual i; t represents the number of iterations; represents the upper bound of the j-th dimension variable; represents the lower bound of the j-th dimension variable; is a random vector between [0,1];
[0184] (302) Foraging
[0185] Nutcrackers start from their initial positions, search for food in the search space, and prey. Then they look for the relatively optimal solution in the problem search space. The position update formula of Nutcrackers in the optimization process is as follows:
[0186]
[0187] in, is the position of the ith nutcracker at the t+1th iteration; is the jth position of the i-th nutcracker at the current t-th iteration; is the mean of all solutions of the jth dimension in the current population in the tth iteration; γ is a random number generated according to the Levy flight; A, B, C are three random numbers; U j is the upper bound of the j-th dimension of the optimization problem; L j is the lower bound of the j-th dimension of the optimization problem; τ1, τ2, r and r1 are random real numbers between [0,1]; T max represents the maximum number of iterations; δ represents the probability estimate of the area that can be covered in the problem search space.
[0188] μ is based on the normal distribution (τ4), the Levy flight (τ5), and a randomly generated number between 0 and 1 (τ3), as shown in the following formula:
[0189]
[0190] Among them, r2 and r3 are random real numbers between [0,1]; μ is used to provide the necessary information for the nutcracker to guide it to explore new positions in the search space, including determining the direction and distance of movement.
[0191] (303) Food Storage
[0192] After finding the food, the individual nutcracker is responsible for transferring it to the designated storage point. During this process, the position of the nutcracker is updated according to a specific formula to ensure that the food can be stored efficiently and safely.
[0193] After finding the relatively optimal track, the individual nutcracker saves the optimal track and transfers it to the set storage point. The formula is as follows:
[0194]
[0195] in, is the new position of the storage area of the individual nutcracker at the current t+1th iteration; is the current t-th iteration position of the i-th nutcracker; is the best position solution of the jth dimension in the tth iteration obtained so far; and is the position of the randomly selected individual nutcracker in the tth iteration; λ is a random number generated according to the Levy flight; τ3 is a random number between [0,1]; l is a linear decreasing factor from 1 to 0, which is used to diversify the food storage process of the NOA algorithm;
[0196] (304) Second stage of foraging
[0197] In the NOA algorithm, the storage and development phases of food are based on two key reference points (RPs) selected by the individual nutcrackers. That is, in order to effectively remember the location of stored food, the nutcrackers will select two specific reference points.
[0198] The position of the reference point is calculated as follows:
[0199] The first reference point is used to explore potential hidden food around the nutcrackers, and these storage points are located by updating the current position in the relevant space. The position update formula of the first reference point is as follows:
[0200]
[0201] The function of the second reference point is to improve the current solution within the search space of the problem, which helps the Starfinch algorithm explore a wider area and thus discover potential high-quality solutions. The calculation formula of the second reference point is as follows:
[0202]
[0203]
[0204] in, and Indicates the location of the i-th Nudibranch storage point in the current t-th iteration Two reference points, where is the first reference point, is the second reference point; α is a linear decreasing factor from 1 to 0; θ is a random number between [0, π]; is the storage point location of the A-th nutcracker in the current t-th iteration; The storage point location of the Bth nutcracker in the current tth iteration; It is the upper bound of the D-dimensional optimization problem; is the lower bound of the D-dimensional optimization problem; τ3 is a random number between [0,1]; It refers to a vector of randomly generated numbers between 0 and 1; RP represents random position; P rp is a parameter used to determine the probability that the nutcracker will globally explore other areas within the search space.
[0205] The parameter α is intended to ensure the convergence of the algorithm and avoid locking the reference point on the sub-optimal solution (i.e., storage point) too early. The calculation formula of α is as follows:
[0206]
[0207] Where t represents the number of current iterations; T max Represents the maximum number of iterations.
[0208] In the second stage of foraging in the NOA algorithm, the nutcrackers use their spatial memory strategy to determine where food is stored.
[0209] In the NOA algorithm, the two reference point matrices of the storage points are defined as follows:
[0210]
[0211] During the optimal trajectory solution process, the NOA algorithm selects appropriate reference points to update the storage location and applies the corresponding exploration mechanism to search for the most promising areas, which may contain a near-optimal solution;
[0212] The position update formula is as follows:
[0213]
[0214] in, is the new position of the ith nutcracker at the t+1th iteration; is the current position of the ith nutcracker at the tth iteration; It is the first reference point;
[0215] (305) Second phase development
[0216] When searching for a save point, the Starling may encounter two possibilities:
[0217] The first possibility is that the nutcracker can use the first reference point to remember the location of the storage point; if the location of the storage point is remembered, there are two possibilities as follows: the first is that food exists, and the second is that food does not exist; the mathematical expression of this behavior is as follows:
[0218]
[0219] in, is the updated position; j is the dimension; is the best position for the current iteration; is the first reference point of the current position of the i-th nutcracker; r1, r2, τ3 and τ4 are random numbers between [0,1]; C refers to the index of a solution randomly selected from the population;
[0220] The second possibility is that the Nutcracker fails to find the storage point using the first reference point, and then continues searching using the second reference point. The specific formula is as follows:
[0221]
[0222] in, is the second reference point;
[0223] If we assume that the nutcracker finds its storage point when using the second reference point, the update formula for the second reference point is:
[0224]
[0225] Among them, r1, r2, τ5 and τ6 are random numbers between [0,1].
[0226] The NOA algorithm is used to find the optimal trajectory of the inspection trajectory solution set selected by the DE algorithm, and the optimal trajectory of the UAV in the substation inspection process is obtained.
[0227] The algorithm flow of steps (2) and (3) is as follows: Figure 2 shown.
[0228] (4) The global optimal inspection track is locally optimized through the improved D3QN algorithm. The improvement method of the D3QN algorithm is as follows: the D3QN algorithm obtains the maximum Q value through the ALS method.
[0229] During the inspection of substations by drones, within a limited area, it is necessary to use sensor data and map information, combined with obstacle avoidance strategies, to enable the drone to quickly and safely plan its own flight path in a complex environment. During the inspection of substations by drones, there may be sudden obstacles, etc., which pose a safety hazard to the drone. The present invention proposes to optimize the local track of the unmanned inspection track in combination with the Dueling Double Deep Q Network algorithm (D3QN), thereby improving the safety of the drone. The present invention defines the driving action of the inspection drone as a discrete form, and controls the linear velocity and angular velocity respectively. The interaction between the drone and the environment can be modeled using a Markov decision process (MDP).
[0230] like Figure 3 As shown, step (4) specifically includes:
[0231] Given a policy π, in state s t The action a of the drone takes based on its current state t ; After taking action, the drone enters a new state s t+1 , and receives a reward value r from the environment t ;
[0232] Set the optimal movement strategy to π * , maximize the cumulative reward value Among them, r τ represents the reward value obtained from the environment at time τ; T is the time at the end; γ∈[0,1] is the discount factor;
[0233] The calculation formula of the inspection drone motion system structure is as follows:
[0234] Given a policy π:s t →a t , state s t and state-action pairs (s t ,a t ) is defined as follows:
[0235] Q π (s t ,a t )=E π [R π |s t ,a t ]
[0236]
[0237] Among them, Q π (s t ,a t) is the state-action value function, which means that under a given strategy π, the drone is in state s t Action taken at a t The cumulative reward value of V π (s t ) is the state value function, indicating that under the strategy π, the drone is in state s t The cumulative reward value at; E represents the expected value; R π Indicates that under a given strategy π, the drone is in state s t Action taken at a t The immediate reward value.
[0238] Reusing advantage function A π (s t ,a t ) to measure each action operation a t In status t The advantage relative to the average is given by the following formula:
[0239] A π (s t ,a t )=Q π (s t ,a t )-V π (s t )
[0240] The D3QN algorithm trains the neural network by minimizing the square error between the target Q value and the predicted Q value. Its loss function is:
[0241]
[0242]
[0243] Among them, θ i is the weight vector of the neural network; is the target value of the Q function; r t is the reward value; γ∈[0,1] is the discount factor;
[0244] Collect experience tuples during the experiment (s t ,a t ,r t ,s t+1 ) and store it in the replay memory D; during the optimization process, randomly sample small batches of empirical values and input them into the network to solve the correlation problem between continuous sample data;
[0245] The D3QN algorithm obtains the maximum Q value through the ALS method (Alternative Least Square) and substitutes it into the Bellman equation to obtain the target Q value;
[0246] Set up matrix R as evaluation matrix, influence matrix P and target value matrix Q; decompose the evaluation matrix R into the multiplication of influence matrix P and target value matrix Q to obtain the screening of target Q value;
[0247]
[0248] Among them, R u,i represents the preference of target u for value i; M is the number of latent factors; P u,m is the preference degree of target u for the mth latent factor; Q i,m is the weight of value i in the mth latent factor;
[0249] The mean square error (RMSE) is used to train the P and Q matrices, and the loss function is:
[0250]
[0251] Among them, r ui represents the scoring result of target u for value i, p u represents the implicit feature factor vector of target u, q i is the implicit feature factor vector of value i; λ represents the regularization term coefficient used to prevent overfitting of the model; ||p u || 2 and||q i || 2 Respectively represent p u and q i The norm of ; k represents the set of all (u,i) elements.
[0252] Use alternating least squares to improve the loss function. Fix one of the matrices, first fix Q, and calculate the loss function L(p,q) for p u The partial derivative is equal to 0, and the formula is as follows:
[0253] p u =(Q T Q+λE) -1 +Q T r u
[0254] Similarly, fix P and continue to calculate L(p,q) versus q i The partial derivative is equal to 0, and the formula is as follows:
[0255] q i =(P T P+λE)-1 +P T r i
[0256] Where E is the identity matrix of f×f; r u represents the scoring result of target u; r i Represents the scoring result of value i.
[0257] The Double DQN (DDQN) algorithm based on the DQN dual network architecture is used to decouple the selection and calculation of the target Q value, that is, an online network is used to calculate the action operation value of the maximum Q value, and then the action operation value and the next state value are replaced into the target network to calculate the target Q value.
[0258] Among them, the value function V(s t ; ω, β) is only related to the state value but not to the action operation value. The advantage function A(s) is related to both the state value and the action operation value. t ,a t ;ω,α);
[0259] The Q function calculation formula is as follows:
[0260]
[0261] Among them, ω is the common parameter of the two sub-networks; α is the parameter of the dominant function sub-network, and β is the parameter of the advantage function sub-network. is the average value of the advantage function;
[0262] The drone motion control task is decomposed into multiple subtasks, each of which is represented by a different reward function; at time t, the total reward value is equal to the sum of the reward values of all subtasks, which is calculated using the following formula:
[0263]
[0264] The motion control task of the drone is decomposed into speed control and direction control, which are specifically implemented as follows:
[0265] The reward function of the speed control branch can be defined as follows:
[0266]
[0267] Among them, r is a parameter used to adjust the reward size; v is the current speed of the drone; v max is the maximum speed of the drone; v min is the minimum speed of the drone;
[0268] Likewise, the fewer times a drone changes direction, the less time it takes to reach its target, reducing unnecessary turns and lowering energy consumption.
[0269] The reward function for direction control can be expressed as:
[0270]
[0271] The speed control reward function and the direction control function are used to control the normal movement of the drone. When a collision occurs, a relatively large feedback should be given to the drone to ensure the safe operation of the drone during the movement.
[0272] The collision reward function is defined as:
[0273]
[0274] Where, d is the minimum distance to surrounding objects; D max and D min are two distance thresholds;
[0275] The inspection drone avoids obstacles by controlling speed and direction, thereby establishing a collision reward function. The calculation formula of the collision reward function is:
[0276]
[0277] r t k (s t ,a t ,s t+1 )=R v (s t ,a t ,s t+1 )+R o (s t ,a t ,s t+1 )
[0278] Among them, r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; It is the cumulative sum of the speed control reward value and the direction control value.
[0279] The reward function of each Q network branch has a corresponding Q value function, and the network composed of all Q value functions forms a multi-task decomposition architecture. MDA Combine them according to their respective weights to select the action operation of the drone;
[0280]
[0281] Furthermore, the loss function can be updated as:
[0282]
[0283]
[0284] Among them, L i represents the loss function of the i-th action; θ i is the weight vector of the neural network; Indicates that in state s t , take action a t , get reward r t and the next state s t+1 Expected value under is the target value of the Q function; r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; γ∈[0,1] is the discount factor; y k,i represents the i-th target Q value in the k-th subtask, where k is the direction control R o With speed control R v .
Claims
1. A substation drone inspection trajectory planning method, characterized in that: include: (1) Construct a three-dimensional grid map of the substation environment; (2) Global multi-track planning is performed on the three-dimensional grid map through the differential evolution algorithm DE; (3) The optimal track of multiple global tracks is solved by using the NOA optimization algorithm to obtain the global optimal inspection track; Step (3) includes: (301) Population Initialization The global multiple tracks obtained by the differential evolution algorithm DE are used as the problem search space solution set of the Nutcracker optimization algorithm NOA; the population initialization formula is as follows: in, represents the j-th dimension variable of individual i, i.e., a global trajectory path; t represents the number of iterations; represents the upper bound of the j-th dimension variable; represents the lower bound of the j-th dimension variable; is a random vector between [0,1]; N is the size of the population; D is the dimension of the problem search space, which does not exceed 3 dimensions; (302) Foraging Find the relatively optimal solution in the problem search space. The position update formula of the individual nutcracker in the optimization process is as follows: in, is the position of the ith nutcracker at the t+1th iteration; is the jth position of the i-th nutcracker at the current t-th iteration; is the mean of all solutions of the jth dimension in the current population in the tth iteration; γ is a random number generated according to the Levy flight; A, B, C are three random numbers; U j is the upper bound of the j-th dimension of the optimization problem; L j is the lower bound of the j-th dimension of the optimization problem; τ1, τ2, r and r1 are random real numbers between [0,1]; T max represents the maximum number of iterations; δ represents the probability estimate of the area that can be covered in the problem search space; μ is based on the normal distribution τ4, Levy flight τ5 and a randomly generated number τ3 between 0 and 1, as shown in the following formula: Among them, r2 and r3 are random real numbers between [0,1]; (303) Food Storage After finding the relatively optimal track, the individual nutcracker saves the optimal track and transfers it to the set storage point. The formula is as follows: in, It is the new location of the current Nutcracker individual storage area; is the current t-th iteration position of the i-th nutcracker; is the best position solution of the jth dimension in the tth iteration obtained so far; and is the position of the randomly selected individual nutcracker in the tth iteration; λ is a random number generated according to the Levy flight; τ3 is a random number between [0,1]; l is a linear decreasing factor from 1 to 0; (304) Second stage of foraging During the optimal trajectory solution process, the Nutcracker Optimization Algorithm (NOA) selects appropriate reference points to update the storage location and applies the corresponding exploration mechanism to search for the most promising areas that contain an approximate optimal solution. The position update formula is as follows: in, is the new position of the ith nutcracker at the t+1th iteration; is the current position of the ith nutcracker at the tth iteration; It is the first reference point; (305) Second phase development When searching for a storage point, the Star Crow encounters two situations: In the first case, the nutcracker uses the first reference point to remember the location of the storage point. If the location of the storage point is remembered, there are two other cases: food exists or food does not exist. The mathematical expression of this behavior is as follows: in, is the updated position; j is the dimension; is the best position for the current iteration; is the first reference point of the current position of the i-th nutcracker; r1, r2, τ3 and τ4 are random numbers between [0,1]; C refers to the index of a solution randomly selected from the population; The second situation is that the Nutcracker fails to find the storage point using the first reference point, and then continues to search using the second reference point. The specific formula is as follows: in, is the second reference point; If we assume that the nutcracker finds its storage point when using the second reference point, the update formula for the second reference point is: Among them, r1, r2, τ5 and τ6 are random numbers between [0,1].
2. The substation drone inspection trajectory planning method according to claim 1 is characterized in that: Step (1) includes: collecting substation environmental data by using a laser radar onboard a drone; The substation area to be measured is divided into m×n×s l×l rectangular grids, and each rectangular grid is assigned a unique row and column number and corresponding angle information in the sensor coordinate system; The positions of the N data points detected by the laser radar are set to: p i =(x i ,y i ,z i ), i∈[1,N], projecting it into m×n×s rectangular grids; p i Represents the location information of the i-th data point; x i ,y i 、z i Represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the i-th data point in turn; Calculate the probability P of each small grid being occupied i , the formula is as follows: The distance r from the sensor's coordinate origin to the center of each small grid is calculated as follows: Where l represents the length of a grid; multiple small grid information stacks form a three-dimensional grid map; The grid information of each substation is fused to obtain a complete three-dimensional grid map of the substation environment.
3. The substation drone inspection trajectory planning method according to claim 2 is characterized in that: The grid status is determined based on the following: ① when , the grid is determined to be in an occupied state, and p i =1; ② when , the grid is determined to be unoccupied, and p i =0.
4. The substation drone inspection trajectory planning method according to claim 1 is characterized in that: Step (2) comprises: (201) Initialize the population Assume that NP initial drone individuals x are uniformly and randomly generated in a D-dimensional problem solution space. j,i,0 , the formula is as follows: x j,i,0 =x j,min +rand[0,1]×(x j,max ,x j,min ), i=1,2,…,NP,j=1,2,…,D Among them, x j,i,0 is the initial individual in the 0th generation population; x j,min is the minimum value of the j-th dimension component of the drone individual x; rand[0,1] is a random number uniformly generated in the interval [0,1]; x j,max is the maximum value of the j-th dimension component of the drone individual x; NP is the size of the population; D does not exceed 3; (202) Mutation Operation Generate variant individuals V through differential strategy i,G , the formula is as follows: Among them, V i,G is the mutant individual of the G generation population; r1 i ,r2 i ,r3 i is the individual number randomly selected; F is the weight factor; and is the individual taken from the G-th generation population as the target individual X; (203) Crossover Operation The mutant individual and the target individual are randomly combined to generate the crossover individual u j,i,G , the formula is as follows: Among them, u j,i,G is the j-th dimension component of the crossover individuals in the G-th generation population; v j,i,G is the j-th dimension component of the mutant individuals in the G-th generation population; x j,i,G is the j-th dimension component of the target individual in the G-th generation population; CR is the crossover probability, ranging from [0,1]; j rand is a random number between [1,2,…,NP]; (204)Select operation Select the best individuals who are more adapted to the environment to enter the next generation and continue to reproduce. The formula is as follows: Among them, X i,G+1 is the target individual position of the G+1 generation population; U i,G is the crossover individual position of the G generation population; X i,G is the target individual position of the G-generation population; (205) Repeat the mutation, crossover, and selection operations until the preset number of iterations or the number of UAV tracks is reached, and multiple global tracks are obtained.
5. The substation drone inspection trajectory planning method according to claim 1 is characterized in that: The position of the reference point is calculated as follows: The position update formula of the first reference point is as follows: The calculation formula for the second reference point is as follows: in, and Indicates the location of the i-th Nudibranch storage point in the current t-th iteration Two reference points, where is the first reference point, is the second reference point; α is a linear decreasing factor from 1 to 0; θ is a random number between [0, π]; is the storage point location of the A-th nutcracker in the current t-th iteration; The storage point location of the Bth nutcracker in the current tth iteration; It is the upper bound of the D-dimensional optimization problem; is the lower bound of the D-dimensional optimization problem; τ3 is a random number between [0,1]; Refers to a vector of randomly generated numbers between 0 and 1; P rp is a parameter used to determine the probability that the nutcracker will globally explore other areas within the search space.
6. The substation drone inspection trajectory planning method according to claim 5 is characterized in that: The calculation formula of linear decreasing factor α from 1 to 0 is as follows: Where t represents the number of current iterations; T max Represents the maximum number of iterations.
7. The substation drone inspection trajectory planning method according to any one of claims 1 to 6, characterized in that: Also includes: (4) The global optimal inspection track is locally optimized through the improved D3QN algorithm. The improvement method of the D3QN algorithm is as follows: the D3QN algorithm obtains the maximum Q value through the ALS method.
8. The substation drone inspection trajectory planning method according to claim 7 is characterized in that: Step (4) comprises: Given a policy π, in state s t The action a of the drone takes based on its current state t ; After taking action, the drone enters a new state s t+1 , and receives a reward value r from the environment t ; Set the optimal movement strategy to π * , maximize the cumulative reward value Among them, r τ represents the reward value obtained from the environment at time τ; T is the time at the end; γ∈[0,1] is the discount factor; The calculation formula of the inspection drone motion system structure is as follows: Given a policy π:s t →a t , state s t and state-action pairs (s t ,a t ) is defined as follows: Q π (s t ,a t )=E π [R π |s t ,a t ] Among them, Q π (s t ,a t ) is the state-action value function, which means that under a given strategy π, the drone is in state s t Action taken at a t The cumulative reward value of V π (s t ) is the state value function, indicating that under the strategy π, the drone is in state s t The cumulative reward value at; E represents the expected value; R π Indicates that under a given strategy π, the drone is in state s t Action taken at a t The immediate reward value of The D3QN algorithm trains the neural network by minimizing the square error between the target Q value and the predicted Q value. Its loss function is: Among them, θ i is the weight vector of the neural network; is the target value of the Q function; r t is the reward value; γ∈[0,1] is the discount factor; Collect experience tuples during the experiment (s t ,a t ,r t ,s t+1 ) and store it in the replay memory D; during the optimization process, randomly sample small batches of empirical values and input them into the network to solve the correlation problem between continuous sample data; The D3QN algorithm obtains the maximum Q value through the ALS method and substitutes it into the Bellman equation to obtain the target Q value; Set up matrix R as evaluation matrix, influence matrix P and target value matrix Q; decompose the evaluation matrix R into the multiplication of influence matrix P and target value matrix Q to obtain the screening of target Q value; Among them, R u,i represents the preference of target u for value i; M is the number of latent factors; P u,m is the preference degree of target u for the mth latent factor; Q i,m is the weight of value i in the mth latent factor; The mean square error (RMSE) is used to train the P and Q matrices, and the loss function is: Among them, r ui represents the scoring result of target u for value i, p u represents the implicit feature factor vector of target u, q i is the implicit feature factor vector of value i; λ represents the regularization term coefficient used to prevent overfitting of the model; ||p u || 2 and||q i || 2 Respectively represent p u and q i The norm of; k represents the set of all (u,i) elements; Use alternating least squares to improve the loss function. Fix one of the matrices, first fix Q, and calculate the loss function L(p,q) for p u The partial derivative is equal to 0, and the formula is as follows: p u =(Q T Q+λE) -1 +Q T r u Similarly, fix P and continue to calculate L(p,q) versus q i The partial derivative is equal to 0, and the formula is as follows: q i =(P T P+λE) -1 +P T r i Where E is the identity matrix of f×f; r u represents the scoring result of target u; r i Represents the scoring result of value i; The DDQN algorithm is used to decouple the selection and calculation of the target Q value, and then the action operation value and the next state value are replaced into the target network to calculate the target Q value; Value function V(s t ; ω, β) is only related to the state value but not to the action operation value. The advantage function A(s) is related to both the state value and the action operation value. t ,a t ;ω,α); The Q function calculation formula is as follows: Among them, ω is the common parameter of the two sub-networks; α is the parameter of the dominant function sub-network, and β is the parameter of the advantage function sub-network. is the average value of the advantage function; The drone motion control task is decomposed into multiple subtasks, each of which is identified by a different reward function; at time t, the total reward value is equal to the sum of the reward values of all subtasks, which is calculated using the following formula:
9. The substation drone inspection trajectory planning method according to claim 8 is characterized in that: Decompose the UAV motion control task into speed control and direction control; The reward function of the speed control branch is defined as follows: Among them, r is a parameter used to adjust the reward size; v is the current speed of the drone; v max is the maximum speed of the drone; v min is the minimum speed of the drone; The reward function for direction control can be expressed as: The speed control reward function and direction control function are used to control the normal movement of the drone. When a collision occurs, feedback should be given to the drone to ensure its safety during movement. The collision reward function is defined as: Where, d is the minimum distance to surrounding objects; D max and D min are two distance thresholds; The inspection drone avoids obstacles by controlling speed and direction, thereby establishing a collision reward function. The calculation formula of the collision reward function is: r t k (s t ,a t ,s t+1 )=R v (s t ,a t ,s t+1 )+R o (s t ,a t ,s t+1 ) Among them, r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; It is the cumulative sum of the speed control reward value and the direction control value; The reward function of each Q-network branch has a corresponding Q-value function; Q MDA Combine them according to their respective weights to select the action operation of the drone; The loss function can be updated as: Among them, L i represents the loss function of the i-th action; θ i is the weight vector of the neural network; Indicates that in state s t , take action a t , get reward r t and the next state s t+1 Expected value under is the target value of the Q function; r t k represents the immediate reward obtained when executing the kth subtask in the tth iteration; γ∈[0,1] is the discount factor; y k,i represents the i-th target Q value in the k-th subtask.
