Quadrotor unmanned aerial vehicle mapping and planning method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of quadrotor unmanned aerial vehicles, and particularly relates to a quadrotor unmanned aerial vehicle mapping and planning method based on deep reinforcement learning, and the method comprises the following steps: S1, feature extraction and screening, S2, feature map creation, S3, feature map position calculation based on value iteration, and iteration through geometric verification and using a TD algorithm, and S4, state estimation. A planning method of quadrotor unmanned aerial vehicle mapping based on deep reinforcement learning is used for planning a generated flight trajectory of quadrotor unmanned aerial vehicle mapping, and the planning method comprises the following steps: S5, generating a strategy network model of the trajectory; according to the method, the common feature points and the feature point set between the frames can be rapidly extracted, the correlation graph of the feature points and the feature set is determined, and finally the corresponding relation and the position of the graph are calculated through a reinforcement learning method to obtain the accurate feature point relation.

Description

Technical Field

[0001] This invention belongs to the field of quadcopter UAV technology, specifically relating to a mapping and planning method for quadcopter UAVs based on deep reinforcement learning. Background Technology

[0002] A quadcopter drone is a multi-rotor drone that generates lift by rotating four rotors. Due to its characteristics such as vertical take-off and landing, simple mechanical structure, and flexible control, it is widely used in various fields. Mapping and planning of quadcopter drones refers to using the sensors (such as cameras and radar) carried by the drone to obtain information about the surrounding environment, and generating a three-dimensional model of the environment and the flight trajectory of the drone according to certain algorithms, so as to realize the autonomous navigation and mission execution of the drone.

[0003] Mapping and planning for quadrotor drones is a highly complex system involving knowledge from multiple fields such as computer vision, machine learning, and control theory. Designing high-performance mapping and planning algorithms is key to achieving quadrotor drone control. Currently, with the development of artificial intelligence technologies such as deep learning and reinforcement learning, some mapping and planning methods for quadrotor drones based on deep learning and reinforcement learning have been proposed. Deep learning is a machine learning method that uses multi-layer neural networks to learn features and patterns from data, while reinforcement learning is a machine learning method that uses the interaction between an agent and its environment to learn the optimal strategy through reward and punishment mechanisms. The advantages of mapping and planning methods for quadrotor drones based on deep learning and reinforcement learning are that they can adaptively learn the features and patterns of complex environments without requiring manual design of features and models, thus improving the efficiency and performance of mapping and planning. The disadvantages of mapping and planning methods for quadrotor drones based on deep learning and reinforcement learning are that they require a large amount of data and computing resources, it is difficult to explain and verify the principles and results of the algorithms, and there are issues with generalization ability and security.

[0004] In the discussion of the above problems, some companies have modeled the path planning problem as a partially observable Markov decision process and extended it to multi-agent systems using a proximal policy optimization algorithm. By designing the state observation space, action space, and reward function of UAVs, they have achieved barrier-free path planning for multiple UAVs. This method can effectively solve the problems of cooperative control and conflict avoidance of multiple UAVs and improve the collective intelligence and cooperation capabilities of UAVs. However, this method also has some shortcomings. For example, it does not consider the mapping problem of UAVs, cannot adapt to and update the dynamic changes in the environment, and does not consider the limitations of UAVs' communication and perception range, which may lead to incomplete and inconsistent information from UAVs.

[0005] In summary, existing mapping and planning methods for quadrotor UAVs, whether feature-based, direct-method, or based on deep learning and reinforcement learning, each have their own advantages and disadvantages. No single method can fully meet the mapping and planning needs of quadrotor UAVs in complex environments, and there is still room for improvement. Therefore, it is necessary to propose a mapping and planning method for quadrotor UAVs based on deep reinforcement learning. Summary of the Invention

[0006] The purpose of this invention is to provide a mapping and planning method for quadrotor UAVs based on deep reinforcement learning. This method can quickly extract common feature points and feature point sets between frames to determine the correlation graph of feature points and feature sets. Finally, it uses reinforcement learning to calculate the correspondence and position of the graph to obtain a more accurate feature point relationship. Based on reducing the computational overhead of environmental mapping, it only calculates the necessary connections and spatial features. At the same time, it completes the relevant aircraft trajectory planning work with the help of planning algorithms.

[0007] The specific technical solution adopted by this invention is as follows:

[0008] A mapping and planning method for quadrotor UAVs based on deep reinforcement learning includes the following steps:

[0009] S1: Feature extraction and filtering;

[0010] S2: Create a feature map;

[0011] S3: Calculate the feature map position based on value iteration, and verify it through geometric verification and iteratively using the TD algorithm;

[0012] S4: State estimation;

[0013] S5: Policy network model for generating trajectories;

[0014] S6: Value network model for evaluating trajectories;

[0015] S7: Generate the target's flight trajectory;

[0016] Specifically, S5 and S6 iterate with each other and complete the trajectory generation in S7.

[0017] In S1, the camera can acquire n frames of data after time k, and the formula for calculating the number of frames F is:

[0018] F = [F k+1 ,…,F k+n ], k = 1, 2, 3, ...

[0019] Define variable i to represent the frame number, that is:

[0020] i = k+1,…,k+n

[0021] For single-frame data F i The formula for calculating the feature points is as follows:

[0022]

[0023] Among them, S i S is the feature point vector of the i-th frame of data. i,m Let m represent the m-th feature point of the i-th frame of data. i This represents the number of feature points in the i-th frame of data;

[0024] Define feature point j, that is:

[0025] j = 1, ..., m

[0026] Where m represents the number of feature points in the current frame, which changes with the frame number i. However, when the formula contains feature point j, the value represented by m will not change.

[0027] And the set of feature points to which feature point j belongs is:

[0028] map(S i,j )={S i,j}

[0029] Among them, map(S) i,j ) represents the feature point S i,j A function mapping to the set of feature points, where the function maps to the set to which the feature points belong, i.e.:

[0030] S i,j ∈map(S i,j )

[0031] Furthermore, the set is defined as:

[0032] U i ={{S i,1},…,{S i,m}}

[0033]

[0034] Among them, U i Let E represent the set of all feature points contained in the i-th frame. Under initial conditions, any feature point is a single set. i This represents the association between feature sets in the i-th frame image;

[0035] To find candidate points with dense feature points, we first need to calculate the minimum interval of feature points in the frame. The formula for this calculation is as follows:

[0036]

[0037] Among them, S i,short S represents the minimum distance between feature points in the i-th frame of data. i,a and S i,a Let a and b represent the image positions of the a-th and b-th feature points of the i-th frame of data, respectively, and m represent the number of feature points of the i-th frame of data;

[0038] For each feature point S i The calculation formula is as follows:

[0039]

[0040] Where η represents the multiple of the feature point distance, which is a constant in the formula, and m represents the number of feature points in the i-th frame of data;

[0041] Feature point S i Merge into a set:

[0042] U i,a,b =map(S i,a )∪map(S i,b ),a∈{1,…,m},b∈{1,…,m},a>b

[0043] Among them, U i,a,b This represents the union of the sets of feature points containing the a-th and b-th feature points in the i-th frame of data; that is, merging two feature points into one set. m represents the number of feature points in the i-th frame of data. i,a and S i,a These represent the image positions of the a-th and b-th feature points in the i-th frame of data, respectively.

[0044] After the above modifications are completed, the feature point set result of the i-th frame is modified, and the formula is as follows:

[0045] E i ←(E i -{<u1,u2> |u1∈map(S i,a ),u2∈map(S i,b )})

[0046] ∪{ <map(S i,a ),map(S i,b )>}

[0047] U i ←(U i -map(S i,a )-map(S i,b ))∪U i,a,b

[0048] map(S i,a)=U i,a,b

[0049] map(S i,b )=U i,a,b

[0050] Where u1 and u2 represent two sets of feature points,<u1,u2> This indicates that two feature point sets u1 and u2 are related;

[0051] After completing the above steps, for U i Remove feature points from sets with fewer internal feature points, and then process E. i Synchronous updates are performed using the following formula:

[0052] U i ={u|u∈U i ,|u|>L}

[0053] E i ={<u1,u2> |u1,u2∈U i ,<u1,u2> ∈E i}

[0054] After completing the above steps, the initial screening of the i-th frame of data is complete.

[0055] We use a convolutional neural network to encode the feature point set locally, and then use a CNN model to hash all the feature points. This results in a smaller computational result as the feature point set becomes more similar in its arrangement, i.e., satisfying the following:

[0056] CrossEntropy (CNN (U l ),CNN(U r ))→∞,U l ≠U r

[0057] Where CNN() represents the result calculated by the CNN model, U l and U r These represent two sets of feature points; CrossEntropy() is the cross-entropy function.

[0058] The structure, size, and optimization methods of a CNN model have little impact on the final result;

[0059] The loss function that has a significant impact on the results is:

[0060]

[0061] U i ←(U i -map(S i,a )-map(S i,b ))∪Ui,a,b

[0062] map(S i,a )=U i,a,b

[0063] map(S i,b )=U i,a,b

[0064] in, Let U be the loss function. c This represents the set of feature points being evaluated. CNN(U c The current value of ), O1 represents all frames that do not contain U. c The total number of feature points in all frames, O2 represents the total number of feature points in all frames. c Number of occurrences, U k1 Indicates different from U c The set of feature points Represents and U c The same set of feature points;

[0065] Statistics F k ,…,F k+n The final output consists of n frames. Statistically, some encoded results appear consistently in every frame. These are defined as stable scene points, i.e.:

[0066] U st,k ={CNN(U i,l )|CNN(U i,l )∈CNN(U h ); l∈{1,…,mu i},h=k,…,k+n}

[0067] E st,k ={<u1,u2> |CNN(u1),CNN(u2)∈U st,k ,<u1,u2> ∈E i}

[0068] W st,k ={{e,O}|e∈E st,k}

[0069] Among them, U st,k E represents the set of feature points that appear stably in n frames starting from frame k. st,k W represents the association of a set of feature points that appear stably in n frames starting from frame k. st,k The sum E represents the spatial relationship of the set of feature points. st,k The associations are one-to-one, mu i U represents the number of feature sets in the i-th frame. i,lThis represents the set of the l-th feature points of the i-th frame of data.<u1,u2> This represents the association of a set of feature points that stably appear in n frames starting from frame k, where O denotes the zero vector; e denotes E. st,k The inner edge, U h Represents the set of all feature points in the h-th frame, CNN(U h ) indicates that for U h Calculate all elements within the range, that is:

[0070]

[0071] Where, m h The number of feature points in the h-th frame of data;

[0072] The final bidirectional weighted map result of the feature point set starting from frame k for n frames is obtained, and its formula is:

[0073] G k =(U st,k E st,k W st,k U k+n )

[0074] Among them, U k+n Let G represent the set of all feature points of the last frame processed in the k-th frame. k For the feature map of the k-th frame, define U st,k For feature map G k The node is defined as E. st,k For feature map G k The edge.

[0075] In step S2, based on the feature map of the k-th frame in S1, the positional relationships between scene points are calculated and corrected. Any set of feature points within the current frame is selected as the initial position, and the calculation formula is as follows:

[0076] U origin ∈U st,k U st,k ∈G k

[0077] The feature map is then constructed using the following formula:

[0078]

[0079] Where U represents the set of all explored feature points, E represents the association of the discovered feature point sets, and W corresponds to the relative positions of the elements in E. last This represents the set of all feature points in the last frame of data.

[0080] In S3, the dual DQN method is used to iteratively update the value of the feature point set, defining specific model details. For the above network, its model input is:

[0081] <a,b> ∈E st,k

[0082] The state is the association of feature point sets in a feature map, also known as graph edges. The relative positions between the two endpoints of an edge are represented by network actions:

[0083] A = [a x ,a y ,a z ]

[0084] For edge e:

[0085] e = a ,u b >

[0086] by u a to u b The process, using action A to represent relative positional displacement, yields a benefit derived from the sum of errors corrected at both ends and from other already corrected results. The calculation formula is as follows:

[0087]

[0088] Where sigmoid() represents the Sigmoid function, MSE() represents the mean squared error function, K is the camera intrinsic parameter, and A is the optimal action for edge e. b,j For the set of feature points u b and u j The optimal action, d(u), represents the number of connections between the feature point set u and the feature point set within U, i.e.:

[0089] u i ,u j ∈U

[0090] The following was calculated during the value iteration process:

[0091]

[0092] in, Let A represent the optimal action for edge e. e Let R(e,A) represent the best action for edge e in the previous iteration, and let R(e,A) represent the benefit of using action A for edge e.

[0093] After this, update the policy A of edge e using the greedy policy. e ,for:

[0094] ​

[0095] Search for and update the set of feature points connected to the existing graph, i.e.:

[0096] E′ G ={ k,a ,S k,b >| k,a ,S k,b >∈E st,k CNN(S) k,b )∈CNN(U)}

[0097] U′ G ={S k,a | k,a ,S k,b >∈E st,k CNN(S) k,b )∈CNN(U)}

[0098] The optimal action obtained through iteration is:

[0099]

[0100] Last updated G:

[0101] G←G∪(U′ G ,E′ G ,W′ G ,U′ last )

[0102] Among them, U' last This represents the data from the last frame.

[0103] In step S4, the current position of the aircraft is determined using the following formula:

[0104] X = [xyz]

[0105] In the equation, x, y, and z represent the body position relative to the initial set of feature points U, respectively. origin The distances along the X, Y, and Z axes;

[0106] At the same time, for each feature set, define a function mapping:

[0107] P(u f = [x0 y0 z0]

[0108] The result of this function mapping is solved using an iterative scheme based on the following formula:

[0109] P(v i )exist

[0110] ​​​Where P(u) represents node u relative to U origin The relative distance, w(u,v) i ) represents nodes u and v i The optimal action obtained from previous iterations, where h represents the number of associations between node u and other nodes.

[0111] The policy network model in S5 has parameters θ and an encoder input:

[0112] I = [G,A o A d ]

[0113] Where I is the system input, G is the feature map, and A is the system input. O Starting position, A d The endpoint of the run;

[0114] The iterative data can be obtained at the decoder end as follows:

[0115] T = T f (I,θ)=[A o ,T0,…,T r A d Alt o Alt d ]

[0116] Where T0,…,T r This is the generated location information, and r is the length of the generated trajectory;

[0117] While generating the sequence, the parameters θ of the above model are adjusted and updated. The adjustment method is as follows:

[0118]

[0119] Where η represents the learning rate of the policy network, Q(G,T) represents the value network in S6, which accepts inputs including feature map G and policy network output T, θ is the parameter of the generator network, and T is the policy with parameter θ and input I.

[0120] The value network model in S6 is generated based on the Transformers model and has a parameter ω. Its generation method is as follows: First, feature maps and trajectory sequences are used as encoder input to obtain data at the decoder end and network output results. Then, the parameters of the value network are updated to obtain the complete reward function. The specific steps are as follows:

[0121] Using the feature map G and trajectory sequence T as encoder input, the data at the decoder end, as well as the actual flight trajectory and flight attitude, can be obtained as follows:

[0122] Trreal =[T rl,0 ,…,T rl,r0 ]

[0123] Alt real =[Alt rl,0 ,…,Alt rl,r0 ]

[0124]

[0125] Among them, T Real For the final actual trajectory, Tr real For the actual spatial location, Alt real The actual attitude during operation is given by r0, where r0 is the sequence length of the actual trajectory.

[0126] Therefore, the network output can be obtained:

[0127] q=Q(ω,G,T,T Real )=[q th,0 ,…,q th,r ]

[0128] Where, q th,i This represents the value estimate of the i-th sequence outcome of the policy network by the value network;

[0129] For cases where no actual flight path exists, the following formula can be used for calculation:

[0130]

[0131] Where O is the zero vector, q n,i This represents the value estimate of the policy network from the starting point to the i-th sequence result in the absence of actual data;

[0132] The output of the above model needs to be calculated using the reward function and the parameters of the value network need to be updated. The following formula defines four types of losses:

[0133]

[0134] Among them, Loss illu (T h G) represents the collision loss of the flight trajectory, where d in the collision loss calculation represents the d nodes closest to the aircraft. Loss E (T h () represents the energy loss during flight. Represents the trajectory point T of the sequence h Find the result of the hd-th order differential, Loss reach (T h ,T real) indicates the loss of flight trajectory realization;

[0135] This represents the sampling point T used to find a suitable actual trajectory. real,i Make it closer to the trajectory point T generated by the strategy. h Loss alt (T,T real ) represents the pose loss, and μ represents the weight of the pose loss;

[0136] Ultimately, the complete reward function can be obtained:

[0137]

[0138] Rev Q (G,T,T Real Let q represent the reward function of the value network. h The value network evaluates the h-th trajectory point of the policy; r represents the length of the policy network's output sequence; T h This represents the result of the h-th trajectory point in the policy network sequence;

[0139] After obtaining the above value function, the network parameters of the model can be updated as follows:

[0140]

[0141] Where γ is the learning rate of the value network, and Δ(ω) represents the error between the calculated target value and the network's calculation result.

[0142] In step S7, when generating the trajectory, the target policy network is first used to calculate the current optimal policy, and the calculation formula is as follows:

[0143] T * =T target (I,θ target )

[0144] Among them, T * Let T represent the current optimal sequence. target Let I represent the target policy network, and θ be the input to the target policy network. target These are the parameters of the target policy network;

[0145] The decision outputs of the target policy network are evaluated using target value network 1 and target value network 2:

[0146] q t1 =Q t1 (ω t1 ,G,T * ,T Real )

[0147] q t2 =Q t2 (ω t2 ,G,T * ,T Real )

[0148] Where, q t1 For the evaluation sequence output of target value network 1, q t2 T is the output of the evaluation sequence of the target value network 2. Real G represents real-time flight trajectory data, and Q represents the feature map. t1 For the target value network 1, Q t2 For the target value network 2, ω t1 Let ω be the parameter of the target value network 1. t2 The parameters for the target value network 2;

[0149] The evaluations of the policy network output by value network 1 and value network 2 are recalculated, and the equation is as follows:

[0150] q1=Q1(ω1,G,T * ,T Real )

[0151] q2=Q2(ω2,G,T * ,T Real )

[0152] Where q1 is the evaluation sequence output of value network 1, q2 is the evaluation sequence output of value network 2, Q1 is value network 1, Q2 is value network 2, ω1 is the parameter of value network 1, and ω2 is the parameter of value network 2.

[0153] Therefore, the TD objective can be calculated as follows:

[0154]

[0155] Wherein, min(q) t1 ,q t2 ) indicates taking vector q t1 ,q t2 The minimum value of each element at each position in the array. This means taking the first to the i-th elements of the target policy network to form a new vector, i.e.:

[0156]

[0157] in, T represents * The i-th element;

[0158] Then the estimation error is calculated, and the equation is:

[0159] ΔDT1 =q1-TD

[0160] Δ DT2 =q2-TD

[0161] Where, Δ DT1 Δ represents the estimation error of value network 1. DT2 This represents the estimation error of value network 2;

[0162] This completes the update of the value network, as follows:

[0163]

[0164]

[0165] Where γ1 represents the learning rate of value network 1 and γ2 represents the learning rate of value network 2;

[0166] After completing the value network update described above, the decision network parameters are updated again, as follows:

[0167]

[0168] Where θ0 represents the parameters of the decision network, and η0 represents the learning rate of the decision network;

[0169] For the target value network 1, target value network 2, and target policy network, parameter iteration is completed through delayed updates, and the equation for the delayed updates is as follows:

[0170] θ target ←τ target θ target +(1-τ target )θ0

[0171] ω t1 ←τ t1 ω t1 +(1-τ t1 )ω1

[0172] ω t12 ←τ t2 ω t2 +(1-τ t2 )ω2

[0173] Where, τ target The updated parameters of the target policy network are represented by τ. target ∈(0,1), τ t1 This represents the updated parameters of target value network 1, ranging from τ. t1 ∈(0,1), τ t2 This represents the update parameters of the target value network 2, ranging from τ. t2∈(0,1);

[0174] Through the above training and iteration, the final path output can be obtained:

[0175]

[0176] Among them, T opt Indicates the optimal path. θ represents the target decision network after the update iteration is completed. * This represents the optimal parameters required to generate the above optimal path.

[0177] The technical effects achieved by this invention are as follows:

[0178] This invention rapidly extracts common feature points and feature point sets between frames to determine the correlation graph of feature points and feature sets. Finally, reinforcement learning is used to calculate the correspondence and position of the graph to obtain a relatively accurate feature point relationship. The DQN method is used to generate the trajectory, offering the advantages of model-free control. It provides better adaptability for aircraft with varying maneuverability, reducing performance losses caused by mismatch between the controller and planning scheme. The trajectory solving performance is better than traditional methods, with superior search capabilities. For path planning requiring mapping, and considering the sensitivity of planning problems to mapping, a mapping scheme that eliminates the need for precise numerical calculations is designed, improving overall performance by reducing computational overhead. Furthermore, the use of reinforcement learning allows for a wider search area and better search results compared to traditional algorithms. The overall method is optimized for specific tasks, offering better performance than methods involving precise calculation and iteration. By reducing the computational overhead of environmental mapping, only necessary connections and spatial features are calculated, and the relevant aircraft trajectory planning is completed in conjunction with the planning algorithm. Attached Figure Description

[0179] Figure 1 This is a flowchart of the technical solution in this invention;

[0180] Figure 2 This is a schematic diagram illustrating the principle of flight trajectory generation in this invention;

[0181] Figure 3 This is a flowchart of the trajectory policy network based on the Transformers model in this invention;

[0182] Figure 4 This is a flowchart of the trajectory value network based on the Transformers model in this invention;

[0183] Figure 5 This is a schematic diagram of the TD3 flight trajectory generation method based on the Transformers model in this invention. Detailed Implementation

[0184] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0185] A mapping and planning method for quadrotor UAVs based on deep reinforcement learning includes the following steps:

[0186] Please see the appendix Figure 1 As shown:

[0187] Step 1: Feature extraction and filtering, and creation of feature maps;

[0188] It is known that the camera can acquire n frames of data after time k:

[0189] F = [F k+1 ,…,F k+ n], k=1,2,3,…

[0190] For ease of description, we define the variable i to represent the frame number, that is:

[0191] i = k+1,…,k+n

[0192] For the single-frame data F mentioned above i Feature points can be collected:

[0193]

[0194] In the formula, S i S is the feature point vector of the i-th frame of data; i,m This represents the m-th feature point of the i-th frame of data; m i This represents the number of feature points in the i-th frame of data.

[0195] For ease of subsequent representation, we define the feature point index j, that is:

[0196] j = 1, ..., m

[0197] In the formula, m represents the number of feature points in the current frame, which changes with the frame number i. This method ensures that there is no comparison between previous and next frames, that is, when the formula exists j, the value represented by m will not change.

[0198] For a feature point, define the set of feature points to which this feature point belongs:

[0199] map(S i,j )={S i,j}

[0200] In the formula, map(S)i,j ) represents the feature point S i,j A function mapping to the set of feature points, where the function maps to the set to which the feature points belong, i.e.:

[0201] S i,j ∈map(S i,j )

[0202] Simultaneously define a set:

[0203] U i ={{S i,1},…,{S i,m}}

[0204]

[0205] In the formula, U i Let E represent the set of all possible feature points contained in the i-th frame, where, under initial conditions, any feature point represents a single set; i This represents the association between feature sets in the i-th frame image. Initially, this method defines the non-existent relationship between sets, i.e., an empty set.

[0206] To find candidate points with dense feature points, we first calculate the minimum interval of feature points in the frame:

[0207]

[0208] In the formula, S i,short S represents the minimum distance between feature points in the i-th frame of data; i,a and S i,a represents the image positions of the a-th and b-th feature points of the i-th frame data, respectively; m represents the number of feature points of the i-th frame data.

[0209] Then for each feature point S i Then calculate:

[0210]

[0211] In the formula, η represents the multiple of the feature point distance, which is a constant; m represents the number of feature points in the i-th frame of data. If the above inequality holds, the feature points are merged into a single set:

[0212] U i,a,b =map(S i,a )∪map(S i,b ),a∈{1,…,m},b∈{1,…,m},a>b

[0213] In the formula, U i,a,bThis represents the union of the sets of feature points containing the a-th and b-th feature points in the i-th frame of data, i.e., merging two feature points into one set; m represents the number of feature points in the i-th frame of data; S i,a and S i,a These represent the image positions of the a-th and b-th feature points in the i-th frame of data, respectively.

[0214] After completing the above modifications, modify the feature point set result of the i-th frame:

[0215] E i ←(E i -{<u1,u2> |u1∈map(S i,a ),u2∈map(S i,b )})

[0216] ∪{ <map(S i,a ),map(S i,b )>}

[0217] U i ←(U i -map(S i,a )-map(S i,b ))∪U i,a,b

[0218] map(S i,a )=U i,a,b

[0219] map(S i,b )=U i,a,b

[0220] In the above formula, u1 and u2 represent two sets of feature points.<u1,u2> This indicates that two feature point sets, u1 and u2, are related.

[0221] After completing all the above results, for U i Remove feature points from sets with fewer internal feature points, and then process E. i To improve the accuracy and stability of detection, synchronized updates are performed.

[0222] U i ={u|u∈U i ,|u|>L}

[0223] E i ={<u1,u2> |u1,u2∈U i ,<u1,u2> ∈E i}

[0224] Completing the above operations means that the initial screening of the i-th frame of data has been completed.

[0225] To ensure the uniqueness of the feature point set and fast retrieval, a convolutional neural network is used to locally encode the feature point set. Specifically, for all feature points, a CNN model is used for hashing, ensuring that the more similar the arrangement of the feature points, the smaller the computational result, i.e., satisfying the following:

[0226] CrossEntropy (CNN (U l ),CNN(U r ))→∞,U l ≠U r

[0227] In the above formula, CNN() represents the result calculated by the CNN model; U l and U r These represent two sets of feature points; CrossEntropy() is the cross-entropy function used to compare the differences in the encoding results of CNN models.

[0228] Specifically, the structure, size, and optimization method of this CNN model have little impact on the final result; the loss function that has a greater impact on the result is as follows:

[0229]

[0230] In the above formula, U is the loss function, and the right side of the equation describes its calculation method; c This represents the set of feature points being evaluated, which is derived from any known set of features. CNN(U c The current value of ); O1 represents all frames that appear, excluding U. c The total number of feature points in all frames; O2 represents the total number of U points in all frames. c Number of occurrences; U k1 Indicates different from U c The set of feature points; Represents and U c The same set of feature points. It should be noted that "same" here refers to a set of feature points describing the same physical location. The feature points of the two are the same or very similar, hence the term "same set".

[0231] For the above calculation results, statistical F k ,…,F k+n If, after statistical analysis, there are encoded results that consistently appear in every frame, then this is defined as a stable scene point, i.e.:

[0232] U st,k ={CNN(U i,l )|CNN(U i,l )∈CNN(U h ); l∈{1,…,mui},h=k,…,k+n}

[0233] E st,k ={<u1,u2> |CNN(u1),CNN(u2)∈U st,k ,<u1,u2> ∈E i}

[0234] W st,k ={{e,O}|e∈E st,k}

[0235] In the above formula, U st,k E represents the set of feature points that appear stably in n frames starting from frame k; st,k W represents the association of a set of feature points that appear stably in n frames starting from frame k; st,k The sum E represents the spatial relationship of the set of feature points. st,k The associations are one-to-one; mu i U represents the number of feature sets in the i-th frame; i,l This represents the set of the l-th feature points of the i-th frame of data;<u1,u2> This represents the association of a set of feature points that appear stably in n frames starting from frame k; O represents the zero vector; e represents E. st,k Inner edge; U h Represents the set of all feature points in the h-th frame; CNN(U h ) indicates that for U h Calculate all elements within the range, that is:

[0236]

[0237] In the above formula, m h denoted as the number of feature points in the h-th frame of data.

[0238] Finally, we can obtain the bidirectional weighted graph result of the feature point set of n frames starting from frame k:

[0239] G k =(U st,k E st,k W st,k U k+n )

[0240] In the above formula, U k+n This represents the set of all feature points in the last frame processed in the k-th frame.

[0241] For ease of use, G is defined. k Let U be the feature map of the k-th frame; define U sk,k For feature map G k The node; define E st,k For feature map G k The edge.

[0242] Step 2: Calculate the feature map position based on value iteration, and verify it through geometry and iterate using the TD algorithm;

[0243] Based on the feature map of the k-th frame mentioned above, the positional relationships between scene points are calculated and corrected.

[0244] To facilitate map construction, any set of feature points within the current frame is selected as the initial position:

[0245] U origin ∈U st,k U st,k ∈G k

[0246] Then, the following feature map is constructed:

[0247]

[0248] In the above formula, U represents the set of all explored feature points, i.e., the nodes of graph G; E represents the associations of the explored feature point sets, i.e., the edges of graph G; W corresponds to the relative positions of the elements in E, i.e., the values ​​of graph G; U last This represents the set of all feature points in the last frame of data.

[0249] This method uses a dual DQN approach to iteratively update the value of the feature point set, defining specific model details. For the network described above, its model input is...<a,b> ∈E st,k That is, the association of the set of feature points in the feature map, also known as the edge of the graph; the action of the network is represented as: A = [a x ,a y ,a z ] represents the relative position between the two endpoint feature point sets of an edge.

[0250] For edge e = a ,u b >, by u a to u b The process, represented by action A, yields a benefit derived from the sum of errors corrected at both ends and from other already corrected results. This benefit is expressed as:

[0251]

[0252] In the above formula, sigmoid() represents the Sigmoid function, used to normalize the value of the loss function; MSE() represents the mean squared error function; K is the camera intrinsic parameter; A is the optimal action for edge e; A b,j For the set of feature points u b and u j ​The optimal action; d(u) represents the number of connections between the feature point set u and the feature point set within U, i.e., u i ,u j ∈U.

[0253] The following was calculated during the value iteration process:

[0254]

[0255] In the above formula, Describe the optimal action for edge e; A e R(e,A) represents the best action for edge e in the previous iteration; R(e,A) represents the benefit of using action A for edge e.

[0256] After this, update the policy A of edge e using the greedy policy. e :

[0257]

[0258] Search for and update the set of feature points connected to the existing graph, i.e.:

[0259] E′ G ={ k,a ,S k,b >| k,a ,S k,b >∈E st,k CNN(S) k,b )∈CNN(U)}

[0260] U′ G ={S k,a | k,a ,S k,b >∈E st,k CNN(S) k,b )∈CNN(U)}

[0261] Then, through iteration, the corresponding optimal action is obtained:

[0262]

[0263] Last updated G:

[0264] G←G∪(U′ G ,E′ G W G U last )

[0265] In the formula U' last This represents the data from the last frame.

[0266] Step 3: State estimation;

[0267] ​​​Considering a quadcopter drone controlled by this method, the current location of the aircraft can be determined based on traditional positioning technology or with hardware assistance:

[0268] X = [xyz]

[0269] In the above equations, x, y, and z represent the body position relative to the initial set of feature points U, respectively. origin The distances along the X, Y, and Z axes.

[0270] At the same time, for each feature set, define a function mapping:

[0271] P(u f = [x0 y0 z0]

[0272] The result of this function mapping is solved using the following iterative scheme:

[0273] P(v i )exist

[0274] In the above formula, P(u) represents node u relative to U. origin The relative distance; w(u,v) i ) represents nodes u and v i The optimal action obtained from previous iterations; h represents the number of associations between node u and other nodes.

[0275] This method uses policy learning for training and inference. It also uses the Transformers model as a prototype to design a policy generation model to meet the needs of flight sequence generation and position encoding matching.

[0276] Please see the appendix Figure 2 As shown, this leads to the basic principle of trajectory generation;

[0277] This method uses the actor-critic method to iteratively generate trajectories. Considering the overestimation and parameter offset issues in practical applications, the TD3 method is further used to complete the generation of all trajectories.

[0278] Please see the appendix Figure 3 As shown, it is used to generate feasible trajectories;

[0279] Step 4: Generate the policy network model for the trajectory;

[0280] For the above strategy model, there is a parameter θ, and an encoder input exists:

[0281] I = [G,A o A d ]

[0282] In the above formula, I is the system input; G is the feature map; A O Starting position; A d This is the endpoint of the operation.

[0283] If initial and final poses exist, then the input is:

[0284] I = [G,A o A d Alt o Alt d ]

[0285] In the above formula, Alt o The starting point's posture; Alt d The desired attitude at the end of the run.

[0286] Finally, the iterative data can be obtained at the decoder:

[0287] T = T f (I,θ)=[A o ,T0,…,T r A d Alt o Alt d ]

[0288] In the above formula, T0,…,T r is the generated location information; r is the length of the generated trajectory.

[0289] While generating the sequence, the parameters θ of the above model need to be adjusted and updated to ensure that the model sequence conforms to the value judgment of the value network. The adjustment method is as follows:

[0290]

[0291] In the above formula, η represents the learning rate of the policy network; Q(G,T) represents the value network of this method, which accepts inputs including feature map G and policy network output T; θ is the parameter of the generator network; and T is the policy with parameter θ and input I.

[0292] Step 5: Evaluate the value network model of the trajectory

[0293] For the evaluation networks already mentioned above, use the appendix... Figure 4 The results shown are generated based on the Transformers model;

[0294] The value network model has a parameter ω, which is generated as follows: First, the feature map and trajectory sequence are used as encoder input to obtain the data at the decoder and the network output. Then, the parameters of the value network are updated to obtain the complete reward function. The specific steps are as follows:

[0295] Using feature map G and trajectory sequence T as encoder input, the data at the decoder can be obtained. In particular, if the network is run or trained online, the actual flight trajectory and flight attitude can be obtained.

[0296] Tr real =[T rl,0 ,…,T rl,r0 ]

[0297] Alt real =[Alt rl,0 ,…,Alt rl,r0 ]

[0298]

[0299] In the above formula, T Real For the final actual trajectory; Tr real For actual spatial location; Alt real r0 represents the actual running posture; r0 represents the sequence length of the actual trajectory. This method will give the maximum length of the sequence to limit the model size.

[0300] Then you can obtain the network output results:

[0301] q=Q(ω,G,T,T Real )=[q th,0 ,…,q th,r ]

[0302] In the above formula, q th,i This represents the value estimate of the i-th sequence result of the policy network by the value network.

[0303] In cases where no actual flight path exists, the calculation can be performed as follows:

[0304]

[0305] In the above formula, O is the zero vector; q n,i This represents the value estimate of the policy network from the starting point to the i-th sequence result in the absence of actual data.

[0306] The output of the above model needs to be calculated using the reward function and the parameters of the value network need to be updated. Considering the special properties of this model, the following four types of losses are defined:

[0307]

[0308] In the above equation: Loss illu (T h(G) represents the collision loss of the flight trajectory. This loss increases when the aircraft is too close to the searched set of feature points. In the collision loss calculation, d represents the d closest nodes to the aircraft, or nodes can also be called the set of feature points. Loss E (T h () represents the energy loss during flight; Represents the trajectory point T of the sequence h Find the result of the hd-th order differential; Loss reach (T h ,T real This represents the loss in the flight trajectory's realization. This loss increases as the degree of alignment between the flight trajectory and the actual trajectory decreases.

[0309] This represents the sampling point T used to find a suitable actual trajectory. real,i Make it closer to the trajectory point T generated by the strategy. h Once the aforementioned sampling points are found, the error of the hd-order derivative can be analyzed and summarized into the final error; Loss alt (T,T real ) represents the attitude loss. Considering that the initial state model's misjudgment of attitude can cause huge realization losses, only the terminal loss is calculated; μ represents the weight of the attitude loss.

[0310] Ultimately, the complete reward function can be obtained:

[0311]

[0312] In the above formula, Rev Q (G,T,T Real ) represents the reward function of the value network; q h The value network evaluates the h-th trajectory point of the policy; r represents the length of the policy network's output sequence; T h This represents the result of the h-th trajectory point in the policy network sequence.

[0313] Once the above value function is obtained, the network parameters of the model can be updated.

[0314]

[0315] In the above formula, γ is the learning rate of the value network; Δ(ω) represents the error between the calculated target value and the network calculation result.

[0316] Based on the above trajectory generation and value model, using the attached... Figure 5 The TD3 method in the text completes the final trajectory generation method, that is... Figure 5 The target value network 1, i.e., Q, is included. t1Target value network 2, i.e. Q t2 Value network 1, i.e., Q t1 And value network 2, i.e. Q t2 There are a total of 4 value networks and a target policy network, namely T. target The policy network, T0, consists of two policy networks.

[0317] This method first uses the target policy network to calculate the current optimal policy when generating the trajectory:

[0318] T * =T target (I,θ target )

[0319] In the above formula, T * T represents the current optimal sequence; target Represents the target policy network; I is the input to the target policy network; θ target These are the parameters of the target policy network.

[0320] Then, the decision output of the target policy network is evaluated using target value network 1 and target value network 2:

[0321] q t1 =Q t1 (ω t1 ,G,T * ,T Real )

[0322] q t2 =Q t2 (ω t2 ,G,T * ,T Real )

[0323] In the above formula, q t1 For the evaluation sequence output of target value network 1, q t2 The output of the evaluation sequence for the target value network 2; T Real G represents real-time flight trajectory data; G is the feature map; Q represents... t1 For the target value network 1; Q t2 For the target value network 2; ω t1 ω represents the parameters of the target value network 1. t2 The parameters are for the target value network 2.

[0324] Then, the evaluations of the policy network output by value network 1 and value network 2 are calculated again:

[0325] q1=Q1(ω1,G,T * ,T Real )

[0326] q2=Q2(ω2,G,T * ,T Real )

[0327] In the above formula, q1 is the evaluation sequence output of value network 1, q2 is the evaluation sequence output of value network 2; Q1 is value network 1; Q2 is value network 2; ω1 is the parameter of value network 1; ω2 is the parameter of value network 2.

[0328] Based on the existing conditions, the TD objective can be calculated as follows:

[0329]

[0330] In the above formula, min(q) t1 ,q t2 ) indicates taking vector a t1 ,q t2 The minimum value of each element at each position in the array; This means taking the first to the i-th elements of the target policy network to form a new vector, i.e.:

[0331]

[0332] In the above formula, T represents * The i-th element.

[0333] Calculate the estimation error:

[0334] Δ DT1 =q1-TD

[0335] Δ DT2 =q2-TD

[0336] In the above formula, Δ DT1 Δ represents the estimation error of value network 1. DT2 This represents the estimation error of value network 2.

[0337] This completes the update of the value network:

[0338]

[0339]

[0340] In the above formula, γ1 represents the learning rate of value network 1; γ2 represents the learning rate of value network 2.

[0341] After completing the value network update described above, update the parameters of the decision network as follows:

[0342]

[0343] In the above formula, θ0 represents the parameters of the decision network; η0 represents the learning rate of the decision network.

[0344] The above steps are passed Figure 5 The computation graph has been updated.

[0345] for Figure 5 The target value network 1, 2, and target policy network in the model iterate their parameters by delaying updates.

[0346] θ target ←τ target θ target +(1-τ target )θ0

[0347] ω t1 ←τ t1 ω t1 +(1-τ t1 )ω1

[0348] ω t12 ←τ t2 ω t2 +(1-τ t2 )ω2

[0349] Steps four and five of the above formula correspond to... Figure 5 The data update graph in the middle, where: τ target The updated parameters of the target policy network are represented by τ. target ∈(0,1); τ t1 This represents the updated parameters of target value network 1, ranging from τ. t1 ∈(0,1); τ t2 This represents the update parameters of the target value network 2, ranging from τ. t2 ∈(0,1).

[0350] Through the above training and iteration, the final path output can be obtained:

[0351]

[0352] In the above formula, T opt Indicates the optimal path; θ represents the target decision network after the update iteration is completed; * This represents the optimal parameters required to generate the above optimal path.

[0353] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A mapping and planning method for quadrotor UAVs based on deep reinforcement learning, characterized in that: Includes the following steps: S1: Feature extraction and filtering; S2: Create a feature map; S3: Calculate the feature map position based on value iteration, and verify it through geometric verification and iteratively using the TD algorithm; S4: State estimation; S5: Policy network model for generating trajectories; S6: Value network model for evaluating trajectories; S7: Generate the target's flight trajectory; Specifically, S5 and S6 iterate with each other and complete the trajectory generation in S7.

2. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: In S1, the camera acquires n frames of data after time k, and the formula for calculating the number of frames F is: F = [F k+1 ,..., F k+n ], k = 1, 2, 3,... Define variable i to represent the frame number, that is: i = k+1,...,k+n For single frame data F i The calculation formula of the feature point acquisition is: wherein S i is a feature point vector of the i-th frame data, S i，m denotes the m-th feature point of the i-th frame data, m i denotes the number of feature points of the i-th frame data; Define feature point j, that is: j = 1, ..., m Where m represents the number of feature points in the current frame, which changes with the frame number i. However, when the formula contains feature point j, the value represented by m will not change. And the set of feature points to which feature point j belongs is: map(S i，j ) = {S i，j} wherein map(S i，j ) denotes a function mapping of the feature point S i，j to the set to which the feature point belongs, i.e. S i，j ∈map(S i，j ) Furthermore, the set is defined as: U i = {{S i，1}..., {S i，m}} Among them, U i Let E represent the set of all feature points contained in the i-th frame. Under initial conditions, any feature point is a single set. i This represents the association between feature sets in the i-th frame image; To find candidate points with dense feature points, we first need to calculate the minimum interval of feature points in the frame. The formula for this calculation is as follows: Among them, S i，short S represents the minimum distance between feature points in the i-th frame of data. i，a and S i,a Let a and b represent the image positions of the a-th and b-th feature points of the i-th frame of data, respectively, and m represent the number of feature points of the i-th frame of data; For each feature point S i The formula for calculating it is: Where η represents the multiple of the feature point distance, which is a constant in the formula, and m represents the number of feature points in the i-th frame of data; The feature points S i Merge into a set: U i，a，b = map(S i，a ) U map(S i，b ), a e {1,..., m}, b e {1,..., m}, a > b Among them, U i,a,b This represents the union of the sets of feature points containing the a-th and b-th feature points in the i-th frame of data; that is, merging two feature points into one set. m represents the number of feature points in the i-th frame of data. i,a and S i,a These represent the image positions of the a-th and b-th feature points in the i-th frame of data, respectively. After the above modifications are completed, the feature point set result of the i-th frame is modified, and the formula is as follows: E i ←(E i -{<u1,u2> |u1∈map(S i，a ),u2∈map(S i，b )}) ∪ { <map(S i，a ), map(S i，b )>} U i ←(U i -map(S i，a )-map(S i，b ))∪U i，a，b map(S i,a ))=U i，a，b map(S i，b )) = U i，a,b Where u1 and u2 represent two sets of feature points,<u1,u2> This indicates that two feature point sets u1 and u2 are related; After completing the above steps, for U i Remove feature points with fewer internal feature points, and then process E. i Synchronous updates are performed using the following formula: U i = {u | u ∈ U i , [u | > L} E i = { <u1, u2> | u1, u2 G U i , <u1, u2> G E i} After completing the above steps, the initial screening of the i-th frame of data is complete. We use a convolutional neural network to encode the feature point set locally, and then use a CNN model to hash all the feature points, thus satisfying the following: CrossEntropy(CNN(U l ), CNN(U r ))→∞, U l ≠ U r wherein CNN() is a CNN model calculation result, U l and U r respectively represent two sets of feature point sets; CrossEntropy() is a cross-entropy function; The structure, size, and optimization methods of a CNN model have little impact on the final result; The loss function that has a significant impact on the results is: U i ←(U i -map(S i，a )-map(S i，b ))∪U i，a，b map(S i，a ) = U i，a，b map(S i，b ) = U i，a，b in, Let U be the loss function. c This represents the set of feature points being evaluated. CNN(U c The current value of ), O1 represents all frames that do not contain U. c The total number of feature points in all frames, O2 represents the total number of feature points in all frames. c Number of occurrences, U k1 Indicates different from U c The set of feature points Represents and U c The same set of feature points; Statistical F k ..., F k+n The final output of n frames, after statistics, the coding results meet the stable appearance of each frame, which is defined as a stable scene point, that is: U st，k = {CNN(U i，l ) | CNN(U i，l ) e CNN(U h ) ; I e {1,..., mu i}, h = k,..., k + n} E st，k = <u1, u2> | CNN(u1), CNN(u2) e U st，k , <u1, u2> e E i} W st，k = {{e, O} | e E st，k} Among them, U st，k E represents the set of feature points that appear stably in n frames starting from frame k. st，k W represents the association of a set of feature points that appear stably in n frames starting from frame k. st，k The sum E represents the spatial relationship of the set of feature points. st，k The associations are one-to-one, mu i U represents the number of feature sets in the i-th frame. i，l This represents the set of the l-th feature points of the i-th frame of data.<u1,u2> This represents the association of a set of feature points that stably appear in n frames starting from frame k, where O denotes the zero vector; e denotes E. st，k The inner edge, U h Represents the set of all feature points in the h-th frame, CNN(U h ) indicates that for U h Calculate all elements within the range, that is: Where, m h The number of feature points in the h-th frame of data; The final bidirectional weighted map result of the feature point set starting from frame k for n frames is obtained, and its formula is: G k = (U st，k , E st，k , W st，k , U k+n ) Among them, U k+n Let G represent the set of all feature points of the last frame processed in the k-th frame. k For the feature map of the k-th frame, define U st,k For feature map G k The node is defined as E. st,k For feature map G k The edge.

3. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: In step S2, based on the feature map of the k-th frame in S1, the positional relationships between scene points are calculated and corrected. Any set of feature points within the current frame is selected as the initial position, and the calculation formula is as follows: U origin ∈U st,k ;U st,k ∈G k The feature map is then constructed using the following formula: Where U represents the set of all explored feature points, E represents the association of the discovered feature point sets, and W corresponds to the relative positions of the elements in E. last This represents the set of all feature points in the last frame of data.

4. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: In S3, the dual DQN method is used to iteratively update the value of the feature point set, defining specific model details. For the above network, its model input is: < a, b > E st,k The state is the association of feature point sets in a feature map, also known as graph edges. The relative positions between the two endpoints of an edge are represented by network actions: A[a x ,a y ,a z ] For edge e: by u a to u b The process, using action A to represent relative positional displacement, yields a benefit derived from the sum of errors corrected at both ends and from other already corrected results. The calculation formula is as follows: Where sigmoid() represents the Sigmoid function, MSE() represents the mean squared error function, K is the camera intrinsic parameter, and A is the optimal action for edge e. b,j For the set of feature points u b and u j The optimal action, d(u), represents the number of connections between the feature point set u and the feature point set within U, i.e.: u i ,u j ∈U The following was calculated during the value iteration process: in, Let A represent the optimal action for edge e. e Let R(e,A) represent the best action for edge e in the previous iteration, and let R(e,A) represent the benefit of using action A for edge e. After that the strategy A of edge e is updated using the greedy strategy e , for: Search for and update the set of feature points connected to the existing graph, i.e.: E′ G = { <S k，a , S k，b > | <S k，a , S k，b > E st，k , CNN(S k，b ) E CNN(U)} U′ G = {S k，a | <S k，a , S k，b > ∈ E st，k , CNN(S k，b ) ∈ CNN(U)} The corresponding optimal action is obtained through iteration: Last updated G: G <- G U (U' G , E' G , W' G , U' last ) where U' = U - U0 last represents the data of the last frame.

5. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: The state estimation in S4 is performed to determine the current position of the aircraft using the following method: X = [xyz] Wherein, the x, y, z of the equation respectively represent the distance of the body position relative to the initial feature point set U origin X, Y, Z axis At the same time, for each feature set, define a function mapping: P(u f ) = [x0 y0 z0] The result of this function mapping is solved using an iterative scheme based on the following formula: exist Where P(u) represents node u relative to U origin The relative distance, w(u, v) i ) represents nodes u and v i The optimal action obtained from previous iterations, where h represents the number of associations between node u and other nodes.

6. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: The policy network model for generating trajectories in S5 includes the following methods: first, training and inference are performed using policy learning methods; simultaneously, a Transformers model is used as a prototype to generate the policy network model, wherein the policy network model has parameters θ and an encoder input: I = [G, A o , A d ] where I is the system input, G is the feature map, A O is the departure location, A d is the end of the run; The iterative data can be obtained at the decoder end as follows: T = T f (I, θ) = [A o , T0,..., T r , A d , Alt o , Alt d ] where T0,..., T r is the generated position information, and r is the length of the generated trajectory. While generating the sequence, the parameters θ of the above model are adjusted and updated. The adjustment method is as follows: Where η represents the learning rate of the policy network, Q(G,T) represents the value network in S6, which accepts inputs including feature map G and policy network output T, θ is the parameter of the generator network, and T is the policy with parameter θ and input I.

7. The quadcopter UAV mapping and planning method according to claim 1, characterized in that: The value network model in S6 is generated based on the Transformers model and has a parameter ω. Its generation method is as follows: First, feature maps and trajectory sequences are used as encoder input to obtain data at the decoder end and network output results. Then, the parameters of the value network are updated to obtain the complete reward function. The specific steps are as follows: Using the feature map G and trajectory sequence T as encoder input, the data at the decoder end, as well as the actual flight trajectory and flight attitude, can be obtained as follows: Tr reat = [T rl，0 ,..., T rl，r0 ] Alt real = [Alt rl，0 ,..., Al tl，r0 ] wherein T Real is the final actual trajectory, Tr real is the actual spatial position, Alt real is the actual running pose, and r0 is the sequence length of the actual trajectory. The network output results are obtained as follows: q=Q(ω,G,T,T Real )=[q th，0 ,...,q th，r ] where q th，i represents the value estimate of the i-th sequence result of the policy network by the value network; For cases where no actual flight path exists, the following formula can be used for calculation: where O is a zero vector, q n，i denotes the value estimate of the policy network from the start point to the i-th sequence result without actual data. The output of the above model needs to be calculated using the reward function and the parameters of the value network need to be updated. The following formula defines four types of losses: Among them, Loss illu (T h G) represents the collision loss of the flight trajectory, where d in the collision loss calculation represents the d nodes closest to the aircraft. Loss E (T h This represents the energy loss during flight. Represents the trajectory point T of the sequence h Find the result of the hd-th order differential, Loss reach (T h ,T real ) indicates the loss of flight trajectory realization; This represents the sampling point T used to find a suitable actual trajectory. real,i Make it closer to the trajectory point T generated by the strategy. h Loss alt (T,T real ) represents the pose loss, and μ represents the weight of the pose loss; Ultimately, the complete reward function can be obtained: Rev Q (G,T,T Real Let q represent the reward function of the value network. h The value network evaluates the h-th trajectory point of the policy; r represents the length of the policy network's output sequence; T h This represents the result of the h-th trajectory point in the policy network sequence; After obtaining the above value function, the network parameters of the model can be updated as follows: Where γ is the learning rate of the value network, and Δ(ω) represents the error between the calculated target value and the network's calculation result.

8. The planning method according to claim 1, characterized in that: In step S7, the TD3 method is used to generate all trajectories during trajectory generation. The TD3 method includes a target value network 1, i.e., Q. t1 Target value network 2, i.e. Q t2 Value network 1, i.e., Q t1 And value network 2, i.e. Q t2 There are a total of 4 value networks and a target policy network, namely T. target The policy network, T0, consists of two policy networks. When using the TD3 method for trajectory generation, the target policy network is first used to calculate the current optimal policy. The calculation formula is as follows: T * = T target ((I, θ target ) Among them, T * Let T represent the current optimal sequence. target Let I represent the target policy network, and θ be the input to the target policy network. target These are the parameters of the target policy network; The decision outputs of the target policy network are evaluated using target value network 1 and target value network 2: q t1 = Q t1 (ω t1 , G, T * , T Real ) q t2 =Q t2 (ω t2 ,G,T * ,T Real ) Where, q t1 For the evaluation sequence output of target value network 1, q t2 T is the output of the evaluation sequence of the target value network 2. Real G represents real-time flight trajectory data, and Q represents the feature map. t1 For the target value network 1, Q t2 For the target value network 2, ω t1 Let ω be the parameter of the target value network 1. t2 The parameters for the target value network 2; The evaluations of the policy network output by value network 1 and value network 2 are recalculated, and the equation is as follows: q1=Q1(ω1,G,T * ,T Real ) q2=Q2(ω2,G,T * ,T Real ) Where q1 is the evaluation sequence output of value network 1, q2 is the evaluation sequence output of value network 2, Q1 is value network 1, Q2 is value network 2, ω1 is the parameter of value network 1, and ω2 is the parameter of value network 2. Therefore, the TD objective can be calculated as follows: Wherein, min(q) t1 q t2 ) indicates taking vector q t1 q t2 The minimum value of each element at each position in the array. This means taking the first to the i-th elements of the target policy network to form a new vector, i.e.: in, T represents * The i-th element; Then the estimation error is calculated, and the equation is: Δ DT1 =q1-TD Δ DT2 =q2-TD Where, Δ DT1 Δ represents the estimation error of value network 1. DT2 This represents the estimation error of value network 2; This completes the update of the value network, as follows: Where γ1 represents the learning rate of value network 1 and γ2 represents the learning rate of value network 2; After completing the value network update described above, the decision network parameters are updated again, as follows: Where θ0 represents the parameters of the decision network, and η0 represents the learning rate of the decision network; For the target value network 1, target value network 2, and target policy network, parameter iteration is completed through delayed updates, and the equation for the delayed updates is as follows: i target ←t target i target +(1-t target )θ0 oh t1 ←t t1 oh t1 +(1-t t1 )ω1 oh t12 ←t t2 oh t2 +(1-t t2 )ω2 Where, τ target The updated parameters of the target policy network are represented by τ. target ∈(0,1),τ t1 This represents the updated parameters of target value network 1, ranging from τ. t1 ∈(0,1),τ t2 This represents the updated parameters of the target value network 2, ranging from τ. t2 ∈(0,1); Through the above training and iteration, the final path output can be obtained: Among them, T opt Indicates the optimal path. θ represents the target decision network after the update iteration is completed. * This represents the optimal parameters required to generate the above optimal path.

Images