Unmanned aerial vehicle reinforcement learning training method adaptive to multi-environment tasks

Abstract

This invention relates to a reinforcement learning training method for unmanned aerial vehicles (UAVs) adapted to multiple environmental tasks. First, the Bernoulli distribution parameter p is set, and the training environment set Λ is initialized. train Replay environment collection Λ seen The parameters are: global environment score S, global environment timestamp C, global round count c, UAV policy π, and experience pool. Next, random parameters d∈{0,1} are generated based on p and combined with the training environment set Λ. train To determine whether there is a non-interactive environment: if d = 0 and Λ train If there is a non-interactive environment, then random sampling Λ is performed. train The new environment in the middle serves as an interactive environment. i Otherwise, calculate the replay distribution P. replay Sampling replay distribution P replay Get the interactive environment i Then, the drone interacts with the environment. i Interact for one round and update l i Environmental score S i Environmental timestamp C i Finally, the policy π is updated by sampling from the experience pool and using any suitable reinforcement learning algorithm. Training UAVs using this method can effectively improve the generalization ability of the UAV policy to the environment and enhance the UAV's rapid adaptability in unknown environments.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and unmanned aerial vehicles (UAVs), specifically to a reinforcement learning training method for UAVs adapted to multi-environment tasks. Background Technology

[0002] Drones possess advantages such as low cost, high flexibility, and strong stealth, enabling them to perform multiple tasks in extreme environments. In recent years, thanks to the rapid development of artificial intelligence technology, drones have played an increasingly important role in transportation, reconnaissance, and extreme environment rescue. However, in practical use, drones typically face diverse and variable environments. The actual deployment environment of drones often changes with time and the progress of the mission, such as using drones for transportation in different seasons or performing rescue missions at various disaster sites such as earthquakes and fires. This environmental diversity places high demands on the adaptability and versatility of drones.

[0003] While existing UAV training algorithms can achieve good results in single environments, they suffer from weak generalization ability and a tendency to overfit to historical experience during training, making them unsuitable for multi-environment tasks. Furthermore, most algorithms train policies with poor flexibility and generalization, requiring relearning for deployment in new environments, which hinders large-scale deployment. Therefore, to ensure UAV performance in multi-environment tasks, a policy training method with high generalization ability and the ability to quickly adapt to new environments is needed. Summary of the Invention

[0004] The technical problem solved by this invention is to address the strong coupling between existing UAV training algorithms and the environment, overcome the shortcomings of poor generalization of UAV strategies under multi-environment tasks, which makes large-scale deployment difficult, and provide a reinforcement learning training method that is adaptable to multi-environment tasks. This method ensures that the trained strategy has strong generalization while avoiding the dilemma of UAVs having to learn from scratch when facing new environments, thereby improving the ability of UAV strategies to quickly adapt to the environment.

[0005] The technical solution of this invention is: a reinforcement learning training method for UAVs adapted to multiple environmental tasks, which not only ensures the strong generalization of the strategy, but also enables the UAV to quickly adapt when facing unknown environments, including:

[0006] Step S1: Set the Bernoulli distribution parameter p and initialize the training environment set Λ train Replay Environment Collection Λ seen Global environment score S, global environment timestamp C, global round count c, drone interaction strategy π, and experience pool

[0007] Step S2: Generate random parameters d = 0 or d = 1 based on the Bernoulli distribution parameter p;

[0008] If d = 0 and |Λ train \Λ seen |>0, from the training environment set Λ train Randomly sample environments from all uninterrupted environments. i As an interactive environment, and l i Add to replay environment set Λ seen In the middle; initialize the new environment l i Environmental score S i and environmental timestamp C i ;

[0009] If d = 1 or |Λ train \Λ seen | = 0, calculate the replay environment set Λ seen The replay distribution P replay According to the replay distribution P replay From the replay environment collection Λ seen Environmental samples obtained from the middle i As an interactive environment;

[0010] Step S3: Drone and Interactive Environment i After a complete round of interaction, update the interacted environment. i Environmental score S i Environmental timestamp C i and experience pool

[0011] Step S4: Adjust the experience pool Sampling is performed, and based on the empirical samples obtained from the sampling, the UAV interaction policy π is updated using a reinforcement learning algorithm. Steps S2-S4 are repeated until training is completed, resulting in a UAV policy that adapts to multiple environmental tasks.

[0012] The term "multi-environment" refers to a variety of different environments, such as when a drone performs its mission by flying through cities, forests, and plains to avoid obstacles.

[0013] Furthermore, in step S1: setting the Bernoulli distribution parameter p and initializing the training environment set Λ train Replay Environment Collection Λ seen Global environment score S, global environment timestamp C, global round count c, drone interaction strategy π, and experience pool The specific implementation is as follows:

[0014] S11: Set the Bernoulli distribution parameter p∈(0,1);

[0015] S12: Initialize the training environment set Λ according to the type of environment M. train ={l1,l2,...l i}, where li Let i∈[1,M] represent the i-th interaction environment, and initialize the set of replay environments. The global environment score S, the global environment timestamp C are empty arrays, and the global round count value c = 0;

[0016] S13: Initialize the drone interaction strategy π;

[0017] S14: Initialize the experience pool Empty set.

[0018] 3. The UAV reinforcement learning training method for adapting to multiple environmental tasks according to claim 2, characterized in that step S2 is specifically implemented as follows:

[0019] judge:

[0020] If d = 0 and |Λ train \Λ seen If |>0, proceed to steps S21-S23:

[0021] S21: From the training environment set Λ train Randomly sample a new interactive environment from the non-interactive environment:

[0022] l i ~P new (l|Λ train ,Λ seen ), where i = |S| + 1 is defined as the interactive environment l i The index value;

[0023] S22: Introduce a new interactive environment i Stored in the replay environment set Λ seen middle;

[0024] S23: Initialize l i Environmental score S i =0, Environment timestamp C i = 0 and save it to the global environment score S and global environment timestamp C; update the global round count value c = c + 1;

[0025] If d = 1 or |Λ train \Λ seen If | = 0, proceed to steps S24-S29:

[0026] S24: Based on the UAV interaction strategy π and the replay environment set Λ seen Calculate Λ based on the most recent interaction trajectory τ of all environments. seen Score S for each environment i As shown in equation (1):

[0027]

[0028] Where T is the total time steps of trajectory τ, and the hyperparameter γ∈(0,1) is the reward discount factor. It is a GAE expression, a general formula for policy gradient estimation in the field of reinforcement learning. The hyperparameter λ∈[0,1] controls the variance and bias balance of the GAE, and δ k The TD error for a time step of k is shown in equation (2):

[0029] δ k =r k +γV(s k+1 )-V(s k (2)

[0030] Where, r k V(s) is the instantaneous reward at time step k. k+1 ) and V(s k ) are states S k+1 and state S k State value;

[0031] S25: Use the priority function h(S i ) for environmental score S i That is, each element in the global environment score S is processed as shown in equation (3):

[0032] h(S i ) = 1 / rank(S i (3)

[0033] The global environment score S is an array that stores the replay environment set Λ seen Score S for all environments i Where rank(S) i ) is the environmental score S i The ranking in descending order of the global environment score S;

[0034] S26: Normalized priority function h(S i ), to obtain the environment l i The preferred distribution P S (l i |Λ seen S), as shown in equation (4):

[0035]

[0036] Where β∈[0,1] is the temperature parameter for adjusting the distribution;

[0037] S27: Obtain the environment l based on the global round count value c and the global environment timestamp C. i Older preferred distribution Pc (l i |Λ seen ,C,c), as shown in equation (5):

[0038]

[0039] S28: According to P S (l i |Λ seen ,S) and P c (l i |Λ seen ,C,c), computational playback environment l i The replay distribution P replay (l i As shown in equation (6):

[0040] P replay (l i )=(1-ρ)·P S (l i |Λ seen ,S)+ρ·P C (l i |Λ seen ,C,c) (6)

[0041] Wherein, ρ is a hyperparameter, and its value range is [0,1];

[0042] S29: Sampling Replay Distribution P replay To obtain the interactive environment i Update the global round count value c, incrementing c by 1.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] (1) This invention discloses a reinforcement learning training method for UAVs adapted to multiple environmental tasks. By calculating the learning potential of all environments in the replay environment set and using heuristic methods, the method determines the environments that the UAV needs to interact with first during the training phase. It then updates the interaction strategy based on sampled experience data, ultimately obtaining a UAV strategy capable of adapting to multiple environmental tasks. The strategy learned through this method exhibits strong generalization ability. Therefore, using this invention to train UAVs can effectively improve the generalization ability of UAV strategies to the environment and enhance the UAV's rapid adaptability in unknown environments.

[0045] (2) This invention proposes a UAV reinforcement learning training method that adapts to multiple environment tasks. When facing a new environment, simply put the new environment into the training environment set and follow the present invention for training. It can efficiently and quickly train a UAV interaction strategy that can adapt to the new environment task based on the original strategy.

[0046] (3) The UAV reinforcement learning training method disclosed in this invention is adaptable to multiple environmental tasks and can be combined with any UAV reinforcement learning algorithm. It has strong universality and is conducive to the large-scale deployment of UAV strategies in various complex environments. Attached Figure Description

[0047] Figure 1 This is a flowchart of a UAV reinforcement learning training method adapted to multiple environmental tasks according to the present invention;

[0048] Figure 2 This is a schematic diagram illustrating the implementation process of training the method of the present invention in a real UAV environment mission. Detailed Implementation

[0049] While existing UAV training algorithms can achieve good results in single environments, they suffer from weak generalization ability and a tendency to overfit to historical experience during training, making them unsuitable for multi-environment tasks. Furthermore, most algorithms produce policies with poor generalization, requiring relearning for deployment in new environments, which is detrimental to large-scale deployment. This invention provides a UAV reinforcement learning training method adapted to multi-environment tasks. This method ensures strong generalization of the trained policy while avoiding the predicament of UAVs learning from scratch when facing new environments, thus improving the UAV policy's ability to quickly adapt to different environments.

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0051] Example 1

[0052] like Figure 1 As shown in the figure, this invention provides a UAV reinforcement learning training method adapted to multiple environmental tasks. This method can be combined with any reinforcement learning algorithm for policy training. In this embodiment, the classic PPO reinforcement learning algorithm is used for demonstration. The specific steps are as follows:

[0053] Step S1: Set the Bernoulli distribution parameter p and initialize the training environment set Λ train Replay Environment Collection Λ seen Global environment score S, global environment timestamp C, global round count c, drone strategy π, and experience pool Implementation details are as follows:

[0054] S11: Set the Bernoulli distribution parameter p∈(0,1);

[0055] S12: Initialize the training environment set Λ according to the type of environment M. train ={l1,l2,...l M}, initialize the replay environment set The global environment score S, the global environment timestamp C are empty arrays, and the global round count value c = 0;

[0056] S13: Initialize the UAV policy π, taking the PPO reinforcement learning algorithm as an example, that is, initialize the Actor and Critical neural network parameters of the PPO algorithm;

[0057] S14: Initialize the experience pool Empty set.

[0058] Step S2: Generate random parameters d = 0 or d = 1 based on the Bernoulli distribution parameter p. Combine the result of d with the training environment set Λ train Are there any non-interactive environments? Sample the drone's interactive environment. i The implementation details are as follows:

[0059] judge:

[0060] If d = 0 and |Λ train \Λ seen If |>0, proceed to steps S21-S23:

[0061] S21: From the training environment set Λ train Randomly sample a new interactive environment from the non-interactive environment:

[0062] l i ~P new (l|Λ train ,Λ seen ), where i = |S| + 1 is defined as the interactive environment l i The index value;

[0063] S22: Introduce a new interactive environment i Stored in the replay environment set Λ seen middle;

[0064] S23: Initialize l i Environmental score S i =0, Environment timestamp C i = 0 and save it to the global environment score S and the global environment timestamp C; update the global round count value c = c + 1.

[0065] If d = 1 or |Λ train \Λ seen If | = 0, proceed to steps S24-S29:

[0066] S24: Based on the drone usage strategy π and replay environment set Λ seen Calculate Λ based on the most recent interaction trajectory τ of all environments. seen Score S for each environmenti As shown in equation (1):

[0067]

[0068] Where T is the total time steps of trajectory τ, and the hyperparameter γ∈(0,1) is the reward discount factor. It is a GAE expression, where the hyperparameter λ∈[0,1] controls the variance and bias balance of the GAE, and δ k The TD error for a time step of k is shown in equation (2):

[0069] δ k =r k +γV(s k+1 )-V(s k (2)

[0070] Where, r k V(s) is the instantaneous reward at time step k. k+1 ) and V(s k ) are states S k+1 and state S k The state value is estimated using a Critical network.

[0071] S25: Use the priority function h(S i ) for environmental score S i That is, each element in the global environment score S is processed as shown in equation (3):

[0072] h(S i ) = 1 / rank(S i (3)

[0073] The global environment score S is an array that stores the replay environment set Λ seen Score S for all environments i Where rank(S) i ) is the environmental score S i The ranking in descending order of the global environment score S;

[0074] S26: Normalized priority function h(S i ), thus obtaining the preferred distribution P S (l i |Λ seen, S), as shown in equation (4):

[0075]

[0076] Where β∈[0,1] is the temperature parameter for adjusting the distribution;

[0077] S27: Obtain the stale priority distribution P based on the global round count value c and the global environment timestamp C. c (l i |Λ seen ,C,c), as shown in equation (5):

[0078]

[0079] S28: According to the preferred distribution P S (l i |Λ seen ,S) and old-favored distribution P c (l i |Λ seen ,C,c), calculate the replay distribution P replay (l i As shown in equation (6):

[0080] P replay (l i )=(1-ρ)·P S (l i |Λ seen ,S)+ρ·P C (l i |Λ seen ,C,c) (6)

[0081] Wherein, ρ is a hyperparameter, and its value range is [0,1];

[0082] S29: Sampling Replay Distribution P replay To obtain the interactive environment i Update the global round count value c = c + 1.

[0083] Step S3: Drone and Interactive Environment i After a complete round of interaction, update the environment score S of the interacting environment. i Environmental timestamp C i and experience pool Implementation details are as follows:

[0084] S31: The drone uses the current strategy and environment. i Interact for one entire round to obtain the trajectory τ;

[0085] S32: Update the environment according to equation (1) i Environmental score S i ;

[0086] S33: Update the environment i Timestamp C i =c;

[0087] S34: τ is represented by empirical samples {(s)t ,a t ,r(s t ,a t ),s t+1 Stored in the experience pool in the form of )}

[0088] Step S4: Adjust the experience pool Sampling is performed, and the UAV interaction strategy π is updated using the PPO algorithm based on the empirical samples obtained from the sampling. Implementation details are as follows:

[0089] S41: From the experience pool A batch of empirical samples (s) were sampled in the middle. t ,a t ,r(s t ,a t ),s t+1 );

[0090] S42: Based on empirical samples, use stochastic gradient descent to update the parameters of the Critic network and the Actor network in turn to update the UAV interaction strategy π.

[0091] In one embodiment, repeating steps S2-S4 above until training is complete can yield a UAV strategy adaptable to multiple environmental tasks.

[0092] Figure 2 The diagram illustrates the implementation process of this invention in a real-world UAV environment mission. The specific steps are as follows:

[0093] Step 1: Set the Bernoulli distribution parameter p;

[0094] Step 2: Initialize the training environment set Λ train ={l1,l2,...l M}, Replay Environment Set Initialize the global environment score S and global environment timestamp C as empty arrays, initialize the global round count c = 0, initialize the drone interaction strategy π, and initialize the experience pool. empty set;

[0095] Step 3: Based on the Bernoulli distribution parameter p, randomly generate the result d=0 or d=1;

[0096] Step 4: Determine if d = 0 && |Λ train \Λ seen Is |>0 true?

[0097] If successful, proceed to steps 5-7:

[0098] Step 5: From the training environment set Λ train Random sampling of interactive environments in non-interactive environmentsi ~P new (l|Λ train ,Λ seen );

[0099] Step 6: Set the interactive environment i Stored in the replay environment set Λ seen middle;

[0100] Step 7: Initialize the interactive environment i Environmental score S i =0, Environment timestamp C i =0, and set S i C i Save the global environment score S and global environment timestamp C; update the global round count value c = c + 1;

[0101] If this does not work, proceed to steps 8-10:

[0102] Step 8: Based on the drone usage strategy π and the replay environment set Λ seen Calculate Λ based on the most recent interaction trajectory τ of all environments. seen Score for each environment Use the priority function h(S) i The playback environment scores are normalized and output to obtain the priority distribution P. S (l i |Λ seen ,S):

[0103]

[0104] Step 9: Obtain the stale priority distribution based on the global round count value c and the global environment timestamp C:

[0105]

[0106] Step 10: Based on the preferred distribution P S (l i |Λ seen ,S) and old-favored distribution P c (l i |Λ seen ,C,c), calculate the replay distribution P replay (l i ):P replay (l i )=(1-ρ)·P S (l i |Λ seen ,S)+ρ·P C (l i |Λ seenC,c), sampling replay distribution P replay Obtain the interactive environment i Update the global round count value c = c + 1;

[0107] Step 11: The drone uses the current strategy π and environment l i Interact for a full round, obtain the trajectory τ, and update the environment l. i Environmental score S i =score(τ,π), environment timestamp C i =c;

[0108] Step 12: Transform the trajectory τ into empirical samples {(s) t ,a t ,r(s t ,a t ),s t+1 Stored in the experience pool in the form of )}

[0109] Step 13: Optimize the experience pool Sampling is performed, and based on the empirical samples obtained from the sampling, a suitable reinforcement learning algorithm is used to update the UAV interaction strategy π.

[0110] This invention addresses the challenges of UAVs performing tasks in diverse environments. Traditional methods involve uniformly sampling all environmental samples, leading to slow training of UAV policies in simpler environments (e.g., obstacle-free environments) but prolonged training in more complex environments (e.g., environments with numerous obstacles). The proposed method considers the varying difficulty of all interactive environments during UAV policy training and quantitatively measures this difficulty through replay distribution calculations. This allows the UAV to prioritize adapting to more challenging environments, increasing policy generalization and improving training efficiency. Furthermore, traditional methods require retraining the policy when facing new environments, whereas this invention simply integrates the new environment into the existing training environment set and trains the policy based on the existing framework.

[0111] In summary, this invention presents a novel training method that comprehensively considers the environmental score and the staleness of the interaction time, quantitatively analyzes the learning priority and potential of each environment, and determines the interaction environment in which the UAV prioritizes experience collection, thereby improving the generalization of the UAV strategy under multi-environment tasks. When the UAV faces a new environment, the new environment is placed in the training environment set, and training is performed using the method disclosed in this invention. This allows for efficient and rapid training of a UAV strategy adapted to the new environment task based on the original strategy. This invention's UAV reinforcement learning training method for adapting to multi-environment tasks ensures strong generalization of the trained strategy while avoiding the dilemma of the UAV learning from scratch when facing a new environment. Furthermore, it can be combined with any reinforcement learning algorithm, facilitating the large-scale deployment of UAV strategies in various complex environments.

[0112] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1.A method for training a UAV reinforcement learning adapted to multi-environment tasks, characterized in that, Comprising: Step S1: Set Bernoulli distribution parameters , initialize a training environment set , replay an environment set , a global environment score , a global environment timestamp , a global round count value , a UAV interaction strategy and an experience pool ; Step S2: generating a random parameter according to Bernoulli distribution parameters , or ; and ; like and From the training environment set Randomly sample environments from all uninterrupted environments. As an interactive environment, and will Add to replay environment set In the middle; initializing the new environment Environmental score and environment timestamp ; If or , compute a replay distribution over a set of replay environments , sample an environment from the set of replay environments according to the replay distribution as the interaction environment; Step S3: Drone and Interactive Environment After a complete round of interaction, update the interacted environment. Environmental score Environment timestamp and experience pool ; Step S4: sampling experience pool According to the experience samples obtained by sampling, the unmanned aerial vehicle interaction strategy is updated using a reinforcement learning algorithm Steps S2-S4 are repeated until the training is completed, and the unmanned aerial vehicle strategy adapted to the multi-environment task is obtained. Step S2: is specifically implemented as: Judging: If and steps S21-S23 are executed: S21 : randomly sample a new interaction environment from the set of training environments S21 : randomly sample a new interaction environment from the set of training environments wherein is defined as the index value of the interaction environment ; S22: The new interactive environment is stored in the set of playback environments S22: The new interactive environment is stored in the set of playback environments​ S23: initialize environment score , environment timestamp and save to global environment score and global environment timestamp ; update global round count value ; If or steps S24-S29 are executed. S24: According to the unmanned aerial vehicle interaction strategy with the replay environment set all environment last time interaction trajectory , calculate Each environment score As shown in equation (1): （1） where, is the total time step, is the total time step, is the reward discount factor, is a GAE expression, which is a generalized advantage estimation formula commonly used in the field of reinforcement learning, and the hyperparameter controls the balance of variance and bias of GAE, is the 1-step TD error at time step （2） wherein, is the first is the second is the immediate reward at time step is the state value of state and state is the state value of state S25: using a priority function to the environment score i.e. the global environment score each element inside is processed as shown in equation (3): （3） global environment score is an array that holds the scores of all environments in the replay environment set where is the environment score in the global environment score ranked in descending order​ S26: Normalized priority function , obtaining the environmental priority distribution as shown in equation (4): （4） wherein is a temperature parameter for adjusting the distribution; S27: Obtain the global round count value and the global environment timestamp get the environment of the stale precedence distribution as shown in equation (5): （5） S28: According to and , the playback distribution of the playback environment is calculated as shown in equation (6): （6） wherein is a hyperparameter, with a value range of ; S29: Sample playback distribution , resulting in an interactive environment ; update the global round count value , add 1. 2.The method of claim 1, wherein, The step S1: setting Bernoulli distribution parameters , initializing a training environment set , replaying an environment set , a global environment score , a global environment timestamp , a global round count value , a UAV interaction strategy and an experience pool , and the specific implementation is as follows: S11: Set Bernoulli distribution parameter ; S12: initialize the set of environments according to the category of the environment Initialize the set of training environments where , representing the i-th interaction environment, initialize the set of replay environments global environment score global environment timestamp empty array and global round count value ; S13: initialize the UAV interaction strategy ; S14: Initialize the experience pool , empty set.

Images