A method for generating UAV maneuver strategies based on sparse rewards
By adopting intelligent models in the aerial game of drones, combining embedded networks, policy networks and value networks, the problem of slow convergence of sparse reward tasks is solved, and the maneuverability and winning rate of drones are improved.
Patent Information
- Application Number
- CN202411280348.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2025-03-11
- Estimated Expiration
- 2044-09-13
AI Technical Summary
In the prior art, drone aerial game tasks based on sparse rewards are slow to converge, and it is difficult for an agent to find useful paths, and due to the "signal delay" problem of sparse rewards and local optimal traps, the winning rate is low.
Adopting an agent-based approach, including embedded networks, policy networks and value networks, through distributed executors and learners working together, collect trajectory data, calculate discount returns, and update the policy network and value network models. At the same time, through parameter masking and weight reshaping, the strategy randomness is increased and the environmental exploration ability is improved.
It improves the convergence of the algorithm and the maneuverability of the drone, and significantly improves the winning rate in a sparse reward environment.
Smart Images
Figure CN118798065B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for generating a maneuvering strategy for an unmanned aerial vehicle, in particular to a method for generating a maneuvering strategy for an unmanned aerial vehicle based on sparse rewards, and belongs to the technical field of maneuvering strategy generating methods. Background Art
[0002] Reinforcement learning has made some progress in the fields of flight control systems, route planning, and air combat games, verifying the feasibility of neural network models to control fixed-wing aircraft. However, most algorithms are based on empirically designed dense reward functions, such as energy advantage, angle advantage, etc., which are not completely positively correlated with the victory of air games. As the game situation changes dramatically, although the simulation curve of the return gradually converges, the winning rate is still very low due to falling into the local optimum.
[0003] Drone aerial gaming is a dynamic process where the situation changes rapidly, and traditional strategy modeling algorithms use empirical methods as rewards, which makes it difficult to prove the relationship between returns and winning rates.
[0004] The reason why sparse reward tasks are difficult to converge is that, first, it is difficult for the agent to find useful paths, and the useful rewards received are few and cannot guide its actions. Second, sparse rewards are usually only obtained when the task is completed, which leads to the "signal delay" problem. Third, due to insufficient exploration of the environment, the agent may fall into a local optimal state. Summary of the invention
[0005] The purpose of the present invention is to solve the problem of slow convergence of sparse reward tasks mentioned in the prior art, and to provide a method for generating a UAV maneuver strategy based on sparse rewards, which not only improves the convergence of the algorithm, but also improves the maneuverability of the UAV.
[0006] In order to solve the above problems, the present application is implemented through the following technical solutions:
[0007] A method for generating a maneuvering strategy for a UAV based on sparse rewards is special in that: the agent model includes an embedding network, a policy network and a value network, wherein the embedding network and the policy network constitute a maneuvering decision model during deployment, the value network is used to evaluate the state value of the agent, the embedding network inputs its own flight parameters, relative situation and other information, extracts implicit features as inputs of the policy network and the value network, the policy network outputs flight control and weapon control instructions, and the value network outputs state value, the agent model includes distributed executors and learners, and specifically includes the following steps:
[0008] S1. Trajectory collection: Distributed executors interact with the environment independently, collect their trajectories and calculate discounted rewards, and store the collected trajectories in a global replay experience pool for offline policy training;
[0009] S2, update the learner's strategy network and value network model;
[0010] S3. Update the policy network model of the executor: The learner publishes the policy network parameters to each executor, thereby updating the policy network models of all executors. Each executor uses the parameter mask to randomly reshape the inactive weights of the embedded network, increase the policy randomness and thus improve the environment exploration ability.
[0011] Furthermore, the step S1 is specifically as follows: each executor has its own model and environment, interacts with the environment independently, and collects trajectories of length N;
[0012] The discount return for: ,
[0013] Where: represents the time variable of the accumulation process, Indicates the current time. express Rewards at all times, express Moment status, express Moment action, T represents the end time of the round, is the discount coefficient, r represents the reward function, a represents the action space, s represents the environment state, and the reward function r is composed of rewards for defeating the opponent, being defeated, successfully attacking, successfully avoiding, etc. The rewards for the above events are positively correlated with the probability of winning;
[0014] Return Estimates: ,
[0015] Where: Represents the estimated return at time t+1, which is R t+1 An estimated value of
[0016] If the last piece of trajectory data collected is the end time of the game, then , is the real return, otherwise To estimate the return, Indicates the state value at time t+1;
[0017] When the trajectory sampling is completed, the trajectory Saved in the global replay experience pool.
[0018] Furthermore, the step S2 specifically includes the following steps:
[0019] Step S2.1: Calculate online policy gradient ;
[0020] Step S2.2: Calculate offline policy gradient ;
[0021] Step S2.3, update the policy network model: update the online policy gradient calculated in step S2.1 and the offline policy gradient calculated in step S2.2 Add together to form a policy gradient , update the policy network model using the gradient ascent method;
[0022] Step S2.4, update the value network model: the learner takes the current value network model parameters as the mean, randomly samples them using a normal distribution, obtains the value network model population, selects the value network parameters with the smallest absolute advantage and updates the value network model.
[0023] Furthermore, the step S2.1 is as follows: the learner first calculates the normalized advantage and calculates the weighted online policy gradient , and then the thread suspends and waits for the next round, as follows:
[0024] The online policy loss function is: ,
[0025] Where: represents the policy network parameters, ε is the trimming coefficient of the PPO algorithm, represents the probability ratio of the new and old strategies, The estimated advantage of the action is calculated using the generalized advantage estimation (GAE) algorithm to evaluate the relative performance of the current action to the average level.
[0026] ,
[0027] Where: i represents the number of each executor, represents the old strategy for interacting with the environment,
[0028] The estimated advantage is defined as: ,
[0029] Where: is the discount factor, is the timing difference error, , λ is a parameter to reduce variance, T represents the end time of the round, and t represents the current time;
[0030] The online policy gradient calculation of the i-th executor is , the online policy gradient is the gradient of the online policy loss function, which is obtained by derivation of the online policy loss function;
[0031] Each executor passes the online policy gradient to the learner;
[0032] Learner's online policy gradient calculation: ,
[0033] Where: , is the normalized advantage of each performer, calculated using the standard deviation or normalization method, and σ represents the sigmoid function.
[0034] Furthermore, the normalized advantage of each performer is calculated using a normalized method, ,
[0035] Where: represents the average advantage of the ith executor , is the estimated advantage of the ith executor at time t.
[0036] Furthermore, the step S2.2 is specifically as follows: the learner's offline strategy loss function is: ,
[0037] Where: represents the value network parameters, , which means that only dominant behaviors are learned;
[0038] Sample global experience pool data and calculate offline policy gradients , the offline policy gradient is the gradient of the offline policy loss function.
[0039] Furthermore, the step S2.3 is specifically as follows: the strategy gradient of the learner is: ;
[0040] The parameters of the policy network model are updated using the gradient ascent algorithm through the obtained policy gradient. ;
[0041] Where: represents the policy network parameters, Represents the learning rate.
[0042] Furthermore, the step S2.4 is specifically as follows: the value network parameters in the value network population Sampling is based on normal distribution, that is, φ-N(φ 0 ,ξ), is the optimal parameter after each round of training, To customize the standard deviation, in the next round of training, select the network parameters with the smallest absolute advantage value ,
[0043] Where: N represents the trajectory length, and 1-k represents the value network number.
[0044] Furthermore, the step S3 is specifically as follows: weights whose absolute weights are less than a threshold are regarded as inactive weights and are randomly redistributed to increase the randomness of the strategy and improve the agent's ability to explore the environment.
[0045] Furthermore, the threshold in step S3 is 10 -5 .
[0046] This application uses event-based sparse reward function reinforcement learning (RL) to train strategy modeling. The reward function r is composed of rewards for events such as defeating the opponent, being defeated, successfully attacking, and successfully avoiding. The rewards for the above events are positively correlated with the chance of winning. Compared with traditional dense rewards based on placeholders, the intelligent agent can obtain a higher winning rate.
[0047] This application proposes a method to redistribute weights by sharing the parameter mask of the embedding network to enhance the environment exploration capability. The simulation results show that the proposed algorithm not only improves the convergence of the algorithm, but also improves the maneuverability of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of the method of the present invention;
[0049] Figure 2 It is a data flow diagram of the present invention;
[0050] Figure 3 This is the reward simulation curve of the red agent;
[0051] Figure 4 is the relative reward simulation curve of the red agent;
[0052] Figure 5 is the winning rate of the red agent;
[0053] Figure 6 is the winning rate of the blue agent;
[0054] Figure 7 Figure 1 is a situation diagram of the game process between the two sides, where Figure (a) shows the simulation initialization, with the two teams flying towards each other at a distance of 7 km. Figure (b) shows the simulation at 10 seconds, when the red team makes a sharp turn and takes advantage. Figure (c) shows the simulation at 20 seconds, when the red team launches a missile and the blue team maneuvers to evade. Figure (d) shows the simulation at 30 seconds, when the red team keeps locked on and launches a missile. Figure (e) shows the simulation at 40 seconds, when the red team shoots down the blue team and wins. DETAILED DESCRIPTION
[0055] The following is a specific implementation of the present invention with reference to the accompanying drawings to further illustrate the composition of the present invention.
[0056] Example 1. A method for generating a UAV maneuver strategy based on sparse rewards, the specific process is as follows: Figure 1 As shown, the agent-based model includes an embedding network, a policy network and a value network, wherein the embedding network and the policy network constitute a maneuver decision model during deployment, the value network is used to evaluate the state value of the agent, the embedding network inputs its own flight parameters, relative situation and other information, extracts implicit features as inputs of the policy network and the value network, the policy network outputs flight control and weapon control instructions, and the value network outputs state value. The agent model includes distributed executors and learners, and specifically includes the following steps:
[0057] S1. Trajectory collection: Distributed executors interact with the environment independently, collect their trajectories and calculate discounted rewards, and store the collected trajectories in a global replay experience pool for offline policy training;
[0058] S2, update the learner's strategy network and value network model;
[0059] S3. Update the policy network model of the executor: The learner publishes the policy network parameters to each executor, thereby updating the policy network models of all executors. Each executor uses the parameter mask to randomly reshape the inactive weights of the embedded network, increase the policy randomness and thus improve the environment exploration ability.
[0060] Furthermore, the step S1 is specifically as follows: each executor has its own model and environment, interacts with the environment independently, and collects trajectories of length N;
[0061] The discount return for: ,
[0062] Where: represents the time variable of the accumulation process, Indicates the current time. express Rewards at all times, express Moment status, express Moment action, T represents the end time of the round, is the discount coefficient, r represents the reward function, a represents the action space, s represents the environment state, and the reward function r is composed of rewards for defeating the opponent, being defeated, successfully attacking, successfully avoiding, etc. The rewards for the above events are positively correlated with the probability of winning;
[0063] Return Estimates: ,
[0064] Where: Represents the estimated return at time t+1, which is R t+1 An estimated value of
[0065] If the last piece of trajectory data collected is the end time of the game, then , is the real return, otherwise To estimate the return, Indicates the state value at time t+1;
[0066] When the trajectory sampling is completed, the trajectory Saved in the global replay experience pool.
[0067] Furthermore, the step S2 specifically includes the following steps:
[0068] Step S2.1: Calculate online policy gradient ;
[0069] Step S2.2: Calculate offline policy gradient ;
[0070] Step S2.3, update the policy network model: update the online policy gradient calculated in step S2.1 and the offline policy gradient calculated in step S2.2 Add together to form a policy gradient , update the policy network model using the gradient ascent method;
[0071] Step S2.4, update the value network model: the learner takes the current value network model parameters as the mean, randomly samples them using a normal distribution, obtains the value network model population, selects the value network parameters with the smallest absolute advantage and updates the value network model.
[0072] Furthermore, the step S2.1 is as follows: the learner first calculates the normalized advantage and calculates the weighted online policy gradient , and then the thread suspends and waits for the next round, as follows:
[0073] The online policy loss function is: ,
[0074] Where: represents the policy network parameters, ε is the trimming coefficient of the PPO algorithm, represents the probability ratio of the new and old strategies, The estimated advantage of the action is calculated using the generalized advantage estimation (GAE) algorithm to evaluate the relative performance of the current action to the average level.
[0075] ,
[0076] Where: i represents the number of each executor, represents the old strategy for interacting with the environment,
[0077] The estimated advantage is defined as: ,
[0078] Where: is the discount factor, is the timing difference error, , λ is a parameter to reduce variance, T represents the end time of the round, and t represents the current time;
[0079] The online policy gradient calculation of the i-th executor is , the online policy gradient is the gradient of the online policy loss function, which is obtained by derivation of the online policy loss function;
[0080] Each executor passes the online policy gradient to the learner;
[0081] Learner's online policy gradient calculation: ,
[0082] Where: , is the normalized advantage of each performer, calculated using the standard deviation or normalization method, and σ represents the sigmoid function.
[0083] Furthermore, the normalized advantage of each performer is calculated using a normalized method, ,
[0084] Where: represents the average advantage of the ith executor , is the estimated advantage of the ith executor at time t.
[0085] Furthermore, the step S2.2 is specifically as follows: the learner's offline strategy loss function is: ,
[0086] Where: represents the value network parameters, , which means that only dominant behaviors are learned;
[0087] Sample global experience pool data and calculate offline policy gradients , the offline policy gradient is the gradient of the offline policy loss function.
[0088] Furthermore, the step S2.3 is specifically as follows: the strategy gradient of the learner is: ;
[0089] The parameters of the policy network model are updated using the gradient ascent algorithm through the obtained policy gradient. ;
[0090] Where: represents the policy network parameters, Represents the learning rate.
[0091] Furthermore, the step S2.4 is specifically as follows: the value network parameters in the value network population Sampling is based on normal distribution, that is, φ-N(φ 0 ,ξ), is the optimal parameter after each round of training, To customize the standard deviation, in the new round of training, select the network parameters with the smallest absolute advantage value: ,
[0092] Where: N represents the trajectory length, and 1-k represents the value network number.
[0093] Furthermore, the step S3 is specifically as follows: weights whose absolute weights are less than a threshold are regarded as inactive weights and are randomly redistributed to increase the randomness of the strategy and improve the agent's ability to explore the environment.
[0094] Furthermore, the threshold in step S3 is 10 -5 .
[0095] The game process is a sequential decision problem, which is formed by a Markov decision process (MDP), which consists of a state space E, an action space a, a transition function P, a reward function r and a discount factor γ, forming a tuple<S, A, P, r, γ> .
[0096] The algorithm corresponding to the method of this application is the PPO-PME algorithm: Parameters Masking and valuenetwork population Evolution Parameter masking and value population evolution PPO algorithm, its data flow is as follows Figure 2 As shown, the algorithm model consists of an embedding network for feature extraction, a policy network, and a value network for estimating state value. Figure 2 The solid line represents the data flow of interaction with the environment, the dotted line represents the data flow of distributed policy gradients, and the dashed line represents the evolution of the policy-based value network population.
[0097] Embodiment 2. This embodiment is a simulation of the method described in Embodiment 1.
[0098] The aerodynamic and kinematic models of the UAV were established based on the JSBSim platform.
[0099] The algorithm proposed in Example 1 trains the model of the red agent, where the blue agent is a rule-based model. The initial altitude of both aircraft is 5000m. The initial relative horizontal distance is 1-20km. The initial speed is 200-300m / s, and the initial heading is reversed.
[0100] Simulation Settings
[0101] In order to control the aircraft to make maneuvers and shooting decisions according to relevant situations, the policy network outputs elevator, aileron, rudder, throttle and shooting commands as actions. The state includes the aircraft's current own state, relative position and attitude, shooting conditions and missile state under maximum threat. The own state consists of normalized altitude, normalized velocity and attitude, the relative state consists of normalized relative position, normalized relative velocity, azimuth and attachment angle, and the shooting condition is a flag indicating that the missile is ready to shoot when the target is within the range and attack angle. In particular, the missile state consists of relative position and attitude only when the aircraft is tracked, otherwise it is zero.
[0102] The reward function is sparse, as shown in Table 1, where the agent receives non-zero rewards only on terminal or special events (such as missile firing or successful evasion), ending with leaving the engagement zone, abnormal posture, stalling, being shot down, or the enemy being shot down.
[0103] Table 1. Sparse rewards and termination conditions
[0104] .
[0105] Simulation Results
[0106] Figure 3 and Figure 4 The reward simulation curve and relative reward simulation curve of the red agent are given respectively.
[0107] In the figure, PPO represents the classic PPO2 algorithm with advantage cutting, PPO-SIL represents the PPO algorithm with self-imitation learning, PPO-PM represents the PPO algorithm with embedding network weight reshaping based on parameter mask, and the PPO-PME algorithm is a PPO algorithm with both embedding network weight reshaping based on parameter mask and value network group evolution.
[0108] Figure 3 It shows that all PPO-based algorithms have converged, while the weight reshaping algorithm of this application converges faster and more stably. However, the game scenario should not only consider one's own interests, but also comprehensively consider the differences in the interests of both parties.
[0109] from Figure 4 It can be seen that although the PPO-SIL algorithm converges faster, the relative return decreases significantly, which may be due to the reduction of state space exploration under sparse rewards. Relatively speaking, through weight reshaping, the network will randomly assign parameters with very low weights, thereby improving the agent's exploration of the environment and avoiding too large a difference from the learning behavior.
[0110] Figure 5 and Figure 6The winning rates of the red and blue agents are given respectively. The simulation results show that the winning rate is not strongly linearly related to the reward, which is because the reward function includes event rewards, and the agent will still fall into the local optimum during the training process.
[0111] The proposed PPO-PME algorithm improves the environment exploration, thus achieving significant improvements in both reward and win rate simulation.
[0112] The simulation of a typical game process is as follows: Figure 7 As shown, Figures (a) to (e) represent different time stages of the simulation. In the initial stage, the Red team was heading at 270°, the Blue team was heading at 90°, the azimuth was about 90°, and the distance was 7km. The two sides were evenly matched, as shown in Figure (a). At the beginning of the game, the Red team chose a horizontal sharp turn and the Blue team chose a low yo-yo action. Both sides chose to quickly establish an angle advantage, as shown in Figure (b). The Red team changed its nose faster and locked the Blue team, launching two missiles in succession when the weapon attack zone conditions were met, as shown in Figure (c). In Figure (d), when the Red team continued to point at the Blue team to maintain its positional advantage, the Blue team was forced to use a broken-S maneuver to dodge the missile. The Blue team's angle and energy advantage was not as good as the Red team because it chose to maneuver significantly to dodge the missile. In the end, in order to avoid falling, the Blue team chose to pull up and inevitably lost kinetic energy. The Red team launched a third missile and shot it down, as shown in Figure (e). The process shows that the Red team model generated by this method has a strong game ability.
[0113] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the defined scope.
Claims
1. A method for generating UAV maneuvering strategies based on sparse rewards, characterized by: The agent-based model includes an embedded network, a policy network, and a value network. The embedded network and the policy network constitute a maneuver decision model during deployment. The value network is used to evaluate the state value of the agent. The embedded network inputs its own flight parameters, relative situation and other information, extracts implicit features as inputs to the policy network and the value network. The policy network outputs flight control and weapon control instructions, and the value network outputs state value. The agent model includes distributed executors and learners, and specifically includes the following steps: S1. Trajectory collection: Distributed executors interact with the environment independently, collect their trajectories and calculate discounted rewards, and store the collected trajectories in a global replay experience pool for offline policy training; Each executor has its own model and environment, interacts with the environment independently, and collects trajectories of length N; The discount return for: , Where: represents the time variable of the accumulation process, Indicates the current time. express Rewards at all times, express Moment status, express Moment action, T represents the end time of the round, is the discount coefficient, r represents the reward function, a represents the action space, s represents the environment state, and the reward function r is composed of rewards for defeating the opponent, being defeated, successfully attacking, and successfully avoiding. The rewards for the above events are positively correlated with the probability of winning; Return Estimates: , Where: Represents the estimated return at time t+1, which is R t+1 An estimated value of If the last piece of trajectory data collected is the end time of the game, then , is the real return, otherwise To estimate the return, Indicates the state value at time t+1; When the trajectory sampling is completed, the trajectory Saved in the global playback experience pool; S2. Update the learner's strategy network and value network model, which specifically includes the following steps: Step S2.1: Calculate online policy gradient ; Step S2.2: Calculate offline policy gradient ; Step S2.3, update the policy network model: update the online policy gradient calculated in step S2.1 and the offline policy gradient calculated in step S2.2 Add together to form a policy gradient , update the policy network model using the gradient ascent method; Step S2.4, update the value network model: the learner takes the current value network model parameters as the mean, randomly samples them using a normal distribution, obtains the value network model population, selects the value network parameters with the smallest absolute advantage and updates the value network model; S3. Update the policy network model of the executor: The learner publishes the policy network parameters to each executor, thereby updating the policy network models of all executors. Each executor uses the parameter mask to randomly reshape the inactive weights of the embedded network, increase the policy randomness and thus improve the environment exploration ability.
2. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 1, characterized in that: The step S2.1 is as follows: the learner first calculates the normalized advantage and calculates the weighted online policy gradient , and then the thread suspends and waits for the next round, as follows: The online policy loss function is: , Where: represents the policy network parameters, ε is the trimming coefficient of the PPO algorithm, represents the probability ratio of the new and old strategies, is the estimated advantage of the action, ， Where: i represents the number of each executor, represents the old strategy for interacting with the environment, The estimated advantage is defined as: , Where: is the discount factor, is the timing difference error, , λ is a parameter to reduce variance, T represents the end time of the round, and t represents the current time; The online policy gradient calculation of the i-th executor is , the online policy gradient is the gradient of the online policy loss function, which is obtained by derivation of the online policy loss function; Each executor passes the online policy gradient to the learner; Learner's online policy gradient calculation: , Where: , is the normalized advantage of each performer, calculated using the standard deviation or normalization method, and σ represents the sigmoid function.
3. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 2, characterized in that: The normalized advantage of each performer is calculated using the normalization method, , Where: represents the average advantage of the ith executor , is the estimated advantage of the ith executor at time t.
4. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 1, characterized in that: The step S2.2 is specifically as follows: the learner's offline strategy loss function is: , Where: represents the value network parameters, , which means that only dominant behaviors are learned; Sample global experience pool data and calculate offline policy gradients , the offline policy gradient is the gradient of the offline policy loss function.
5. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 1, characterized in that: The learner's policy gradient is: ; The parameters of the policy network model are updated using the gradient ascent algorithm through the obtained policy gradient. ; Where: represents the policy network parameters, Represents the learning rate.
6. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 1, characterized in that: Value network parameters in the value network population Sampling is based on normal distribution, i.e. φ-N(φ0,ξ), is the optimal parameter after each round of training, To customize the standard deviation, in the new round of training, select the network parameters with the smallest absolute advantage value: , Where: N represents the trajectory length, and 1-k represents the value network number.
7. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 1, characterized in that: The step S3 is specifically as follows: weights whose absolute weights are less than a threshold are regarded as inactive weights and are randomly redistributed to increase the randomness of the strategy and improve the agent's ability to explore the environment.
8. The method for generating a UAV maneuver strategy based on sparse rewards according to claim 7, characterized in that: The threshold in step S3 is 10 -5 .