A UAV control method and system based on multi-expert imitation learning
By adopting multi-expert imitation learning methods in drone control, using generative adversarial networks and reinforcement learning models, the problem of difficulty in obtaining high-quality expert data is solved, and the accuracy and robustness of drone control is improved. It is suitable for Complex flight environment.
Patent Information
- Application Number
- CN202410560553.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-05-08
AI Technical Summary
Among the existing drone control methods, it is difficult to obtain high-quality expert data, which leads to inaccurate drone control decisions, especially in the flight operations and target mission completion of multiple drones.
Using a multi-expert imitation learning method, a generative adversarial network is built by using multiple expert strategies to deduce expert trajectories in a drone simulation environment, including a strategy generator and a discriminator, and the reward value is calculated through the discriminator accuracy, and the strategy is updated. The generator combines reinforcement learning model (PPO algorithm) to train drone action strategies to achieve comprehensive utilization of multi-quality expert data and noise removal.
Improve the accuracy and target completion rate of UAV control decisions, enhance the robustness of the model, and enable multi-quality expert imitation learning training to approach higher quality expert trajectories, suitable for complex and diverse flight environments .
Smart Images

Figure CN118567372B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a method and system for controlling unmanned aerial vehicles based on multi-expert imitation learning. Background Art
[0002] Drones are increasingly used in various fields due to their good maneuverability and high safety factor. However, the flight environment of drones is diverse, complex, and rapidly changing, and achieving autonomous flight control has always been the focus of technical research.
[0003] Imitation learning is an important technology in machine learning, and its goal is to learn a task or solve a problem by observing and imitating the behavior of experts. Compared with traditional rule-based or reward-based learning methods, imitation learning can directly learn high-quality behavior strategies from expert demonstrations. The advantage of behavior cloning (BC) in imitation learning is that the method is simple, but it is easy to cause state drift problems; Generative Adversarial Imitation Learning (GAIL) in imitation learning learns expert experience distribution rather than reward function through generative adversarial network (GAN), learns rewards through expert data, and updates strategies through reinforcement learning, avoiding the defects of behavior cloning methods.
[0004] However, imitation learning methods all assume that the expert's strategy is the best and that the data is obtained from high-quality experts. However, in reality, there are problems such as incomplete expert data and uneven expert experience, which makes it difficult to obtain high-quality expert data. In the process of imitation learning, if the quality of expert data is diverse and the distribution of quality is unclear, and there is no additional information to evaluate and screen expert data of different qualities, imitation learning becomes very difficult. Therefore, how to make multi-quality expert imitation learning training tend to approach higher-quality expert trajectories, better control the flight actions of multiple drones in a drone cluster, and complete the target task has become the main bottleneck problem restricting its application in the field of drone measurement and control. Summary of the invention
[0005] In view of the above analysis, the embodiments of the present invention aim to provide a drone control method and system based on multi-expert imitation learning, so as to solve the problem of inaccurate drone control decisions caused by the difficulty in obtaining high-quality expert data.
[0006] On the one hand, an embodiment of the present invention provides a method for controlling a drone based on multi-expert imitation learning, comprising the following steps:
[0007] Based on the UAV simulation environment, multiple expert strategies are used to deduce expert trajectories.
[0008] Construct a generative adversarial network, including a policy generator and multiple discriminators; during the training process, the policy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into a discriminator, updates the discriminator parameters and obtains the accuracy of the discriminator; according to the accuracy of each discriminator, calculates the reward value to update the policy generator; after the training, the trained policy generator is obtained;
[0009] It interacts with the real environment, collects and splices the state vectors of each drone in the drone cluster, inputs them into the trained strategy generator, outputs the action vectors of each drone, converts the action vectors of each drone into instructions and distributes them to the corresponding drones to achieve control of each drone.
[0010] Based on the further improvement of the above method, the expert strategy is a reinforcement learning model obtained by training different numbers of rounds with the PPO algorithm; the reinforcement learning model includes a policy network and a value network, wherein the policy network is used to output the drone action vector in a two-dimensional plane area according to the input drone state vector, and the value network is used to estimate the value function value of the input drone state vector.
[0011] Based on the further improvement of the above method, the drone state vector includes: the horizontal and vertical speed of the current drone, the horizontal and vertical coordinates of the current drone, the horizontal and vertical coordinates of the target, the horizontal and vertical coordinates of other drones, and the communication status between the current drone and other drones; the drone action vector includes: the current drone's horizontal positive speed, the current drone's vertical positive speed, the current drone's horizontal negative speed, the current drone's vertical negative speed and communication strength.
[0012] Based on the further improvement of the above method, the number of discriminators is the same as the number of expert strategies; the accuracy of the discriminator is the sum of the expert accuracy and the imitator accuracy; the expert accuracy refers to the probability that the state-action pair in the expert trajectory is judged as correct by the discriminator, and the imitator accuracy refers to the probability that the state-action pair in the imitator trajectory is judged as correct by the discriminator.
[0013] Based on the further improvement of the above method, for the state-action pairs in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has judged correctly; for the state-action pairs in the imitator trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has judged correctly.
[0014] Based on the further improvement of the above method, the reward value is calculated according to the accuracy of each discriminator, including: taking the ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators as the expert reliability coefficient of each discriminator, and calculating the reward value according to the expert reliability coefficient of each discriminator and the output corresponding to the imitator trajectory in each discriminator.
[0015] Based on the further improvement of the above method, the reward value is calculated by the following formula according to the expert reliability coefficient of each discriminator and the output corresponding to the imitator trajectory in each discriminator:
[0016]
[0017] Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the discriminator corresponding to the jth expert trajectory during the i-th round of training; It represents the discriminant value output by the discriminator corresponding to the j-th expert trajectory for the state-action pair (s, a) in the imitator trajectory during the i-th round of training; s represents the state vector in the trajectory, and a represents the action vector in the trajectory.
[0018] Based on the further improvement of the above method, the discriminator parameters are updated by calculating the gradient according to the following formula:
[0019]
[0020] in, represents the computational expectation, Indicates the calculation gradient, w(j) i represents the discriminator parameters corresponding to the jth expert trajectory during the i-th round of training, T i represents the imitator trajectory sampled during the i-th round of training, T E represents the expert trajectory, They represent the discriminator corresponding to the jth expert trajectory during the i-th round of training. For the imitator state-action pair (s, a), the expert state pair (s, a) E The output discriminant value.
[0021] Based on the further improvement of the above method, the reward value of the expert strategy includes: distance reward and collision reward; the distance reward is the negative of the distance between the drone and the target, and the collision reward is the negative of the number of times the drone collides.
[0022] On the other hand, an embodiment of the present invention provides a drone control system based on multi-expert imitation learning, including:
[0023] The expert trajectory acquisition module is used to deduce expert trajectories based on the UAV simulation environment using multiple expert strategies;
[0024] The imitation learning module is used to construct a generative adversarial network, including a policy generator and multiple discriminators. During the training process, the policy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into a discriminator, updates the discriminator parameters and obtains the accuracy of the discriminator. According to the accuracy of each discriminator, the reward value is calculated to update the policy generator. After the training, the trained policy generator is obtained.
[0025] The control execution module is used to interact with the real environment, collect and splice the state vectors of each drone in the drone cluster, input them into the trained strategy generator, output the action vectors of each drone, convert the action vectors of each drone into instructions and distribute them to the corresponding drones to achieve control of each drone.
[0026] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0027] 1. Based on the simulation platform, we obtain multi-quality expert trajectories, which are close to the actual application scenarios. We input the drone trajectories and expert trajectories of different qualities into different discriminators at the same time, neutralize the high-quality expert data and the low-quality expert data, and make the multi-quality expert imitation learning training tend to approach the higher-quality expert trajectories, so as to achieve better learning effect and enhance the robustness of the model.
[0028] 2. Calculate the reliability coefficients of different expert data based on the accuracy of different discriminators, and use them as weights to remove the noise influence of different trajectories and improve the accuracy of UAV control decisions and target completion rate.
[0029] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0031] Figure 1 This is a flow chart of a drone control method based on multi-expert imitation learning in Example 1 of the present invention;
[0032] Figure 2This is an example diagram of the process of multi-expert imitation learning in Example 1 of the present invention;
[0033] Figure 3 This is a schematic diagram of the structure of a drone control system based on multi-expert imitation learning in Example 2 of the present invention. DETAILED DESCRIPTION
[0034] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0035] Example 1
[0036] A specific embodiment of the present invention discloses a drone control method based on multi-expert imitation learning, such as Figure 1 As shown, the following steps are included:
[0037] S1. Based on the UAV simulation environment, multiple expert strategies are used to deduce expert trajectories.
[0038] It should be noted that the expert strategy can be a rule defined according to expert experience, and drone control experts of different levels control the drone to interact with the simulation environment according to their own rules, and output the control action to obtain the expert trajectory. The expert strategy of this embodiment is a reinforcement learning model obtained by training different rounds of the PPO algorithm. This embodiment does not limit the number of expert strategies. Preferably, 3 expert strategies are trained to represent experts of 3 levels. For example, the expert strategy obtained after less than 10 rounds of training has a low task completion rate, representing an expert of poor level; the expert strategy obtained after 2000 rounds of training has a good task completion rate, representing an expert of average level; the expert strategy obtained after 4000 rounds of training has the best task completion rate, representing an expert of the best level.
[0039] Specifically, the reinforcement learning model includes a policy network and a value network, wherein the policy network is used to output the drone action vector according to the incoming drone state vector in a two-dimensional plane area, and the value network is used to estimate the value function value of the incoming drone state vector. The drone state vector includes: the horizontal and vertical speed of the current drone, the horizontal and vertical coordinates of the current drone, the horizontal and vertical coordinates of the target, the horizontal and vertical coordinates of other drones, and the communication status between the current drone and other drones; the drone action vector includes: the horizontal positive speed of the current drone, the vertical positive speed of the current drone, the horizontal negative speed of the current drone, the vertical negative speed of the current drone and the communication strength. It should be noted that during the training process, the positive and negative speeds in the same direction are superimposed as the speed of the current drone in that direction; when the model is trained, only one of the positive and negative speeds in the same direction has a value, and the other is close to 0.
[0040] For example, there are 3 drones in the drone cluster, and the task to be performed is that the 3 drones chase a moving target in a preset two-dimensional plane area. Then the state of each drone has 14 dimensions: the horizontal and vertical speed of the current drone (2 dimensions), the horizontal and vertical coordinates of the current drone (2 dimensions), the horizontal and vertical coordinates of the target (2 dimensions), the horizontal and vertical coordinates of the other 2 drones (2 groups of 2 dimensions), and the communication status between the current drone and the other 2 drones (2 groups of 2 dimensions, each group includes a 2-dimensional identifier of communication and non-communication); each drone action has 5 dimensions, which belongs to the continuous action space.
[0041] In this embodiment, the policy network and value network of the reinforcement learning model are both two-layer fully connected networks, using the Relu activation function. After setting the hyperparameters for training the reinforcement learning model, the drone continuously interacts with the simulation environment by issuing actions, obtains the next state and reward value, inputs them into the PPO algorithm model, and updates the policy network based on the PPO algorithm. The reward values of the expert strategy include: distance reward and collision reward; the distance reward is the negative of the distance between the drone and the target, and the collision reward is the negative of the number of collisions of the drone.
[0042] Exemplarily, the hyperparameters include: the discount factor gamma is set to 0.98, the GAE hyperparameter lmbda is set to 0.95, the batch size batch_size is set to 64, the policy network learning rate learning_rate_p is set to 3e-4, the value network learning rate learning_rate_v is set to 1e-3, the number of iterations ppo_epochs when updating the algorithm is set to 10, the step size per round is set to 50, and the number of hidden layer neurons hidden_size is set to 64.
[0043] After training, the model structures of multiple expert strategies are the same, but the network parameters are different. Next, multiple expert strategies are used to deduce expert trajectories, including:
[0044] In the simulation environment, the same task is performed, and the state vectors of each drone are spliced and input into the strategy network of each expert strategy respectively, and the value of the action vector of each drone is output respectively; according to the state vector s input by each expert strategy and the corresponding output action vector a, a state-action pair (s, a) is obtained. E , according to the execution step length m, multiple state-action pairs of each expert strategy are constructed into expert trajectories, representing expert data of different qualities. Among them, the jth expert trajectory is represented as
[0045]
[0046] Compared with the prior art, this embodiment takes into account that in actual situations, the experience of experts varies greatly and it is difficult to obtain or evaluate expert data with the best quality. Therefore, multi-quality expert data is obtained for comprehensive consideration to improve the applicability of the technical solution.
[0047] S2. Construct a generative adversarial network, including a policy generator and multiple discriminators. During the training process, the policy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into a discriminator respectively, updates the discriminator parameters and obtains the accuracy of the discriminator. According to the accuracy of each discriminator, the reward value is calculated to update the policy generator. After the training, a trained policy generator is obtained.
[0048] It should be noted that the number of discriminators in the generative adversarial network of this embodiment is the same as the number of expert strategies. The strategy generator also includes a strategy network and a value network, and the grid structure is the same as the network structure of the expert strategy.
[0049] According to the simulation requirements, set the policy network learning rate, value network learning rate, number of iterations when updating the algorithm, number of hidden layer neurons and other model parameters in the policy generator. For example, the policy network learning rate learning_rate_p is set to 2e-4, the value network learning rate learning_rate_v is set to 1e-4, the number of iterations ppo_epochs when updating the algorithm is set to 10000, and the number of hidden layer neurons hidden_size is set to 64.
[0050] If three expert trajectories are derived using three different levels of expert strategies in step S1, the imitation learning process in step S2 is as follows: Figure 2 shown.
[0051] exist Figure 2In each round of training in this embodiment, each drone performs the same task. The strategy generator generates an action according to the current state of the simulation environment, executes the action in the simulation environment, obtains the next state, and transmits the current state, the executed action, the next state, and the flag data of whether the execution is completed to the experience pool; through the interaction between the above strategy generator and the simulation environment, the state-action pair (s, a) sequence is obtained multiple times as the imitator trajectory, and the imitator trajectory and each expert trajectory are respectively input into a discriminator, and the parameters of each discriminator are updated by calculating the gradient of the discriminator, and the reward value is calculated according to the accuracy of each discriminator and transmitted to the experience pool. The loss function of the strategy generator is calculated based on the data stored in the experience pool based on the PPO algorithm, and the gradient descent algorithm is used to update the parameters of the strategy generator. Among them, when the drone exceeds the preset area boundary or any drone reaches the target position, the flag data of whether the execution is completed is 1, and the interaction with the simulation environment ends, otherwise it is 0, and the interaction with the simulation environment continues.
[0052] Specifically, the discriminator uses a three-layer fully connected network. The input is the state-action pairs in the expert trajectory and the state-action pairs in the imitator trajectory. The output is a two-dimensional real number, which corresponds to the discriminant value of the state-action pairs in the expert trajectory and the discriminant value of the state-action pairs in the imitator trajectory. The range of the discriminant value is [0,1], which is used to indicate whether the discriminator believes that the state-action pair comes from the expert trajectory or the imitator trajectory.
[0053] The gradient of each discriminator is calculated according to the following formula to update the parameters of each discriminator:
[0054]
[0055] in, represents the computational expectation, Indicates the calculation gradient, w(j) i represents the discriminator parameters corresponding to the j-th expert trajectory (i.e., the j-th discriminator) during the i-th round of training, T i represents the imitator trajectory sampled during the i-th round of training, represents the j-th expert trajectory, They represent the discriminator corresponding to the jth expert trajectory during the i-th round of training. For the imitator state-action pair (s, a), the expert state pair (s, a) E The output discriminant value.
[0056] It should be noted that for the state-action pair in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has judged it correctly; for the state-action pair in the imitator trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has judged it correctly. Further, the probability of the state-action pair in the expert trajectory being judged as correct by the discriminator is calculated to obtain the expert accuracy, that is, the number of state-action pairs in the expert trajectory that the discriminator has judged as correct divided by the total number of state-action pairs in the expert trajectory, expressed by formula (2); the probability of the state-action pair in the imitator trajectory being judged as correct by the discriminator is calculated to obtain the imitator accuracy, that is, the number of state-action pairs in the imitator trajectory that the discriminator has judged as correct divided by the total number of state-action pairs in the imitator trajectory, expressed by formula (3):
[0057]
[0058]
[0059] in, represents the expert accuracy of the j-th discriminator during the i-th round of training, represents the imitator accuracy of the jth discriminator during the i-th round of training; in formula (2), It indicates that the state-action pair in the expert trajectory is judged correct by the discriminator, It represents the probability that the state-action pair in the expert trajectory is correctly judged by the discriminator; in formula (3), It indicates that the state-action pair in the imitator's trajectory is judged to be correct by the discriminator. It represents the probability that the state-action pair in the imitator's trajectory is judged to be correct by the discriminator.
[0060] The accuracy of the discriminator is the sum of the expert accuracy and the imitator accuracy, expressed by formula (4):
[0061]
[0062] in, represents the accuracy of the j-th discriminator during the i-th round of training.
[0063] Furthermore, the reward value is calculated according to the accuracy of each discriminator, including: through formula (5), the ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators is used as the expert reliability coefficient of each discriminator; through formula (6), the reward value is calculated according to the expert reliability coefficient of each discriminator and the output corresponding to the imitator trajectory in each discriminator.
[0064]
[0065]
[0066] Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the jth discriminator during the i-th round of training; It represents the discriminant value output by the discriminator corresponding to the j-th expert trajectory for the state-action pair (s, a) in the imitator trajectory during the i-th round of training.
[0067] According to formulas (5) and (6), this embodiment targets multi-quality expert trajectories. During training, high-quality and low-quality expert trajectories are integrated. The higher the accuracy of the discriminator, the higher the expert reliability coefficient. This allows the imitator trajectory output by the strategy generator to learn the experience of the best expert in the current situation while learning the experience of different experts.
[0068] The calculated reward value is passed into the experience pool and matched with the current state of the drone, the action executed, the next state, and the flag data indicating whether the execution is completed. The loss function of the strategy generator is calculated based on these data based on the PPO algorithm, and the gradient descent algorithm is used to update the strategy generator parameters.
[0069] When the number of training rounds reaches the maximum, the trained strategy generator is obtained.
[0070] S3, interact with the real environment, collect and splice the state vectors of each drone in the drone cluster, input them into the trained strategy generator, output the action vectors of each drone, convert the drone action vectors into instructions and distribute them to the corresponding drones to achieve control of each drone.
[0071] It should be noted that the input is the splicing vector of each drone state, and the output is the splicing vector of each drone action. Since the action vector of each drone in this embodiment is continuous, the value of each dimension in the output action vector is used as the corresponding action instruction value, which is constructed into an instruction according to the instruction format of the drone and sent to the corresponding drone. The corresponding drone executes according to the action instruction value, thereby obtaining the new environmental state of the drone after executing the action instruction, and then iterates the environmental interaction until the task is completed to achieve control of each drone.
[0072] Compared with the prior art, the present embodiment provides a UAV control method based on multi-expert imitation learning, which obtains multi-quality expert trajectories based on a simulation platform, close to the actual application scenario, and inputs the UAV trajectory and expert trajectories of different qualities into different discriminators at the same time, neutralizes high-quality expert data and low-quality expert data, and makes the multi-quality expert imitation learning training tend to approach the higher-quality expert trajectory, so that the learning effect is better and the robustness of the model is enhanced; the reliability coefficient of different expert data is calculated according to the accuracy of different discriminators, and this is used as a weight to remove the noise influence of different trajectories, thereby improving the accuracy of UAV control decision-making and the target completion rate.
[0073] Example 2
[0074] Another embodiment of the present invention discloses a drone control system based on multi-expert imitation learning, thereby realizing a drone control method based on multi-expert imitation learning in Embodiment 1. The specific implementation of each module refers to the corresponding description in Embodiment 1. Figure 3 As shown, the system includes:
[0075] The expert trajectory acquisition module 101 is used to deduce expert trajectories using multiple expert strategies based on the UAV simulation environment;
[0076] The imitation learning module 102 is used to construct a generative adversarial network, including a strategy generator and multiple discriminators. During the training process, the strategy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into a discriminator, updates the discriminator parameters and obtains the accuracy of the discriminator. According to the accuracy of each discriminator, the reward value is calculated to update the strategy generator. After the training, the trained strategy generator is obtained.
[0077] The control execution module 103 is used to interact with the real environment, collect and splice the state vectors of each drone in the drone cluster, input them into the trained strategy generator, output the action vectors of each drone, convert the action vectors of each drone into instructions and distribute them to the corresponding drones to realize the control of each drone.
[0078] Since the present embodiment and the aforementioned drone control method based on multi-expert imitation learning can be mutually referenced, it is a repeated description here, so it will not be repeated here. Since the present system embodiment and the aforementioned method embodiment have the same principle, the present system embodiment also has the corresponding technical effects of the aforementioned method embodiment.
[0079] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0080] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A drone control method based on multi-expert imitation learning, characterized in that: The following steps are involved: Based on the UAV simulation environment, multiple expert strategies are used to deduce expert trajectories. Construct a generative adversarial network, including a strategy generator and multiple discriminators; the number of the discriminators is the same as the number of expert strategies; during the training process, the strategy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into different discriminators respectively, updates the discriminator parameters and obtains the accuracy of the discriminator; according to the accuracy of each discriminator, calculates the reward value to update the strategy generator; After the training is completed, the trained strategy generator is obtained; Interact with the real environment, collect and splice the state vectors of each drone in the drone cluster, input them into the trained strategy generator, output the action vectors of each drone, convert the action vectors of each drone into instructions and distribute them to the corresponding drones to achieve control of each drone; The accuracy of the discriminator is the sum of the expert accuracy and the imitator accuracy; wherein the expert accuracy refers to the probability that the state-action pair in the expert trajectory is judged to be correct by the discriminator, and the imitator accuracy refers to the probability that the state-action pair in the imitator trajectory is judged to be correct by the discriminator; For the state-action pair in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has made a correct judgment; For the state-action pair in the imitator's trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has made a correct judgment; The reward value is calculated according to the accuracy of each discriminator, including: taking the ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators as the expert reliability coefficient of each discriminator, and calculating the reward value according to the expert reliability coefficient of each discriminator and the output corresponding to the imitator trajectory in each discriminator by the following formula: Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the discriminator corresponding to the jth expert trajectory during the i-th round of training; It represents the discriminant value output by the discriminator corresponding to the j-th expert trajectory for the state-action pair (s, a) in the imitator trajectory during the i-th round of training; s represents the state vector in the trajectory, and a represents the action vector in the trajectory.
2. The drone control method based on multi-expert imitation learning according to claim 1 is characterized in that: The expert strategy is a reinforcement learning model obtained by training different numbers of rounds using the PPO algorithm; the reinforcement learning model includes a policy network and a value network, wherein the policy network is used to output a drone action vector in a two-dimensional plane area according to an input drone state vector, and the value network is used to estimate the value function value of the input drone state vector.
3. The drone control method based on multi-expert imitation learning according to claim 2 is characterized in that: The drone state vector includes: the horizontal and vertical speed of the current drone, the horizontal and vertical coordinates of the current drone, the horizontal and vertical coordinates of the target, the horizontal and vertical coordinates of other drones, and the communication status between the current drone and other drones; the drone action vector includes: the horizontal positive speed of the current drone, the vertical positive speed of the current drone, the horizontal negative speed of the current drone, the vertical negative speed of the current drone and the communication strength.
4. The drone control method based on multi-expert imitation learning according to claim 1 is characterized in that: The updating of the discriminator parameters is performed by calculating the gradient according to the following formula: in, represents the computational expectation, represents the calculation gradient, w(j) i represents the discriminator parameters corresponding to the jth expert trajectory during the i-th round of training, T i represents the imitator trajectory sampled during the i-th round of training, T E represents the expert trajectory, They represent the discriminator corresponding to the jth expert trajectory during the i-th round of training. For the imitator state-action pair (s, a), the expert state pair (s, a) E Output discriminant value.
5. The drone control method based on multi-expert imitation learning according to claim 2 is characterized in that: The reward value of the expert strategy includes: a distance reward and a collision reward; the distance reward is the negative of the distance between the drone and the target, and the collision reward is the negative of the number of collisions that occur with the drone.
6. A UAV control system based on multi-expert imitation learning, characterized in that: include: The expert trajectory acquisition module is used to deduce expert trajectories based on the UAV simulation environment using multiple expert strategies; An imitation learning module, used for constructing a generative adversarial network, comprising a strategy generator and a plurality of discriminators; the number of the discriminators is the same as the number of the expert strategies; During the training process, the strategy generator interacts with the UAV simulation environment, samples the imitator trajectory, inputs the imitator trajectory and each expert trajectory into different discriminators, updates the discriminator parameters and obtains the accuracy of the discriminator; according to the accuracy of each discriminator, calculates the reward value to update the strategy generator; After the training is completed, the trained strategy generator is obtained; The control execution module is used to interact with the real environment, collect and splice the state vectors of each drone in the drone cluster, input them into the trained strategy generator, output the action vectors of each drone, convert the action vectors of each drone into instructions and distribute them to the corresponding drones to realize the control of each drone; In the imitation learning module, the accuracy of the discriminator is the sum of the expert accuracy and the imitator accuracy; wherein the expert accuracy refers to the probability that the state-action pair in the expert trajectory is judged to be correct by the discriminator, and the imitator accuracy refers to the probability that the state-action pair in the imitator trajectory is judged to be correct by the discriminator; For the state-action pair in the expert trajectory, when the discriminant value output by the discriminator is greater than or equal to 0.5, it is considered that the discriminator has made a correct judgment; For the state-action pair in the imitator's trajectory, when the discriminant value output by the discriminator is less than 0.5, it is considered that the discriminator has made a correct judgment; In the imitation learning module, the reward value is calculated according to the accuracy of each discriminator, including: taking the ratio of the accuracy of each discriminator to the sum of the accuracy of all discriminators as the expert reliability coefficient of each discriminator, and calculating the reward value according to the expert reliability coefficient of each discriminator and the output corresponding to the imitator trajectory in each discriminator by the following formula: Among them, r i represents the reward value during the i-th round of training; n represents the number of expert trajectories, represents the expert reliability coefficient of the discriminator corresponding to the jth expert trajectory during the i-th round of training; It represents the discriminant value output by the discriminator corresponding to the j-th expert trajectory for the state-action pair (s, a) in the imitator trajectory during the i-th round of training; s represents the state vector in the trajectory, and a represents the action vector in the trajectory.
Citation Information
Patent Citations
Method for selecting reward function in adversarial imitation learning
CN111401556A
Mechanical arm complex operation skill learning method and system based on generative adversarial imitation learning
CN117733841A
Computer-implemented method of and apparatus for training a neural network
EP3742344A1
Architecture for explainable reinforcement learning
US20220147876A1
Highly humanoid safe-driving decision-making method for autonomous commercial vehicle
WO2023155231A1
