Unmanned aerial vehicle control method and device based on robust reinforcement learning and adversarial reinforcement learning
By combining robust reinforcement learning and adversarial reinforcement learning, the performance of the UAV controller is optimized, and the problems of high cost of deep reinforcement learning and poor immunity are solved, and stable control is achieved in complex environments.
Patent Information
- Application Number
- CN202410984216.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, when deep reinforcement learning is used for UAV control, the training cost is high and the risk is high. The modeling of uncertain disturbances in the simulation environment is different from the real environment, resulting in the training controller in extreme cases. The immunity of the disturbance is poor and it is difficult to meet the control needs of complex environments.
Using a method based on robust reinforcement learning and adversarial reinforcement learning, the controller performance is optimized through the combination of robust partial control volume and reinforcement learning controller and adversarial network, and the robust control module is used to provide basic performance in the early stage of training, and through adversarial The network applies interference in a timely manner during training to improve its immunity.
It reduces training time overhead, improves the performance and immunity of the controller, reduces the computational complexity, and ensures stable control performance in complex environments.
Smart Images

Figure CN119024878A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method and device for controlling an unmanned aerial vehicle based on robust reinforcement learning and adversarial reinforcement learning. Background Art
[0002] Quad-rotor drones have the characteristics of high maneuverability, hovering, and simple structure. They have been widely used in many fields such as military, disaster relief, surveying and mapping, and agriculture. Considering the various uncertainties in the real environment, such as external wind disturbance, environmental changes, communication delays, etc., the drone controller is required to have the ability to optimize autonomously to adapt to complex environments, while traditional control methods are difficult to meet control requirements in complex situations. In recent years, deep reinforcement learning has achieved success in many fields. Deep reinforcement learning is a method that actively interacts with the environment without human intervention and learns through trial and error. Using deep reinforcement learning for drone control can enable the controller to have autonomous optimization capabilities, offset system uncertainties such as disturbances through training, and improve control performance.
[0003] Considering the cost and safety of system training, the direct use of deep reinforcement learning may require a lot of training and trial and error, so the cost is high and the risk is high. It is necessary to use known system information and combine it with a basic controller with basic control capabilities to provide basic performance guarantees for the system in the early stages of training. Considering the input delay problem caused by calculation delay, actuator delay, and communication delay in the real environment, the traditional deep reinforcement learning control method is difficult to deal with the delay problem, and the deep reinforcement learning method needs to be improved to offset the impact of the delay. In addition, considering that the modeling of uncertain disturbances in the simulation environment is likely to be different from the real environment, the trained controller has poor anti-disturbance ability and cannot meet the control performance of some extreme situations in the real environment. Therefore, a more reasonable training method is needed to improve the anti-disturbance ability of the controller. Summary of the invention
[0004] In order to solve the technical problem that the modeling of uncertain disturbances in the simulation environment in the prior art is likely to be different from the real environment, resulting in poor anti-disturbance ability of the trained controller and failure to meet the control performance of some extreme situations in the real environment, the embodiment of the present invention provides a UAV control method and device based on robust reinforcement learning and adversarial reinforcement learning. The technical solution is as follows:
[0005] On the one hand, a UAV control method based on robust reinforcement learning and adversarial reinforcement learning is provided, and the method includes:
[0006] S1. Initialize the drone system and obtain the current status of the drone;
[0007] S2. Based on the current state, the robust partial control amount of the UAV is calculated, and based on the robust partial control amount, it is determined whether the reinforcement learning control needs to be triggered. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount;
[0008] S3, input accurate control quantity into the UAV system and get reward;
[0009] S4. Use the reward, action and accurate control amount as experience to update the reinforcement learning controller and the adversarial network, and determine whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0010] Optionally, in S1, the UAV system is initialized and the current state of the UAV is obtained, including:
[0011] Initialize the evaluation network and action network of the drone's reinforcement learning control module;
[0012] Initialize the robust control module parameters;
[0013] The current state x(t) is collected from the environment.
[0014] Optionally, in S2, based on the current state, the robust partial control amount of the UAV is calculated, and based on the robust partial control amount, it is determined whether the reinforcement learning control needs to be triggered. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount, including:
[0015] Input the current state x(t) into the robust control module to obtain the control quantity u of the current robust control module r (t);
[0016] Determine whether to trigger reinforcement learning control based on the robust partial control quantity:
[0017] If so, get the reinforcement learning action u of the previous step from the action buffer RL (t-1), get the current extended state x E (t)=[x(t),U RL (t-1)]; the current extended state x E (t) Input to the policy network of the reinforcement learning control module to obtain the current action u of the reinforcement learning control module RL (t), and give action u RL (t) Add normal distribution noise; change the current extended state sE (t) Input to the policy network of the adversarial reinforcement learning part to obtain the output d of the current adversarial reinforcement learning part anti (t), and add normal distribution noise; the accurate control quantity is u(t)=u r (t)+u RL (t);
[0018] If not, the robust part of the control quantity is directly used as the accurate control quantity.
[0019] Optionally, the triggering time interval is a control period of the reinforcement learning control module, and the control period is set to be greater than or equal to the delay time.
[0020] Optionally, S2 further includes:
[0021] The output d of adversarial reinforcement learning anti (t) Each channel is randomly multiplied by 1 or -1 to make the adversarial perturbation symmetric.
[0022] Optionally, in S3, the accurate control amount is input into the UAV system to obtain a reward, including:
[0023] Input accurate control quantities into the UAV system;
[0024] The next state collected from the environment is x(t+1), and the next extended state is calculated as x E (t+1)=[x(t+1),u RL (t)], calculate the instant reward r(t) of the reinforcement learning control module and the instant reward r of the adversarial reinforcement learning anti (t);
[0025] Collect the current expansion status S E (t), the next extended state x E (t+1), the action u of the current reinforcement learning control module RL (t), the output d of adversarial reinforcement learning anti (t), the immediate reward r(t) of the reinforcement learning control module and the immediate reward r of the adversarial reinforcement learning anti (t), stored in the experience replay pool.
[0026] Optionally, in S3, the rewards, actions, and accurate control amounts are used as experience to update the reinforcement learning controller and the adversarial network, and determine whether the reinforcement learning controller and the adversarial network have converged. If not, return to S2; if so, save the network parameters, and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning, including:
[0027] Collect a batch of rewards, actions, and accurate control amounts as experience from the experience replay pool, update the evaluation network and action network of the reinforcement learning controller, and soft-update the target evaluation network and target action network of the reinforcement learning controller; update the evaluation network and action network of the adversarial network, and soft-update the target evaluation network and target action network of the adversarial network;
[0028] Determine whether the reinforcement learning controller and the adversarial network have converged. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0029] On the other hand, a UAV control device based on robust reinforcement learning and adversarial reinforcement learning is provided, and the device is applied to a UAV control method based on robust reinforcement learning and adversarial reinforcement learning, and the device includes:
[0030] The initialization module is used to initialize the drone system and obtain the current status of the drone;
[0031] The reinforcement learning control module is used to calculate the robust partial control amount of the UAV based on the current state, and judge whether the reinforcement learning control needs to be triggered based on the robust partial control amount. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount;
[0032] The experience replay module is used to input accurate control quantities into the UAV system and obtain rewards;
[0033] The network update and control module is used to update the reinforcement learning controller and the adversarial network using rewards, actions, and accurate control quantities as experience, and to determine whether the reinforcement learning controller and the adversarial network have converged. If not, it returns to the reinforcement learning control module; if so, it saves the network parameters and completes the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0034] On the other hand, a drone control device based on robust reinforcement learning and adversarial reinforcement learning is provided, and the drone control device based on robust reinforcement learning and adversarial reinforcement learning includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned drone control methods based on robust reinforcement learning and adversarial reinforcement learning is implemented.
[0035] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned drone control methods based on robust reinforcement learning and adversarial reinforcement learning.
[0036] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0037] The method provided by the present invention uses a robust reinforcement learning method in the controller, which can optimize the controller through continuous iteration, improve the controller performance, and offset external disturbances. At the same time, it also reduces the training time overhead and ensures the safety of the early stage of training. The reinforcement learning method is improved for the input delay problem. By extending the state and extending the control cycle method, the one-step historical controller information is used to reconstruct the extended state without significantly increasing the computational complexity to solve the delay problem. The present invention uses an adversarial reinforcement learning method. Adversarial reinforcement learning plays a game with the target strategy by attacking the target strategy and reducing the performance of the target. For drone systems, this zero-sum game is likely to cause the controller to fail in the early stage of training due to excessive disturbances. Therefore, by designing the reward function of the adversarial network, appropriate interference is applied to the reinforcement learning control module at different stages of training to improve the anti-interference ability of the reinforcement learning control module. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0039] Figure 1 This is a flow chart of a drone control method using robust reinforcement learning and adversarial reinforcement learning provided by an embodiment of the present invention;
[0040] Figure 2 It is an overall framework diagram provided by an embodiment of the present invention;
[0041] Figure 3 is a reward curve diagram of the reinforcement learning control module provided by an embodiment of the present invention;
[0042] Figure 4 is a reward curve diagram of the adversarial network provided by an embodiment of the present invention;
[0043] Figure 5 It is a block diagram of a drone control device for robust reinforcement learning and adversarial reinforcement learning provided by an embodiment of the present invention;
[0044] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0046] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0047] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0048] In the embodiments of the present invention, sometimes a subscript such as W1 may be expressed in a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.
[0049] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0050] The embodiment of the present invention provides a method for controlling a drone based on robust reinforcement learning and adversarial reinforcement learning. The method can be implemented by a drone control device based on robust reinforcement learning and adversarial reinforcement learning. The drone control device based on robust reinforcement learning and adversarial reinforcement learning can be a terminal or a server. Figure 1 The flowchart of the UAV control method based on robust reinforcement learning and adversarial reinforcement learning is shown in the figure. Figure 2 The overall framework diagram provided by the embodiment of the present invention is shown in FIG. 1 . The processing flow of the method may include the following steps:
[0051] S1. Initialize the drone system and obtain the current status of the drone;
[0052] In a new implementation manner, in S1, the UAV system is initialized and the current state of the UAV is obtained, including:
[0053] Initialize the evaluation network and action network of the drone's reinforcement learning control module;
[0054] Initialize the robust control module parameters;
[0055] The current state x(t) is collected from the environment.
[0056] In a feasible implementation, the quad-rotor drone is firstly system-modeled as follows:
[0057]
[0058] where p x (t), p y (t), p z (t) θ(t) and ψ(t) represent the tracking errors in the x, y, z directions and the pitch, roll, and yaw angles, respectively. d1, d2, d3, d4, d5, and d6 represent uncertainty disturbances. m represents the mass of the drone, I x ,I y ,I z represents the moment of inertia of the drone on the x, y, and z axes, and τ is the input delay. F1, F2, F3, and F4 are control signals, and their relationship with the actual control values of the motors u1, u2, u3, and u4 is as follows:
[0059]
[0060] In order to facilitate the design of robust control modules, the quadrotor model can be expressed as the following state equation:
[0061]
[0062] The whole system is divided into position loop subsystem and attitude loop subsystem, where x p (t), x a (t) are the current position error and attitude error, M p (x(t)),M a (x(t)) is the nonlinear part, q p (x(t)),q a (x(t)) is the uncertainty disturbance, where
[0063]
[0064] The two subsystems have the same matrix form but different parameters, where C p1 =0, C p2 =0, C p3 =-g, d p1 =d1,d p2 =d2,d p3 =d3,d a1 =d4,d a2 =d5,d a3 =d6.
[0065] The external environment only gives the target values of x, y, z and yaw angle ψ, so the error x of the position subsystem can be directly calculated. p (t). For the position loop subsystem, its control signal F p (t-τ)=[F x (t-τ),F y (t-τ),F4(t-τ)] T , where F x (t-τ) and F y (t-τ) is the virtual control signal in the x and y directions, which is used to inversely solve the formula to obtain the target state of the pitch angle and roll angle, as shown below:
[0066]
[0067] The target states of pitch and roll angles obtained will be used to calculate the attitude error x in the attitude loop subsystem. a (t), the control signal of the attitude loop subsystem is F a (t-τ)=[F1(t-τ), F2(t-τ), F3(t-τ)] T Finally, the actual motor control quantities u1, u2, u3, and u4 can be obtained according to the control signals F1, F2, F3, and F4.
[0068] S2. Based on the current state, the robust partial control quantity of the UAV is calculated, and based on the robust partial control quantity, it is determined whether reinforcement learning control needs to be triggered. If so, the robust partial control quantity is expanded, and the accurate control quantity is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control quantity is directly used as the accurate control quantity.
[0069] In a new implementation, in S2, based on the current state, the robust partial control amount of the UAV is calculated, and based on the robust partial control amount, it is determined whether reinforcement learning control needs to be triggered. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount, including:
[0070] Input the current state x(t) into the robust control module to obtain the control quantity u of the current robust control module r (t);
[0071] Determine whether to trigger reinforcement learning control based on the robust partial control quantity:
[0072] If so, get the reinforcement learning action u of the previous step from the action buffer RL (t-1), get the current extended state x E (t) = [x(t), u RL(t-1)]; the current extended state x E (t) Input to the policy network of the reinforcement learning control module to obtain the current action u of the reinforcement learning control module RL (t), and give action u RL (t) Add normal distribution noise; change the current extended state s E (t) Input to the policy network of the adversarial reinforcement learning part to obtain the output d of the current adversarial reinforcement learning part anti (t), and add normal distribution noise; the accurate control quantity is u(t)=u r (t)+u RL (t);
[0073] If not, the robust part of the control quantity is directly used as the accurate control quantity.
[0074] In a new implementation manner, the triggering time interval is a control period of the reinforcement learning control module, and the control period is set to be greater than or equal to the delay time.
[0075] In a feasible implementation, the robust control module has the same structure in the attitude loop subsystem and the position loop subsystem, and the subsystems are connected by inverse formulas. The robust control module includes a feedback part and a filtering part. The feedback part ensures the basic control performance of the controller, and the filtering part is used to offset the error introduced by the uncertainty disturbance and input delay. Its structure is as follows:
[0076]
[0077] The filter f is the filter coefficient. By calculating the control signals of the position loop subsystem and the attitude loop subsystem in sequence, we get {F1, F2, F3, F4}, and then through the virtual control signal to motor control quantity conversion formula, we can finally get the control quantity u of the robust control module. r .
[0078] In a feasible implementation, for the traditional reinforcement learning method, the current action u(t) is obtained from the current state x(t), that is, u(t) = μ(x(t)). However, due to the input delay τ, the action u(t) of the reinforcement learning control module at time t will actually act on the system at time t+τ, that is, act on the state x(t+τ). Therefore, when there is an input delay, the current action is not only related to the current state, but also to the historical actions, that is, the reinforcement learning control module should satisfy u(t) = μ(x(t), u(t-τ), u(t-τ+1), ..., u(t-1)).
[0079] In order to deal with input delay, the extended state method is adopted to combine the historical action information and the current state into an extended state, that is, xE (t) = [x(t), u(t-τ), u(t-τ+1), ..., u(t-1)], and the extended state is input into the neural network of the reinforcement learning control module, that is, u(t) = μ(x E (t)).
[0080] Directly using the extended state method requires multiple historical action information, which results in the extended state dimension being too large, which in turn results in the input parameters of the reinforcement learning neural network being too large, increasing the computational burden. Using the extended control cycle method, the control cycle of the reinforcement learning control module is set to be greater than or equal to the maximum delay time, that is, k ≥ τ, where k represents the control cycle of the reinforcement learning control module. It is triggered once in each control cycle, and the action of the reinforcement learning control module is calculated when triggered. At other times, a first-order retainer is used to keep the control output unchanged, that is, u(tk) = u(t-k+1) = ... = u(t-1). Using this extended control cycle method, the extended state only requires one historical action, that is, x E (t) = [x(t), u(tk)], which avoids the impact caused by too large input dimension.
[0081] In a new implementation manner, S2 further includes:
[0082] The output d of adversarial reinforcement learning anti (t) Each channel is randomly multiplied by 1 or -1 to make the adversarial perturbation symmetric.
[0083] In a feasible implementation, the reinforcement learning control module adopts the extended state plus extended control cycle method to solve the input delay problem. It mainly includes two steps:
[0084] Step 1: Extended state, that is, input the current state and historical action information into the reinforcement learning control module as extended state. The historical actions contain timing information, so the extended state method can handle the delay problem.
[0085] Step 2: Extend the control cycle, that is, set the control cycle of the reinforcement learning control module to be greater than or equal to the maximum input delay time. This setting ensures that only one step of historical action information is used when the state is extended. This prevents the neural network input dimension of the reinforcement learning control module from being too large due to too much historical action information in the extended state, thereby increasing the computational burden.
[0086] S3, input accurate control quantity into the UAV system and get reward;
[0087] In a new implementation mode, in S3, inputting the accurate control amount into the UAV system to obtain a reward includes:
[0088] Input accurate control quantities into the UAV system;
[0089] The next state collected from the environment is x(t+1), and the next extended state is calculated as x E (t+1)=[x(t+1),u RL (t)], calculate the instant reward r(t) of the reinforcement learning control module and the instant reward r of the adversarial reinforcement learning anti (t);
[0090] Collect the current extension status E (t), the next extended state x E (t+1), the action u of the current reinforcement learning control module RL (t), the output t of adversarial reinforcement learning anti (t), the immediate reward r(t) of the reinforcement learning control module and the immediate reward r of the adversarial reinforcement learning anti (t), stored in the experience replay pool.
[0091] In a feasible implementation, the reinforcement learning control module adopts an end-to-end approach to directly obtain the final motor control variables u1, u2, u3, and u4 based on the errors of x, y, z, and the yaw angle ψ. The reinforcement learning control module adopts the DDPG algorithm. DDPG has four neural networks, namely the Actor Neural Network, the TargetActor Neural Network, the Critic Neural Network, and the TargetCritic Neural Network. The parameters corresponding to the four networks are θ u ,θ′ u ,θ Q and θ′ Q .
[0092] The control amount of the reinforcement learning control module is obtained from the action network, that is,
[0093] u RL (t) = μ(x E (t)|θ u )
[0094] For the reinforcement learning control module, the expression of the reward function is
[0095]
[0096] Among them, x(t) represents the error between the drone state and the target position, u RL(t) is the output of the reinforcement learning control module. The required drone performance can be obtained by setting the weight matrices W and R. The reward function r(t) expects the controller to make the tracking error approach 0 with the smallest possible energy consumption.
[0097] All four neural networks of reinforcement learning use two-layer fully connected networks, and the activation function uses the tanh function. The evaluation network and the target evaluation network are responsible for outputting evaluation indicators based on the input expansion state and action. The action network and the target action network are responsible for outputting actions based on the input expansion state. During the training process, the evaluation network and the action network update the weights according to the loss function, and the weights of the target evaluation network and the target action network are soft-updated and regularly updated from the evaluation network and the action network. The final quadcopter system includes the following control quantities:
[0098] u(t)=u RL (t)+u r (t)
[0099] The adversarial network part in adversarial reinforcement learning also uses the DDPG algorithm and has exactly the same network structure and update method as the reinforcement learning control module, so I will not go into details about network design and network update. The only difference between the two is the setting of the reward function.
[0100] The reward function is used to guide reinforcement learning to complete a specific task. For adversarial reinforcement learning, the expression of the reward function is
[0101] r anti (t)=-r(t)-c‖d anti (t)‖ 2
[0102] Based on the idea of adversarial game, the reward function consists of two parts. The first part -r(t) expects the output of the adversarial network to make the reward of the reinforcement learning control module as small as possible to achieve the purpose of interference. However, in the early stage of reinforcement learning training, since the network has not converged yet, if too much interference is applied at this time, it will make it difficult for the reinforcement learning control module network to converge or even diverge. So the second term -c||d anti (t)|| 2 The size of the disturbance is constrained, and the greater the output disturbance, the lower the reward. Compared with giving fixed disturbances or random disturbances, the size of the disturbance output by the adversarial network is determined based on the training of the controller network. This setting can ensure that a smaller disturbance is applied in the early stage of training to ensure that the reinforcement learning control module network can converge, and it can also ensure that the largest possible disturbance is applied in the later stage of training to improve the anti-disturbance ability of the reinforcement learning control module network.
[0103] After this setting, in the early stage of training, since the reinforcement learning control module has not been fully trained, its control performance is poor and the reward r(t) is small, so the adversarial network does not need to output too much interference to obtain a higher reward. The second term of the reward function -c||d anti (t)|| 2 As the reinforcement learning control module converges, its control performance gets better and better, and the reward r(t) also increases. anti (t), it is necessary to impose greater interference on the system to reduce r(t). At this time, the constraint effect of the second term of the reward function is reduced, and the effect of the first term is more obvious.
[0104] Through such a setting, the adversarial network can not only avoid the divergence of the control network caused by excessive disturbance in the early stage, but also output a large amount of disturbance in the later stage to simulate the extreme environment and improve the anti-disturbance ability of the controller network. The output of the adversarial network is obtained from its action network, that is,
[0105]
[0106] In addition, this application does not preset the modeling of the interference, but directly expresses it as the impact on the output of the four motors. This setting can simulate the interference of the working situation as much as possible, such as wind disturbance, actuator failure, inaccurate modeling, etc. Therefore, the disturbance of the adversarial network output will be directly added to the output of the controller.
[0107] In a feasible implementation, the controller combines a robust control module and a reinforcement learning control module. The robust control module has a simple structure and can ensure the basic control performance of the UAV system. Considering that the reinforcement learning control module cannot function in the early stage of training and may even introduce network errors, the robust control module can offset the uncertainty and this part of the error in the early stage of training. The reinforcement learning control module is based on the deep reinforcement learning method, using a policy network, a target policy network, an action network, and a target action network. It only needs the flight information obtained in the training phase to autonomously optimize the control performance. For the reinforcement learning control module, the required UAV performance can be obtained by setting the weight matrices W and R. The reward function r(t) expects the controller to make the tracking error approach 0 with the lowest possible energy consumption.
[0108] S4. Use the reward, action and accurate control amount as experience to update the reinforcement learning controller and the adversarial network, and determine whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0109] In a new implementation method, in S4, the rewards, actions, and accurate control amounts are used as experience to update the reinforcement learning controller and the adversarial network, and determine whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; if so, save the network parameters, and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning, including:
[0110] Collect a batch of rewards, actions, and accurate control amounts as experience from the experience replay pool, update the evaluation network and action network of the reinforcement learning controller, and soft-update the target evaluation network and target action network of the reinforcement learning controller; update the evaluation network and action network of the adversarial network, and soft-update the target evaluation network and target action network of the adversarial network;
[0111] Determine whether the reinforcement learning controller and the adversarial network have converged. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0112] In a feasible implementation, Figure 3 is a reward curve diagram of the reinforcement learning control module provided in an embodiment of the present invention; Figure 4 It is a reward curve diagram of the adversarial network provided by an embodiment of the present invention.
[0113] In a feasible implementation, the evaluation network of the reinforcement learning control module is updated, and the update formula is as follows:
[0114] Q target =r(t)+γQ′(x E (t+1),u′(x E (t+1)|θ′ u )|θ′ Q )
[0115]
[0116] Update the action network of the reinforcement learning control module. The update formula is as follows:
[0117]
[0118] The target evaluation network and target action network of the reinforcement learning control module are soft updated. The update formula is as follows:
[0119] θ′ Q ←εθ Q +(1-ε)θ′ Q
[0120] θ′ u ←εθ u +(1-ε)θ′u
[0121] Update the evaluation network of the adversarial network. The update formula is as follows:
[0122]
[0123] Update the action network of the adversarial network, and the update formula is as follows:
[0124]
[0125] The target evaluation network and target action network of the adversarial network are soft updated, and the update formula is as follows:
[0126]
[0127] In an embodiment of the present invention, a robust deep reinforcement learning method is used to address the problem of high training cost and high risk of deep reinforcement learning. The robust reinforcement learning controller includes a robust control module and a reinforcement learning control module, wherein the robust control module is constructed based on known system information, and can ensure basic control performance and offset disturbances in the early stage of training the reinforcement learning control module. On this basis, the reinforcement learning control module can optimize the controller performance according to the performance indicator function. Therefore, the robust reinforcement learning method can not only ensure that the optimal strategy is obtained in training, but also reduce the training time overhead and ensure the safety in the early stage of training.
[0128] Aiming at the problem of input delay in the system, the extended state method and extended control cycle method are used to improve the reinforcement learning control module. Compared with other methods, this method has a simple structure and a small computational burden.
[0129] In order to solve the problem that the controller trained in the simulation environment is difficult to meet the control performance in the actual environment, the adversarial reinforcement learning method is adopted to introduce the interference network to play a zero-sum game with the controller to improve the anti-interference ability of the controller. Considering that this zero-sum game is likely to cause the controller to fail due to excessive disturbance in the early stage of training. The reward function of the adversarial network is designed so that the interference output in the early stage of training is small to avoid the failure of the controller network to converge. Then, a larger interference is output for training to improve the anti-interference ability of the controller.
[0130] Figure 5 The invention is a block diagram of a drone control device based on robust reinforcement learning and adversarial reinforcement learning according to an exemplary embodiment. The device is used in a drone control method based on robust reinforcement learning and adversarial reinforcement learning. Figure 5 The device includes: an initialization module 310, a reinforcement learning control module 320, an experience playback module 330 and a network update and control module 340. Among them:
[0131] Initialization module 310, used to initialize the drone system and obtain the current state of the drone;
[0132] The reinforcement learning control module 320 is used to calculate the robust partial control amount of the UAV based on the current state, and determine whether the reinforcement learning control needs to be triggered based on the robust partial control amount. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount;
[0133] The experience playback module 330 is used to input accurate control quantities into the UAV system and obtain rewards;
[0134] The network update and control module 340 is used to update the reinforcement learning controller and the adversarial network using rewards, actions, and accurate control quantities as experience, and to determine whether the reinforcement learning controller and the adversarial network have converged. If not, it returns to the reinforcement learning control module; if so, it saves the network parameters and completes the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0135] Optionally, an initialization module 310 is used to initialize an evaluation network and an action network of a reinforcement learning control module of the drone;
[0136] Initialize the robust control module parameters;
[0137] The current state x(t) is collected from the environment.
[0138] Optionally, the reinforcement learning control module 320 is used to
[0139] Input the current state x(t) into the robust control module to obtain the control quantity u of the current robust control module r (t);
[0140] Determine whether to trigger reinforcement learning control based on the robust partial control quantity:
[0141] If so, get the reinforcement learning action u of the previous step from the action buffer RL (t-1), get the current extended state x E (t) = [x(t), u RL (t-1)]; the current extended state x E (t) Input to the policy network of the reinforcement learning control module to obtain the current action u of the reinforcement learning control module RL (t), and give action u RL (t) Add normal distribution noise; change the current extended state s E(t) Input to the policy network of the adversarial reinforcement learning part to obtain the output d of the current adversarial reinforcement learning part anti (t), and add normal distribution noise; the accurate control quantity is u(t)=u r (t)+u RL (t);
[0142] If not, the robust part of the control quantity is directly used as the accurate control quantity.
[0143] Optionally, the triggering time interval is a control period of the reinforcement learning control module, and the control period is set to be greater than or equal to the delay time.
[0144] Optionally, the reinforcement learning control module 320 is used to convert the output d of the adversarial reinforcement learning anti (t) Each channel is randomly multiplied by 1 or -1 to make the adversarial perturbation symmetric.
[0145] Optionally, an experience playback module 330 is used to input accurate control quantities into the UAV system;
[0146] The next state collected from the environment is x(t+1), and the next extended state is calculated as x E (t+1)=[x(t+1),u RL (t)], calculate the instant reward r(t) of the reinforcement learning control module and the instant reward r of the adversarial reinforcement learning anti (t);
[0147] Collect the current extension status E (t), the next extended state x E (t+1), the action u of the current reinforcement learning control module RL (t), the output d of adversarial reinforcement learning anti (t), the immediate reward r(t) of the reinforcement learning control module and the immediate reward r of the adversarial reinforcement learning anti (t), stored in the experience replay pool.
[0148] Optionally, the network update and control module 340 is used to collect a batch of rewards, actions and accurate control amounts as experience from the experience replay pool, update the evaluation network and action network of the reinforcement learning controller, and soft-update the target evaluation network and target action network of the reinforcement learning controller; update the evaluation network and action network of the adversarial network, and soft-update the target evaluation network and target action network of the adversarial network;
[0149] Determine whether the reinforcement learning controller and the adversarial network have converged. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
[0150] Figure 6 is a schematic diagram of a structure of a drone control device based on robust reinforcement learning and adversarial reinforcement learning provided by an embodiment of the present invention, such as Figure 6 As shown, the drone control device based on robust reinforcement learning and adversarial reinforcement learning may include the above Figure 5 The drone control device based on robust reinforcement learning and adversarial reinforcement learning is shown. Optionally, the drone control device based on robust reinforcement learning and adversarial reinforcement learning 410 may include a first processor 2001.
[0151] Optionally, the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning may further include a memory 2002 and a transceiver 2003 .
[0152] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0153] Combine the following Figure 6 The components of the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning are specifically introduced:
[0154] The first processor 2001 is the control center of the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).
[0155] Optionally, the first processor 2001 can perform various functions of the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.
[0156] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 6 CPU0 and CPU1 are shown in FIG.
[0157] In a specific implementation, as an embodiment, the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning may also include multiple processors, such as Figure 6 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0158] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0159] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be controlled by the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0160] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0161] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 6 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0162] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be controlled by the interface circuit ( Figure 6 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0163] It should be noted that Figure 6 The structure of the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0164] In addition, the technical effects of the drone control device 410 based on robust reinforcement learning and adversarial reinforcement learning can refer to the technical effects of the drone control method based on robust reinforcement learning and adversarial reinforcement learning described in the above method embodiment, and will not be repeated here.
[0165] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0166] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0167] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0168] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0169] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0170] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0171] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0173] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0176] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.
[0177] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A UAV control method based on robust reinforcement learning and adversarial reinforcement learning, characterized in that: The method comprises: S1. Initialize the drone system and obtain the current status of the drone; S2. Based on the current state, the robust partial control amount of the UAV is calculated, and based on the robust partial control amount, whether the reinforcement learning control needs to be triggered is determined. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount; S3, inputting the accurate control amount into the UAV system and obtaining a reward; S4. Use the reward, action and accurate control amount as experience to update the reinforcement learning controller and the adversarial network, and determine whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; if so, save the network parameters and complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
2. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 1 is characterized in that: In S1, the UAV system is initialized and the current state of the UAV is obtained, including: Initialize the evaluation network and action network of the drone's reinforcement learning control module; Initialize the robust control module parameters; The current state x(t) is collected from the environment.
3. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 2 is characterized in that: In S2, based on the current state, the robust partial control amount of the UAV is calculated, and based on the robust partial control amount, it is determined whether reinforcement learning control needs to be triggered. If so, the robust partial control amount is expanded, and accurate control amounts are outputted through the reinforcement learning controller and the adversarial network respectively; If not, the robust part of the control quantity is directly used as the accurate control quantity, including: Input the current state x(t) into the robust control module to obtain the control quantity u of the current robust control module r (t); Based on the robust partial control amount, it is determined whether reinforcement learning control needs to be triggered: If so, get the reinforcement learning action u of the previous step from the action buffer RL (t-1), get the current extended state x E (t) = [x(t), u RL (t-1)]; the current extended state x E (t) Input to the policy network of the reinforcement learning control module to obtain the current action u of the reinforcement learning control module RL (t), and add normal distribution noise to the action uRL(t); the current extended state s E (t) Input to the policy network of the adversarial reinforcement learning part to obtain the output d of the current adversarial reinforcement learning part anti (t), and add normal distribution noise; the accurate control quantity is u(t)=u r (t)+u RL (t); If not, the robust part of the control quantity is directly used as the accurate control quantity.
4. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 3 is characterized in that: The triggering time interval is the control period of the reinforcement learning control module, and the control period is set to be greater than or equal to the delay time.
5. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 4 is characterized in that: The S2 further includes: The output d of adversarial reinforcement learning anti (t) Each channel is randomly multiplied by 1 or -1 to make the adversarial perturbation symmetric.
6. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 5 is characterized in that: In S3, the accurate control amount is input into the UAV system to obtain a reward, including: inputting the accurate control quantity into the UAV system; The next state collected from the environment is x(t+1), and the next extended state is calculated as x E (t+1)=[x(t+1),u RL (t)], calculate the instant reward r(t) of the reinforcement learning control module and the instant reward r of the adversarial reinforcement learning anti (t); Collect the current expansion status and the next expansion status x E (t+1), the action u of the current reinforcement learning control module RL (t), the output d of adversarial reinforcement learning anti (t), the immediate reward r(t) of the reinforcement learning control module and the immediate reward r of the adversarial reinforcement learning anti (t), stored in the experience replay pool.
7. The drone control method based on robust reinforcement learning and adversarial reinforcement learning according to claim 6 is characterized in that: In S3, the reward, action and accurate control amount are used as experience to update the reinforcement learning controller and the adversarial network, and it is determined whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; If yes, save the network parameters and complete the UAV control based on robust reinforcement learning and adversarial reinforcement learning, including: Collect a batch of rewards, actions and accurate control amounts as experience from the experience replay pool, update the evaluation network and action network of the reinforcement learning controller, and soft-update the target evaluation network and target action network of the reinforcement learning controller; update the evaluation network and action network of the adversarial network, and soft-update the target evaluation network and target action network of the adversarial network; Determine whether the reinforcement learning controller and the adversarial network converge. If not, return to S2; if so, save the network parameters to complete the drone control based on robust reinforcement learning and adversarial reinforcement learning.
8. A drone control device based on robust reinforcement learning and adversarial reinforcement learning, wherein the drone control device based on robust reinforcement learning and adversarial reinforcement learning is used to implement the drone control method based on robust reinforcement learning and adversarial reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The initialization module is used to initialize the drone system and obtain the current status of the drone; A reinforcement learning control module is used to calculate the robust partial control amount of the UAV based on the current state, and determine whether reinforcement learning control needs to be triggered based on the robust partial control amount. If so, the robust partial control amount is expanded, and the accurate control amount is output through the reinforcement learning controller and the adversarial network respectively; if not, the robust partial control amount is directly used as the accurate control amount; An experience playback module, used to input the accurate control amount into the UAV system to obtain a reward; The network update and control module is used to update the reinforcement learning controller and the adversarial network using rewards, actions and accurate control amounts as experience, and to determine whether the reinforcement learning controller and the adversarial network have converged. If not, the module returns to the reinforcement learning control module; if so, the network parameters are saved to complete the UAV control based on robust reinforcement learning and adversarial reinforcement learning.
9. A drone control device based on robust reinforcement learning and adversarial reinforcement learning, characterized in that: The drone control device based on robust reinforcement learning and adversarial reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.
Citation Information
Cited By
Adversarial reinforcement learning training method for robust control of fixed-wing aircraft
CN121115529A
An adversarial reinforcement learning training method for robust control of fixed-wing aircraft
CN121115529B
