A quadrotor unmanned aerial vehicle attitude preset performance control method based on reinforcement learning

Abstract

The application discloses a quad-rotor unmanned aerial vehicle attitude preset performance control method based on reinforcement learning, and comprises the following steps: considering time-varying inertia parameters and external unknown disturbances, establishing an unmanned aerial vehicle attitude dynamics equation; constructing a split-channel preset performance controller, combining preset performance and non-singular fast terminal sliding mode control technology to generate a robust control quantity; designing a reinforcement learning parameter generator, dynamically optimizing 12 time-varying parameter estimates through a time sequence feature extraction network and a residual network; establishing an online reinforcement learning training mechanism, constructing a multi-objective reward function, and optimizing the output of the parameter generator through the multi-objective reward function. The application adopts a reinforcement learning method to replace a traditional adaptive method to estimate time-varying parameters, realizes multi-degree-of-freedom decoupling optimization through a split-channel control architecture, realizes preset performance constraints of tracking errors under time-varying inertia parameter conditions, and significantly improves the dynamic adjustment capability and anti-interference robustness of an attitude system.

Description

Technical Field

[0001] This invention relates to a method for pre-setting the attitude performance of a quadcopter unmanned aerial vehicle (UAV) based on reinforcement learning, belonging to the field of UAV control technology. Background Technology

[0002] In recent years, high-precision attitude control of quadrotor UAVs in complex dynamic environments has faced severe challenges. Traditional pre-defined performance control methods, such as adaptive control based on sliding mode variable structures, can achieve tracking with limited accuracy. However, their fixed-parameter strategies have significant limitations under time-varying inertial parameters, such as abrupt changes in the inertial matrix caused by load variations, and the combined effects of complex disturbances such as aerodynamic disturbances and sensor noise coupling. In existing literature, model-based adaptive control methods heavily rely on accurate dynamic models, making it difficult to cope with real-world scenarios such as time-varying inertial matrices and unknown upper bounds of external disturbances. Furthermore, the tuning of control parameters in traditional methods relies on expert experience, and once the parameters are fixed, they cannot be adjusted adaptively online. This makes it difficult to balance control robustness and dynamic performance, especially in attitude control problems with multi-degree-of-freedom coupling and strong nonlinear characteristics, which can easily lead to amplification of inter-channel interference.

[0003] Deep reinforcement learning technology offers a new solution for UAV attitude control. However, existing methods still face numerous technical bottlenecks. Most reinforcement learning schemes employ a single-objective reward function, focusing only on tracking error penalties and neglecting comprehensive optimization of control energy consumption, parameter stability, and transient performance (such as overshoot and convergence rate). This makes it difficult for the algorithm to balance multi-objective conflicts during training, leading to drastic fluctuations in control input or actuator saturation in practical applications. Regarding network architecture design, traditional fully connected networks have limited feature extraction capabilities and cannot effectively handle time-varying parameter estimation and multi-channel coupling interference, resulting in a large steady-state error standard deviation that fails to meet high-precision control requirements. Furthermore, fixed exploration rates and static experience replay strategies in training mechanisms can easily lead to slow convergence speeds and local optima, especially when sudden torque disturbances or parameter mutations occur, significantly increasing the fluctuation amplitude of control input and threatening system stability. Therefore, to avoid attitude tracking failures in quadrotor UAVs due to time-varying inertial parameters and external disturbances, and to enable UAVs to comprehensively consider multiple performance indicators during attitude tracking, it is necessary to design a control method that allows quadrotor UAVs to stably achieve attitude tracking. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for attitude preset performance control of quadrotor UAV based on reinforcement learning, which solves the attitude control problem under the coupling effect of time-varying inertial parameters and external unknown disturbances. It achieves independent optimization of multiple degrees of freedom through a multi-channel dynamic decoupling architecture, and generates adaptive control parameters in real time based on reinforcement learning, thereby improving anti-interference robustness and parameter stability while meeting preset performance constraints.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for attitude preset performance control of a quadrotor UAV based on reinforcement learning includes the following steps:

[0007] Step 1: Considering time-varying parameters and external unknown disturbances, establish a mathematical model of the quadcopter UAV, define state variables, and establish the control-oriented state equations for the quadcopter UAV.

[0008] Step 2: Select a time-varying preset performance boundary function and introduce an error transformation function. Combine the state equation of the quadcopter UAV to design a multi-channel preset performance attitude controller. The error transformation function ensures that the tracking error of the UAV attitude system meets the preset performance constraints.

[0009] Step 3: Construct a deep reinforcement learning parameter generator, i.e., an Actor network. Determine the input and output of the Actor network based on a channel-based preset performance attitude controller. The input is the UAV's state vector, and the output is the time-varying parameter adjustment amount.

[0010] Step 4: Train the Actor network and the corresponding Critic network based on the online reinforcement learning training mechanism of deep deterministic policy gradient, define a multi-objective reward function, and optimize the output of the Actor network through the multi-objective reward function;

[0011] Step 5: Utilize the Actor network to output the time-varying parameter adjustment amount at the current moment, and fuse it with the time-varying parameter at the previous moment through a smooth update mechanism to obtain the time-varying parameter at the current moment. Substitute this parameter into the sub-channel preset performance attitude controller to obtain the control input of each channel of the UAV at the current moment, thereby realizing the preset performance control of the UAV attitude.

[0012] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0013] 1. This invention constructs a quadcopter UAV attitude model by considering time-varying inertial parameters and external unknown disturbances. By decoupling state variables, a control-oriented sub-channel model is constructed, which improves the controller's adaptability to time-varying environments.

[0014] 2. This invention uses a channel-specific preset performance and non-singular fast mid-terminal sliding mode control technology to design an attitude controller, enabling each channel to independently meet preset performance constraints, avoiding the problem of inter-channel interference amplification in traditional coupled control, and improving attitude tracking stability.

[0015] 3. This invention innovatively designs a reinforcement learning parameter generator, which replaces the traditional adaptive parameter calculation method and improves the control effect of the system. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for pre-setting the attitude performance of a quadrotor UAV based on reinforcement learning, according to the present invention.

[0017] Figure 2 This is a network structure diagram of the reinforcement learning parameter generator of this invention;

[0018] Figure 3 This is a comparison of the roll angle of the quadcopter UAV of the present invention with the tracking effect of traditional methods;

[0019] Figure 4 This is a comparison of the roll angle of the quadcopter UAV of the present invention with the tracking error of the traditional method;

[0020] Figure 5 This is a comparison of the pitch angle of the quadcopter UAV of the present invention with the tracking effect of traditional methods;

[0021] Figure 6 This is a comparison of the pitch angle of the quadcopter UAV of the present invention with the tracking error of the traditional method;

[0022] Figure 7 This is a comparison of the yaw angle of the quadcopter UAV of the present invention with the tracking effect of traditional methods;

[0023] Figure 8 This is a comparison of the yaw angle of the quadcopter UAV of the present invention with the tracking error of the traditional method. Detailed Implementation

[0024] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0025] like Figure 1 As shown, this invention proposes a reinforcement learning-based attitude preset performance control method for quadrotor UAVs. It employs a multi-channel independent control architecture, where each attitude channel (roll, pitch, yaw) generates dynamic error boundaries based on preset performance constraints. A reinforcement learning parameter generator optimizes the controller parameters, thereby achieving the attitude tracking task of the quadrotor UAV. The specific steps are as follows:

[0026] Step 1: Considering time-varying parameters and external unknown disturbances, establish a mathematical model of the quadrotor UAV, define state variables, and establish the control-oriented state equations for the quadrotor UAV. The specific process is as follows:

[0027] S101. For a quadcopter UAV with time-varying parameters, its mathematical model is described as follows:

[0028] ,

[0029] in, Represents the attitude angle of the drone in the inertial frame. , , Let be the system's inertia matrix. The air drag coefficient, External disturbances acting on each channel. , The control torques that represent the roll, pitch, and yaw motions of the drone.

[0030] S102. Establish a control-oriented design model

[0031] Selecting state variables The control-oriented design model is as follows:

[0032] ,

[0033] in, , , , , , , , , .

[0034] Step 2: Select a preset performance function, design a channel-specific non-singular fast terminal sliding mode function and a preset performance attitude controller. Each channel subsystem independently calculates the control quantity, and the tracking error is ensured to meet the preset performance constraints through an error transformation function. The specific process is as follows:

[0035] S201. Select a preset performance function and introduce error transformation.

[0036] The desired performance index is quantitatively described using a performance function, and then a controller is designed to ensure that the performance constraints are met across the entire time domain. A smoothly decreasing positive function is selected. For time-varying performance boundary functions, consider satisfying the following performance constraints:

[0037] ,

[0038] in, , , For a pre-given positive constant, This indicates the preset lower limit of steady-state error. Indicates the upper limit of steady-state error. Limit the minimum convergence rate of the system. Indicates time. , The overshoot suppression parameter satisfies .

[0039] To handle time-varying performance constraints, the tracking error of the UAV attitude system with time-varying parameters is defined. , , ,in, , , They are respectively , , The reference trajectory. Let , To normalize the tracking error, an error transformation function is introduced:

[0040] ,

[0041] For error transformation function Differentiating, we get:

[0042] ,

[0043] in,

[0044] .

[0045] S202. Design a three-channel non-singular fast terminal sliding mode function and a three-channel attitude control law for a quadrotor UAV.

[0046] The control design process is illustrated below using the roll angle subsystem as an example. The non-singular fast terminal sliding mode function of the roll angle subsystem is designed as follows:

[0047] ,

[0048] in, , All are constants greater than zero. and All are positive odd numbers, and satisfy the following conditions: , ,right Taking the derivative, we get:

[0049] ,

[0050] Substituting the control-oriented design model, we get:

[0051] ,

[0052] To handle time-varying parameters , , ,Will , , Separate the constant terms , , and time-varying error term , , ,Right now , , The above constant terms are estimated using reinforcement learning methods, and a sliding mode controller is designed to suppress the above time-varying error terms.

[0053] The specific design of the roll angle subsystem controller is as follows:

[0054] ,

[0055] ,

[0056] .

[0057] The non-singular fast terminal sliding mode functions for the pitch and yaw subsystems are as follows:

[0058] ,

[0059] ,

[0060] in, , , and All parameters are greater than zero.

[0061] The design of the pitch angle subsystem controller is as follows:

[0062] ,

[0063] ,

[0064] ;

[0065] The specific design of the yaw angle subsystem controller is as follows:

[0066] ,

[0067] ,

[0068] ,

[0069] in, , , They are The estimated value, , , To design the control gain, , , They are , , The estimated value, , , They are , , The estimated value, , , They are , , The estimated value, yes The separated constant term, A positive time-varying integrable function that satisfies: , , and They are respectively The upper and lower bounds, express: .

[0070] Step 3: Construct a deep reinforcement learning parameter generator, design a 27-dimensional input state vector, and dynamically optimize the estimated values ​​of 12 time-varying parameters through a temporal feature extraction network and a residual network.

[0071] The deep reinforcement learning parameter generator, i.e., the policy network (Actor network), has the following structure: Figure 2 As shown, the network input is a 27-dimensional state vector, which is composed of normalized real-time attitude angles. angular velocity Three-channel tracking error and its differential Historical error moving average integral and the current parameter estimates constitute.

[0072] The normalized 27-dimensional state vector is then standardized sequentially through a learnable layer normalization module. This module independently calculates the mean and variance for each feature dimension of the input data, achieving standardization through affine transformation. The standardized data is then input into a temporal feature extraction network, which consists of a fully connected layer (27 input dimensions, 64 output dimensions), a GELU activation function, a layer normalization layer, a second fully connected layer (both input and output dimensions are 64), another GELU activation function, and another layer normalization layer, all concatenated in series. The output of the temporal feature extraction network is connected to a residual module consisting of three residual blocks. Each residual block contains, sequentially, a fully connected layer (64 input dimensions, 128 output dimensions), a GELU activation function, a layer normalization layer, another fully connected layer (128 input dimensions, 64 output dimensions), another GELU activation function, and another layer normalization layer. The output of the residual module is mapped through a fully connected layer (64 input dimensions, 12 output dimensions) to generate an action vector, i.e., a 12-dimensional parameter adjustment. Parameter adjustment amount The new parameter estimates are incorporated into the parameter estimates of the traditional controller through a smooth update mechanism. The specific formula is as follows:

[0073] ,

[0074] Among them, the smoothing coefficient Configure independently for each control channel.

[0075] Step 4: Establish an online reinforcement learning training mechanism based on DDPG (Deep Deterministic Policy Gradient) and define a multi-objective reward function.

[0076] The online reinforcement learning training mechanism is based on Markov decision processes, and its state space is a 27-dimensional reinforcement state vector. This refers to the input of the deep reinforcement learning parameter generator. The action space represents the 12-dimensional parameter adjustment space of the deep reinforcement learning parameter generator. The parameter generator network output is optimized using a multi-objective reward function. This multi-objective reward function consists of a tracking error penalty term, an error derivative penalty term, a control energy penalty term, and a control smoothness penalty term, and its mathematical expression is:

[0077] ,

[0078] Among them, tracking error Error derivative and the rate of change of control quantity All have been limited in size. , ,in, The weight of the squared error penalty term. The weight of the penalty term for the absolute value of the error derivative is... To control the weight of the input penalty term, The weight of the penalty term for the law of change of control quantity. This represents the sum of all awards.

[0079] Step 5: Optimize network parameters by combining experience playback, construct the target Q-value function and loss function, and establish the Actor and Critic network update process.

[0080] The experience replay buffer stores the most recent 5000 transfer records. During training, the network parameters are updated in batches of 512 randomly sampled data points. The initial 511 steps are a warm-up phase, during which data is collected but no network updates are performed. Once the buffer reaches the minimum batch size (512 data points), the network begins to update its parameters. The system randomly samples batches of data from the buffer. The state data is input into the parameter generation network, i.e., the policy network (Actor network), for computation. The 12-dimensional vector output by the Actor network represents the parameter adjustments, while the Critic network evaluates the state-action pairs (here, the action is the parameter adjustment output by the Actor network), outputting the Q-value representing the value assessment of the current state-action pair. The target Q-value is constructed as follows: First, the Actor network generates the target action based on the next state; then, the next state and the target action are input into the Critic network to obtain the target Q-value, whose mathematical expression is:

[0081] ,

[0082] in, The discount factor highlights the importance of recent rewards. For the Critic target network's value assessment of the next state-target action pair. The action policy generated by the Actor target network based on the next state. This indicates the next state that the environment transitions to after the current action is performed.

[0083] For the Critic network, the time difference (TD) error method is used to update its neural network parameters, which is implemented using mean squared error. Its loss function is defined as follows:

[0084] ,

[0085] in, , The parameters representing the Critic network. It is the Q-value estimate of the current state-action pair by the Critic network.

[0086] The gradient of the loss function is calculated using gradient descent, and then the Critic network parameters are updated based on the calculation results.

[0087] ,

[0088] ,

[0089] in, This represents the gradient of the loss function of the Critic network. For updated Critic network parameters, is the learning rate of the Critic network.

[0090] Similarly, the update formula for the Actor network is as follows:

[0091] ,

[0092] ,

[0093] in, The parameters representing the Actor network, This represents the gradient of the objective function of the Actor network. It is a policy function about gradient, The Q-value function representing the Critic network Regarding the action The gradient, and in Calculated under the conditions For updating the Actor network parameters, is the learning rate of the Actor network.

[0094] The Actor target network and the Actor network have identical network structures, but the Actor network is updated in real time, while the Actor target network is not updated in real time, but rather periodically "soft-updates" from the Actor main network. By selecting an appropriate soft-update rate, the Actor target network and the Critic target network can be soft-updated by their respective Actor and Critic networks:

[0095] ,

[0096] in, For soft update rate, These are the parameters of the Critic target network. These are the parameters of the Actor target network.

[0097] To evaluate the effectiveness and innovativeness of the designed control strategy, simulation verification was performed based on an embodiment. The quadcopter UAV system parameters were set as follows: , initial posture , , Given a reference trajectory , , air drag coefficient External disturbances are set to , , The control objective of this invention is to design a control input that causes the output signal to... , , Asymptotic tracking reference instructions , , And the transient convergence rate is not less than .

[0098] To meet the above preset performance requirements, the preset performance function is selected as follows: The controller parameters are selected as follows: , , , , , Controller gain Time-varying integrable functions: , , , , , .

[0099] Figure 3-8 The results are simulations of the system under the three-channel attitude control law established by this invention. Figure 3 , Figure 4 , Figure 5 This indicates that the control strategy designed in this invention can track the control target faster, reduce overshoot, and achieve a smoother tracking effect compared to the traditional adaptive law calculation method. Figure 6 , Figure 7 , Figure 8 The results show that the control strategy designed in this invention reduces errors and improves system performance compared to the traditional adaptive law calculation method.

[0100] Based on the same inventive concept, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned reinforcement learning-based quadcopter drone attitude preset performance control method.

[0101] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based quadcopter drone attitude preset performance control method.

[0102] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0103] This invention is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each step in the flowchart, and combinations of steps in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the steps in the flowchart. Figure 1 A device for a function specified in one or more processes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0105] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0106] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for pre-setting the attitude performance of a quadrotor unmanned aerial vehicle based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Considering time-varying parameters and external unknown disturbances, establish a mathematical model of the quadrotor UAV, define state variables, and establish the control-oriented state equations for the quadrotor UAV; the specific process is as follows: Considering time-varying parameters and unknown external disturbances, the mathematical model of the quadcopter UAV is described as follows: in, and These represent the roll, pitch, and yaw attitude angles of the UAV in the inertial frame, respectively. and These are the drag coefficients for the roll, pitch, and yaw channels, respectively. , and These are all inertial matrices of the UAV attitude system. and These represent the control torques for the roll, pitch, and yaw motions of the UAV, respectively. and These are external disturbances acting on the roll, pitch, and yaw channels, respectively. Define state variables The state equations for a control-oriented quadrotor UAV are designed as follows: in, , , , , , , , , , and These represent the control inputs for the roll, pitch, and yaw channels of the quadcopter UAV, respectively. Step 2: Select a time-varying preset performance boundary function and introduce an error transformation function. Combine this with the state equations of the quadcopter UAV to design a multi-channel preset performance attitude controller. The error transformation function ensures that the tracking error of the UAV attitude system meets the preset performance constraints. The specific process is as follows: Step 2.1, select a smoothly decreasing positive function. Pre-defined performance boundary functions for time-varying data. , These represent the preset lower and upper limits of the steady-state error, respectively. is the base of the natural logarithm. For preset positive numbers, Indicates time, These represent the roll, pitch, and yaw channels, respectively. Tracking error of UAV attitude system The following performance constraints must be met: in, and All are overshoot suppression parameters. , ,satisfy ; Introducing the error transformation function as follows: in, To normalize the tracking error, ; Step 2.2, Design a three-channel non-singular fast termination sliding mode function. : in, , , , , and All parameters are greater than zero. and All are positive odd numbers. It is a positive real number and satisfies , ; Step 2.3: Establish the three-channel attitude control law for the quadrotor UAV based on the sliding mode function: in, for The estimated value, for The separated constant term, , , and All are controllable gains to be designed. , , , , and They are respectively , and The estimated value, and They are respectively and The separated constant term, , and They are respectively , and The estimated value, and They are respectively and The separated constant term, , and They are respectively , and The estimated value, and They are respectively and The separated constant term, and They are respectively and The reference trajectory; A positive time-varying integrable function that satisfies: , , For integration variables, and They are respectively The upper and lower bounds, ; Step 3: Construct a deep reinforcement learning parameter generator, i.e., an Actor network. Determine the input and output of the Actor network based on a channel-based preset performance attitude controller. The input is the UAV's state vector, and the output is the time-varying parameter adjustment amount. Step 4: Train the Actor network and the corresponding Critic network based on the online reinforcement learning training mechanism of deep deterministic policy gradient, define a multi-objective reward function, and optimize the output of the Actor network through the multi-objective reward function; Step 5: Utilize the Actor network to output the time-varying parameter adjustment amount at the current moment, and fuse it with the time-varying parameter at the previous moment through a smooth update mechanism to obtain the time-varying parameter at the current moment. Substitute this parameter into the sub-channel preset performance attitude controller to obtain the control input of each channel of the UAV at the current moment, thereby realizing the preset performance control of the UAV attitude.

2. The method for attitude preset performance control of a quadrotor UAV based on reinforcement learning according to claim 1, characterized in that, In step 3, the deep reinforcement learning parameter generator includes a layer normalization module, a temporal feature extraction network, a residual module, and a fifth fully connected layer connected in sequence. The temporal feature extraction network includes a first fully connected layer, a first GELU activation function, a first normalization layer, a second fully connected layer, a second GELU activation function, and a second normalization layer connected in sequence. The residual module includes three residual blocks with the same structure connected in sequence. Each residual block includes a third fully connected layer, a third GELU activation function, a third normalization layer, a fourth fully connected layer, a fourth GELU activation function, and a fourth normalization layer connected in sequence. The input to the deep reinforcement learning parameter generator is a 27-dimensional state vector, including normalized real-time pose angles. angular velocity Three-channel tracking error and its differential Historical error moving average integral and parameter estimates ; The 27-dimensional state vector, after being standardized by the layer normalization module, is then passed sequentially through the temporal feature extraction network, the residual module, and the fifth fully connected layer to output the action vector, which is the 12-dimensional time-varying parameter adjustment. .

3. The method for attitude preset performance control of a quadrotor UAV based on reinforcement learning according to claim 1, characterized in that, In step 4, the multi-objective reward function consists of a tracking error penalty term, an error derivative penalty term, a control energy penalty term, and a control smoothness penalty term. The expression is: in, This represents the time step corresponding to the current moment. The weight of the error squared penalty term. The weight of the penalty term for the absolute value of the error derivative. To control the weight of the penalty term, To control the weight of the penalty term for the rate of change of the quantity, To track errors, To track the error derivative, To control the quantity, , To control the rate of change of quantity; Starting from the initial moment, collect data corresponding to each moment. Place them in the experience replay buffer, with each moment corresponding to a time step. Representing time steps The corresponding states, actions, and rewards are defined as follows: the state is the state vector input to the Actor network, and the action is the time-varying parameter adjustment amount output by the Actor network. Indicates time step The corresponding states are as follows: the first to the 511th time steps are the warm-up phase, during which only data is collected and the Actor network and Critic network parameters are not updated; when the experience replay buffer accumulates to the minimum batch size of 512 data points, the network parameter update begins. Randomly sample batches of data from the experience playback buffer. , the current state The input is processed by the Actor network, which then outputs the corresponding action. The Critic network evaluates the current state-action pair and outputs the Q-value of the current state-action pair evaluation; the Actor target network then evaluates the next state. Generate the target action, then set the next state. The target action is input into the Critic network to obtain the target Q-value; the Actor network has the same structure as the Actor network, and the Critic network has the same structure as the Critic network; the expression for the target Q-value is as follows: in, For the target Q value, Let the multi-objective reward function be the one corresponding to the current state-action pair. As a discount factor, For the Critic target network's value assessment of the next state-target action pair. The target action is generated by the Actor target network based on the next state. This indicates the next state the environment transitions to after the current action is performed; The parameters of the Critic network are updated using gradient descent, and the parameters of the Actor network are updated using the gradient information of the Critic network.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning-based attitude preset performance control method for quadrotor unmanned aerial vehicles as described in any one of claims 1 to 3.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning-based attitude preset performance control method for quadrotor UAVs as described in any one of claims 1 to 3.

Images