Four-rotor unmanned aerial vehicle fault-tolerant control method based on SAC method

Abstract

The embodiment of the invention provides a quadrotor unmanned aerial vehicle fault-tolerant control method based on an SAC method, and aims to reduce high time delay of a quadrotor unmanned aerial vehicle active fault-tolerant control method and improve the tracking effect and control performance of passive fault-tolerant control. The method comprises the steps of establishing a four-rotor unmanned aerial vehicle six-degree-of-freedom kinematics and dynamics mathematical model, designing a cascade control structure based on a PID controller, a PA controller and a strategy network, and improving a selector-controller network architecture according to an SAC algorithm interface. The method comprises the following steps: designing a simulation platform on a Python platform on the basis of a Pytorch library and a Stable-Baseline 3 architecture, and training a strategy network on the simulation platform; and the trained strategy network is deployed on a simulation platform for testing, and the method can enable the four rotors to quickly respond and adjust positions and postures under the condition that a single rotor is completely faulted, has relatively high tracking performance and control performance, and has remarkable theoretical value and engineering application prospect.

Description

Technical Field

[0001] Examples disclosed herein relate to control technologies for unmanned aerial vehicles (UAVs), specifically to a fault-tolerant control technology for a single rotor of a quadcopter UAV based on reinforcement learning (RL) methods. Background Technology

[0002] Quadrotors are autonomous unmanned aerial vehicles with four symmetrical rotors. They have been widely used in various fields, from military to civilian, covering reconnaissance, inspection, photography, transportation, search and rescue, and many other areas. Their safety has also attracted increasing attention, with ensuring the safety of property and lives during their flight missions being a key focus for the industry.

[0003] With the increasing application of quadcopter drones in the civilian sector, their safety hazards are becoming increasingly apparent, among which rotor failure is a typical and high-risk problem. During flight, when a rotor malfunctions (such as blade breakage, motor aging, or wiring faults), the lift provided by that rotor may decrease to some extent or even disappear completely. Given the relatively limited control freedom of a quadcopter system, when the thrust performance of one rotor is severely weakened or stops, the remaining rotors may struggle to rebalance in thrust and torque, leading to attitude instability. If a single rotor still retains some lift (i.e., partial failure), and if the remaining rotors have sufficient lift margin and the system responds promptly, system balance can be maintained through thrust compensation from the remaining rotors and control allocation reconfiguration. However, if the rotor fails completely, i.e., lift is completely lost, the yaw channel will become completely uncontrollable, and the aircraft will quickly fall into a chaotic state, making a safe landing impossible even for experienced pilots. Therefore, research on fault detection and fault-tolerant control in quadcopter systems is of great significance for reducing major accidents, improving flight mission reliability, and ensuring the stability of drone applications. Numerous studies have shown that by employing fault detection and fault-tolerant control strategies, unmanned aerial vehicle (UAV) systems can achieve greater robustness and reliability during mission execution.

[0004] In the current research and industrialization of unmanned aerial vehicles (UAVs), ensuring flight safety, enhancing fault tolerance, and improving mission adaptability have become key bottlenecks in the transition of UAVs from the laboratory to real-world applications. Fault-Tolerant Control (FTC) has become a hot topic in the field of UAV control. Related research has shown that FTC can maintain system performance or achieve safe landing even when sensors or actuators fail, which is particularly crucial for the widespread deployment of UAVs. Furthermore, incorporating advanced control methods such as machine learning, neural networks, and Active Disturbance Rejection Control (ADRC) into UAV fault-tolerant control is a cutting-edge research direction. If path planning and mission continuation / safe return can be achieved in the event of rotor failure, it will significantly improve the reliability and availability of UAVs in disaster relief, logistics transportation, agricultural inspection, and other application scenarios, thereby generating positive impacts on society, the economy, and the military.

[0005] Reinforcement Learning (RL) is a machine learning method where an agent learns a policy by interacting with its environment to maximize cumulative reward in the long run. Soft Actor-Critic (SAC) is a mainstream algorithm in RL that not only maximizes cumulative reward but also maximizes the policy's entropy, encouraging the agent to maintain a degree of randomness and diversity in its actions to better explore the environment. Because SAC uses an offline Actor-Critic architecture, it offers stable training and high sampling efficiency, making it particularly suitable for high-dimensional, continuous action spaces such as UAV control. Summary of the Invention

[0006] In view of this, the present invention aims to propose a fault-tolerant control method for single-rotor failure of quadcopter UAV based on the SAC method, which is used to overcome the problems of complexity, high latency and high computational burden in controller design, while ensuring control performance and enabling the quadcopter to remain stable in the event of single-rotor failure.

[0007] To achieve the above objectives, the technical approach of the present invention is as follows:

[0008] Step 1: Establish a six-degree-of-freedom kinematic and dynamic mathematical model for the quadcopter UAV;

[0009] Step 2: Design a high-level-low-level cascaded control structure based on a PID controller, a spindle controller, and a strategy network;

[0010] Step 3: Design the policy network loss function, supervised learning rules, and expert-based behavior cloning based on the reinforcement learning SAC algorithm:

[0011] Step 4: Develop a quadcopter simulation platform in a Python environment using the PyTorch library and the Stable-Baselines3 architecture. Train the quadcopter using the SAC algorithm to reach the specified position and maintain hovering under complete single-rotor failure. After training, verify the reliability and robustness of the strategy on the platform.

[0012] In step 1, considering the complexity of the quadcopter UAV system required for tracking and controlling the target, the initial six-degree-of-freedom quadcopter model is established using the quaternion method as follows:

[0013]

[0014]

[0015] Where p and v represent the position and velocity vectors of the quadrotor in the inertial coordinate system, ω = [ω x ω y ω z ] T q represents the angular velocity vector of the quadrotor in the body coordinate system, q = [q ω q x q y q z ] T This represents a unit quaternion, where m and J represent the mass and moment of inertia matrices of the quadrotor. This represents quaternion multiplication, ⊙ represents rotation of a vector using quaternions, and the unmodeled extra torque caused by aerodynamic disturbances and model errors is represented by T. ext The total thrust and torque provided by the rotor are expressed as:

[0016] F = [f1 f2 f3 f4] T

[0017] [T τ] T =GF

[0018] Where F and τ represent the thrust vector and torque of the quadrotor, T represents the total thrust of the quadrotor, and G is the control effectiveness matrix:

[0019]

[0020] Where, r x,l and r y，l The position vector r in the body coordinate system i x and y axis components; k t This represents the torque coefficient, which quantifies the proportional relationship between the thrust and drag torque generated by a single rotor.

[0021] Since the rotor cannot output the thrust required by the command in real time, this invention uses a first-order approximation to model the rotor thrust:

[0022]

[0023] Where σ is the rotor time constant, k i This is the rotor failure coefficient. k i =0 indicates that rotor i is completely faulty, the rotor stops and loses lift, while k i =1 indicates that the rotor is normal and has no faults.

[0024] In step 2, the advanced controller consists of a PID controller and a primary axis (PA) controller connected in series. The PID controller controls the position of the quadcopter, taking the desired position, current position, and velocity as input, and outputting the desired acceleration.

[0025]

[0026] The PA controller is used to handle underactuation issues following a failure. In the event of a complete single-rotor failure, lift on that side is completely lost, the quadrotor loses all control over the yaw path, and the airframe rapidly becomes unbalanced and subsequently rotates around the main axis. Based on this, the PA controller abandons direct control of all attitude degrees of freedom, focusing instead on maintaining a single critical direction without deviating from the intended trajectory. The main axis direction is designed to coincide with the thrust direction, allowing the quadrotor to hover without wobbling. Since the yaw path is uncontrollable, the PA controller outputs the quadrotor's desired roll rate, desired pitch rate, and z-axis. B Expected acceleration in the direction:

[0027]

[0028] Where, p des and q des This represents the desired roll rate and desired pitch rate of a quadcopter. Indicates the quadcopter at z B Expected acceleration in direction, This represents the unit vector indicating the direction of the total thrust in volume coordinates. express The x and y components, v out Indicates virtual output:

[0029]

[0030] in, Indicates the reference spindle.

[0031] The low-level controller mainly consists of a policy network. After receiving high-level control commands, it outputs rotor control commands to the actuators, thereby changing the thrust generated by the rotor. The policy network adopts a selector-controller network architecture, consisting of four controller networks and one selector network connected in parallel. Each controller network represents a fault condition. Each sub-network is a three-layer hidden layer MLP structure, with 64 nodes in each hidden layer. The observation space consists of position p, velocity v, and acceleration a. B Attitude quaternion q, angular velocity ω, angular acceleration Δω f Previous time step instruction u prev Desired angular velocity ω des and z B Expected acceleration in direction The system consists of an interface adapted to the SAC algorithm. The controller network outputs a Gaussian distribution of control commands, and the selector outputs the controller weights after processing by a softmax function.

[0032] Σ l ω l =1, ω i >0

[0033] Where, ω i These represent the weights of the corresponding controllers. Based on the selector weights and controller outputs, the final control strategy is output in the form of a Gaussian Mixture Model (GMM):

[0034]

[0035] in, This represents the output distribution of controller i. This represents the distribution of the policy output. During training, a reparameterization technique is used to sample from the distribution, and then the sampled values ​​are multiplied by the standard deviation and added to the mean to generate the policy output. Conversely, during testing, a deterministic policy is used, and the mean of the policy is used as the policy output.

[0036] In step 3, the present invention designs a supervised learning loss based on the mean squared error (MSE) between the selector output and the actual fault information to guide the learner's learning process:

[0037] L s (π)=||k label -k|| 2

[0038] Where, k label This indicates actual quadcopter malfunction information, for example, when the third rotor fails, k label =[1 1 0 1]T .

[0039] The reward function of the reinforcement learning SAC algorithm is used to guide the quadrotor to the target position and maintain hover during training, while penalizing airframe oscillations in case of rotor failure. To encourage smoother control maneuvers, the reward further penalizes control commands with large deviations between consecutive controls. Because the research focuses on real-world scenarios, the reward also penalizes excessive yaw angular velocities when the yaw channel is uncontrollable. Furthermore, due to the presence of the PA control module, the reward also affects z... B A penalty is imposed for directional acceleration deviations. An auxiliary survival reward is introduced during the initial training phase to encourage longer quadcopter survival and prevent premature flight failures due to immature early strategies. The overall reward function is expressed as a weighted combination:

[0040]

[0041] Due to the high complexity and randomness of quadcopter control, maintaining flight stability is difficult in the early stages of training. To enhance control performance and accelerate convergence, this invention uses expert experience to guide the initial training. An incremental nonlinear dynamic inversion (INDI) method for active fault-tolerant control is introduced. During training, the expert uses known fault information, while the strategy does not.

[0042]

[0043] Where f represents the variable after low-frequency noise filtering; τ f and α d This represents the estimated torque and desired angular acceleration. The rotor commands for each rotor can be calculated.

[0044]

[0045] in, This represents the dimension reduction control effectiveness matrix, where the corresponding faulty rotor number column is set to zero in the control matrix G.

[0046] By comparing the instructions output by the expert experience-driven command and policy network, the behavioral cloning loss can be obtained:

[0047] L BC (π)=||u des -u|| 2

[0048] The total loss function of the policy network is defined as a weighted combination of three parts: the selector-supervised learning loss, which minimizes the mean square error between the selector's predicted output and the actual fault information; the reinforcement learning loss, which optimizes the long-term accumulated reward; and the behavior cloning loss, which uses expert experience to guide the early training. This combined loss function is used to train the policy network within the SAC algorithm.

[0049] L(π)=L s (π)+(1-α)L RL (π)+αL BC (π)

[0050] Here, α is a decay factor used to adjust the balance between expert experience and autonomous policy learning, allowing the model to rely more on expert experience in the early stages of training, while relying on the policy network output in the later stages. This mechanism effectively reduces the instability caused by immature policies in the early training phase and accelerates the convergence of the loss function.

[0051] In step 4, a quadcopter simulation platform was developed in a Python environment based on the PyTorch library and the Stable-Baselines3 architecture. The SAC algorithm was used to train the quadcopter to fly to a designated location and maintain hovering even in the event of a complete single-rotor failure. The trained strategy was then tested to obtain curves showing the changes in the quadcopter's position, velocity, quaternion, and angular velocity over time during the flight mission, thus verifying the feasibility and stability of the invention.

[0052] The advantages and advancements of this invention compared to the prior art are mainly reflected in the following aspects:

[0053] First, this invention achieves fault-tolerant control of a quadcopter UAV under complete single-rotor failure by combining deep learning networks with the reinforcement learning SAC algorithm. Second, by modifying the selector-controller policy network according to the SAC algorithm, a multi-expert parallel network structure can be used to replace the original single-structure policy network, thereby increasing training diversity and stability. Furthermore, the reward function designed based on the SAC reinforcement learning algorithm effectively improves the survival time in the early stages of training, avoiding premature crashes caused by immature policies at the beginning of training, thus improving training efficiency and policy convergence speed. Finally, the trained policy is tested on a simulation platform, verifying that the proposed method can effectively control a quadcopter UAV with a single-rotor failure and maintain stability in hovering, demonstrating good robustness and control performance. Attached Figure Description

[0054] To more clearly illustrate the technical solution of the invention, the accompanying drawings of the invention will be briefly described below:

[0055] Figure 1 This is a flowchart of the present invention;

[0056] Figure 2 This is a structural diagram of a quadcopter drone;

[0057] Figure 3 A control structure diagram for cascading high-level and low-level control modules;

[0058] Figure 4 Here is a diagram of the selector-controller network structure;

[0059] Figure 5 A graph showing the change in total reward during the evaluation phase of the training process;

[0060] Figure 6 To test the parameter change curves under the condition of rotor failure using the trained strategy;

[0061] Figure 7 To test the parameter variation curves under the condition of rotor failure using the trained strategy;

[0062] Figure 8 To test the parameter variation curves under the condition of rotor failure using the trained strategy;

[0063] Figure 9 To test the parameter variation curves under the condition of a fourth rotor failure using the trained strategy; Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, further explanation and description will be provided below in conjunction with the accompanying drawings. Obviously, the described examples are only some, not all, embodiments of this invention. All other embodiments obtained by those skilled in the art based on the described embodiments without inventive effort are also within the scope of protection of this invention.

[0065] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the subject matter of this invention pertains.

[0066] In order to reduce the high latency under active fault-tolerant control of quadrotors and improve the tracking effect and control performance of passive fault-tolerant control, this invention proposes a fault-tolerant control method for single rotor of quadrotor UAV based on the SAC method. Figure 1 This is an exemplary flowchart of a single-rotor fault-tolerant control design for a quadcopter UAV based on the SAC method, as proposed in this invention.

[0067] Figure 2A simplified geometric model of a quadcopter unmanned aerial vehicle (UAV) is provided. The object of this invention consists of a main frame and four symmetrically distributed rotors. The rotors are evenly spaced on the plane of the fuselage in an X-shaped layout, and each rotor support has the same length with an included angle of 90°. The body coordinate system is set at the center of mass of the UAV.

[0068] Step 1: Establish a six-degree-of-freedom kinematic and dynamic model of the quadcopter UAV:

[0069] Considering the complexity of the quadcopter UAV system required for tracking and controlling the target, a six-degree-of-freedom initial quadcopter model is established using the quaternion method, as shown in equation (1.1):

[0070]

[0071] Where p and v represent the position and velocity vectors of the quadrotor in the inertial coordinate system, ω = [ω x ω y ω z ] T q represents the angular velocity vector of the quadrotor in the body coordinate system, q = [q ω q x q y q z ] T This represents a unit quaternion, where m and J represent the mass and moment of inertia matrices of the quadrotor. This represents quaternion multiplication, ⊙ represents the rotation of a vector using quaternions, and τ represents the unmodeled extra torque caused by aerodynamic disturbances and model errors. ext The total thrust and torque provided by the rotor are shown in equation (1.2):

[0072]

[0073] Where F and τ represent the thrust vector and torque of the quadrotor, T represents the total thrust of the quadrotor, and G is the control effectiveness matrix:

[0074]

[0075] Where, r x，i and r y，i The position vector r in the body coordinate system i x and y axis components; κ t This represents the torque coefficient, which quantifies the proportional relationship between the thrust and drag torque generated by a single rotor.

[0076] Since the rotor cannot output the thrust required by the command in real time, this study uses a first-order approximation to model the rotor thrust:

[0077]

[0078] Where σ is the rotor time constant, k i This is the rotor failure coefficient. k i =0 indicates that rotor i is completely faulty, the rotor stops and loses lift, k i =1 indicates that the rotor is normal and has no faults.

[0079] like Figure 3 As shown, the control structure consists of a high-level controller and a low-level controller cascaded together. The high-level controller is composed of a PID controller and a PA controller connected in series. The PID controller calculates the desired acceleration based on the desired position and the current position velocity, which is used to control the position of the quadcopter. Since the quadcopter cannot yaw during a fault and the yaw path is uncontrollable, the PA controller is used to maintain the quadcopter's attitude and keep it stable without wobbling. The PA controller calculates the desired angular velocities for the pitch and roll paths and z-axis based on the desired acceleration and the angular velocity of the yaw path. B The desired acceleration in the yaw direction. The low-level controller mainly consists of a policy network, which calculates actuator commands based on high-level control instructions. Since the yaw channel is uncontrollable, the desired angular velocity of the yaw channel needs to be manually set.

[0080] In step 2, the advanced controller consists of a PID controller and a primary axis (PA) controller connected in series. The PID controller is used to control the position of the quadcopter, taking the desired position, current position, and velocity as inputs, and outputting the desired acceleration (2.1):

[0081]

[0082] The PA controller is used to handle underactuation issues following a failure. In the event of a complete single-rotor failure, lift on that side is completely lost, the quadrotor loses all control over the yaw path, and the airframe rapidly becomes unbalanced and subsequently rotates around the main axis. Based on this, the PA controller abandons direct control of all attitude degrees of freedom, focusing instead on maintaining a single critical direction without deviating from the intended trajectory. The main axis direction is designed to coincide with the thrust direction, allowing the quadrotor to hover without wobbling. Since the yaw path is uncontrollable, the PA controller outputs the quadrotor's desired roll rate, desired pitch rate, and z-axis. B The desired acceleration in the direction is shown in equation (2.2):

[0083]

[0084] Where, p des and q des This represents the desired roll rate and desired pitch rate of a quadcopter. Indicates the quadcopter at z B Expected acceleration in direction, This represents the unit vector indicating the direction of the total thrust in volume coordinates. express The x and y components, v out Indicates virtual output:

[0085]

[0086] in, Indicates the reference spindle.

[0087] Figure 4 The selector-controller network framework consists of four controller networks connected in parallel with one selector network. Each network is a 27×(3×64)×4 MLP structure. The controller network outputs a Gaussian distribution of actuator control commands, and the selector output is processed by a softmax function to become the weights of each controller network. The weighted combination forms a Gaussian mixture distribution as the final output.

[0088] The low-level controller mainly consists of a policy network. After receiving high-level control commands, it outputs rotor control commands to the actuators, thereby changing the thrust generated by the rotor. The policy network adopts a selector-controller network architecture, consisting of four controller networks and one selector network connected in parallel. Each controller network represents a fault condition. Each sub-network is a three-layer hidden layer MLP structure, with 64 nodes in each hidden layer. The observation space consists of position p, velocity v, and acceleration a. B Attitude quaternion q, angular velocity ω, angular acceleration Δω f Previous time step instruction u prev Desired angular velocity ω des and z B Expected acceleration in direction The system consists of an interface adapted to the SAC algorithm. The controller network outputs a Gaussian distribution of control commands, and the selector outputs the controller weights after processing by a softmax function.

[0089] ∑ l ω l =1, ω i >0 (2.4)

[0090] Where, ω i These represent the weights of the corresponding controllers. Based on the selector weights and controller outputs, the final control strategy is output in the form of a Gaussian Mixture Model (GMM):

[0091]

[0092] in, This represents the output distribution of controller i. This represents the distribution of the policy output. During training, a reparameterization technique is used to sample from the distribution, and then the sampled values ​​are multiplied by the standard deviation and added to the mean to generate the policy output. Conversely, during testing, a deterministic policy is used, and the mean of the policy is used as the policy output.

[0093] In step 3, the present invention designs a supervised learning loss based on the mean squared error (MSE) between the selector output and the actual fault information to guide the learner's learning process:

[0094] L s (π)=||k label -k|| 2 (3.1)

[0095] Where, k label This indicates actual quadcopter malfunction information, for example, when the third rotor fails, k label =[1 1 0 1] T .

[0096] The reward function of the reinforcement learning SAC algorithm is used to guide the quadrotor to the target position and maintain hover during training, while penalizing airframe oscillations in case of rotor failure. To encourage smoother control maneuvers, the reward further penalizes control commands with large deviations between consecutive controls. Because the research focuses on real-world scenarios, the reward also penalizes excessive yaw angular velocities when the yaw channel is uncontrollable. Furthermore, due to the presence of the PA control module, the reward also affects z... B A penalty is imposed for directional acceleration deviations. An auxiliary survival reward is introduced during the initial training phase to encourage longer quadcopter survival and prevent premature flight failures due to immature early strategies. The overall reward function is expressed as a weighted combination:

[0097]

[0098] Due to the high complexity and randomness of quadcopter control, maintaining flight stability is difficult in the early stages of training. To enhance control performance and accelerate convergence, this invention uses expert experience to guide the initial training. An incremental nonlinear dynamic inversion (INDI) method for active fault-tolerant control is introduced. During training, the expert uses known fault information, while the strategy does not.

[0099]

[0100] Where f represents the variable after low-frequency noise filtering; τ f and αd This represents the estimated torque and desired angular acceleration. The rotor commands for each rotor can be calculated.

[0101]

[0102] in, This represents the dimension reduction control effectiveness matrix, where the corresponding faulty rotor number column is set to zero in the control matrix G.

[0103] By comparing the instructions output by the expert experience-driven command and policy network, the behavioral cloning loss can be obtained:

[0104] L BC (π)=||u des -u|| 2 (3.5)

[0105] The total loss function of the policy network is defined as a weighted combination of three parts: the selector-supervised learning loss, which minimizes the mean square error between the selector's predicted output and the actual fault information; the reinforcement learning loss, which optimizes the long-term accumulated reward; and the behavior cloning loss, which uses expert experience to guide the early training. This combined loss function is used to train the policy network within the SAC algorithm.

[0106] L(π)=L s (π)+(1-α)L RL (π)+αL BC (π) (3.6)

[0107] Here, α is a decay factor used to adjust the balance between expert experience and autonomous policy learning, allowing the model to rely more on expert experience in the early stages of training, while relying on the policy network output in the later stages. This mechanism effectively reduces the instability caused by immature policies in the early training phase and accelerates the convergence of the loss function.

[0108] In step 4, a quadcopter simulation platform is developed in the Python environment based on the PyTorch library and the Stable-Baselines3 architecture. The SAC algorithm is used to train the quadcopter to fly to a specified location and to hover in the event of a complete failure of a single rotor.

[0109] Figure 5 This represents the change in total reward during training. The invention was trained on a workstation using an NVIDIA GeForce RTX 4060 GPU for approximately 20 hours, totaling over 20 million steps. It can be seen that convergence occurred around 17.5 million steps, with the final reward converging to approximately -50.

[0110] Figure 6-9The curves represent the changes in position, velocity, attitude, and angular velocity under each fault condition. The quadrotor is initially generated at an arbitrary position within a cube with sides of one meter. It is then guided to the target position and hover. At approximately 5 seconds, a rotor is randomly selected and input a fault, causing it to completely fail. The quadrotor then remains stable and hovers at the target position under the fault condition. It can be observed that before the fault occurs, position, velocity, attitude, and angular velocity are all stable. At approximately 5 seconds, at the instant of rotor failure, the position, especially in the x-direction, shows a significant deviation, and the velocity and angular velocity also exhibit large transient disturbances. Simultaneously, the attitude quaternions show high-frequency oscillations. Subsequently, the quadrotor rapidly adjusts its attitude, and the oscillations of all state variables gradually decay, eventually reaching a new equilibrium state. This indicates that the trained strategy can effectively suppress the disturbances caused by the fault and enable the system to quickly recover and stabilize.

Claims

1. A fault-tolerant control method for a quadrotor unmanned aerial vehicle based on the SAC method, characterized in that, Includes the following steps: Step 1: Establish a six-degree-of-freedom kinematic and dynamic mathematical model for the quadcopter UAV; Step 2: Design a high-level-low-level cascaded control structure based on a PID controller, a spindle controller, and a strategy network; Step 3: Design the policy network loss function, supervised learning rules, and expert-based behavior cloning based on the reinforcement learning SAC algorithm; Step 4: Develop a quadcopter simulation platform in a Python environment using the PyTorch library and the Stable-Baselines3 architecture. Train the quadcopter using the SAC algorithm to reach the specified position and maintain hovering under complete single-rotor failure. After training, verify the reliability and robustness of the strategy on the platform.

2. The fault-tolerant control method for a quadrotor UAV based on the SAC method according to claim 1, characterized in that: In step 1, considering the complexity of the quadcopter UAV system required by the fault-tolerant control strategy, an initial quadcopter model with six degrees of freedom is established as shown in equation (1.1): Where p and v represent the position and velocity vectors of the quadrotor in the inertial coordinate system, ω = [ω x ω y ω z ] T q represents the angular velocity vector of the quadrotor in the body coordinate system, q = [q ω q x q y q z ] T This represents a unit quaternion, where m and J represent the mass and moment of inertia matrices of the quadrotor. This represents quaternion multiplication, ⊙ represents rotation of a vector using quaternions, and the unmodeled extra torque caused by aerodynamic disturbances and model errors is represented by T. ext The total thrust and torque provided by the rotor are shown in equation (1.2): Where F and τ represent the thrust vector and torque of the quadrotor, T represents the total thrust of the quadrotor, and G is the control effectiveness matrix: Where, r x，i and r y，i The position vector r in the body coordinate system i x and y axis components; κ t This represents the torque coefficient, which quantifies the proportional relationship between the thrust and drag torque generated by a single rotor. Since the rotor cannot output the thrust required by the command in real time, this invention uses a first-order approximation to model the rotor thrust: Where σ is the rotor time constant, k i This is the rotor failure coefficient. k i =0 indicates that rotor i is completely faulty, the rotor stops and loses lift, k i =1 indicates that the rotor is normal and has no faults.

3. The method according to claim 1, wherein, The control structure design described in step 2 includes designing a high-level controller and a low-level controller. The advanced controller consists of a PID controller and a primary axis (PA) controller connected in series. The PID controller is used to control the position of the quadcopter, taking the desired position, current position, and velocity as inputs, and outputting the desired acceleration (2.1): The PA controller is used to handle underactuation issues after a fault and is responsible for controlling the quadcopter's attitude. The PA controller outputs the quadcopter's desired roll rate, desired pitch rate, and z-axis. B The desired acceleration in the direction is shown in equation (2.2): Where, p des and q des This represents the desired roll rate and desired pitch rate of a quadcopter. This indicates that the quadcopter is in z B Expected acceleration in the direction, This represents the unit vector indicating the direction of the total thrust in volume coordinates. express The x and y components, v out Indicates virtual output: in, Indicates the reference spindle. The low-level controller mainly consists of a policy network. After receiving high-level control commands, it outputs rotor control commands to the actuators, thereby changing the thrust generated by the rotor. The policy network adopts a selector-controller network architecture, consisting of four controller networks and one selector network connected in parallel. Each controller network represents a fault condition. Each sub-network is a three-layer hidden layer MLP structure, with 64 nodes in each hidden layer. The observation space consists of position p, velocity v, and acceleration a. B Attitude quaternion q, angular velocity ω, angular acceleration Δω f Previous time step instruction u prev Desired angular velocity ω des and z B Expected acceleration in direction The system consists of an interface adapted to the SAC algorithm. The controller network outputs a Gaussian distribution of control commands, and the selector outputs the controller weights after processing by a softmax function. ∑ i oh i =1,ω i >0 (2.4) Where, ω i These represent the weights of the corresponding controllers. Based on the selector weights and controller outputs, the final control strategy is output in the form of a Gaussian Mixture Model (GMM): in, This represents the output distribution of controller i. This represents the distribution of the policy output. During training, a reparameterization technique is used to sample from the distribution, and then the sampled values ​​are multiplied by the standard deviation and added to the mean to generate the policy output. Conversely, during testing, a deterministic policy is used, and the mean of the policy is used as the policy output.

4. The method according to claim 1, wherein, The policy network loss function design described in step 3 includes supervised learning loss, reward function, and behavior cloning. This invention designs a supervised learning loss based on the mean squared error (MSE) between the selector output and the actual fault information to guide the selector's learning process. L s (π)=||k label -k|| 2 (3.1) Where, k label This indicates actual quadcopter malfunction information, for example, when the third rotor fails, k label =[1 1 0 1] T . The reward function of the reinforcement learning SAC algorithm is used to guide the quadrotor to the target position and maintain hovering during training, while penalizing the airframe oscillation in the event of rotor failure. This invention utilizes expert experience to guide the initial training. It introduces an incremental nonlinear dynamic inversion (INDI) method for active fault-tolerant control, where experts use known fault information during training, while the strategy does not. Where f represents the variable after low-frequency noise filtering; τ f and α d This represents the estimated torque and desired angular acceleration. The rotor commands for each rotor can be calculated. in, This represents the dimension reduction control effectiveness matrix, where the corresponding faulty rotor number column is set to zero in the control matrix G. By comparing the instructions output by the expert experience-driven command and policy network, the behavioral cloning loss can be obtained: L BC (π)=||u des -u|| 2 (3.5) The total loss function of the policy network is defined as a weighted combination of three parts: L(π)=L s (π)+(1-α)L RL (π)+αL BC (p) (3.6) Here, α is a decay factor used to adjust the balance between expert experience and autonomous strategy learning.

5. The method according to claim 1, wherein the control method includes building a simulation platform in a Python environment based on the PyTorch library and the Stable-Baselines3 architecture, and training it using the SAC algorithm. The trained policy network is then tested on the simulation platform under various fault conditions to verify the feasibility and robustness of the method. The control structure designed in step 2 of claim 1 is combined with the loss function designed in step 3 and applied to the initial unsimplified six-DOF quadcopter UAV system in step 1. The system is then tested on a simulation platform built in the Python environment.

6. A fault-tolerant control system for a quadcopter unmanned aerial vehicle, employing the method according to any one of claims 1 to 5, the system comprising: • A module used to build six-degree-of-freedom kinematic and dynamic models; • Modules used for designing control structures; • A module for designing learning-based, reward-function, and behavior-cloning-loss-function approaches; • A simulation platform module based on the PyTorch library and the Stable-Baseline3 architecture for training and testing in a Python environment.

Images