Fault handling decision-making method and device for double-unmanned-aerial-vehicle cooperative hoisting system and medium

Abstract

The invention relates to the technical field of multi-unmanned-aerial-vehicle intelligent cooperative control and fault tolerance, and provides a double-unmanned-aerial-vehicle cooperative hoisting system fault handling decision-making method and device and a medium, and the method comprises the steps: constructing a simulation environment capable of simulating the dynamics and various faults of a double-unmanned-aerial-vehicle cooperative hoisting system; key elements used for reinforcement learning in the double-unmanned-aerial-vehicle cooperative hoisting system are defined; the key elements comprise states, actions and rewards; in the simulation environment, based on the key elements, carrying out offline training by adopting a Q-learning algorithm; and the Q table obtained through training is used for online fault disposal decision making of the double-unmanned-aerial-vehicle cooperative hoisting system. According to the method, after a sudden fault occurs, collaborative decision can be made autonomously, quickly and intelligently, multiple targets such as machine protection and load protection are dynamically balanced, and finally, maximization of system safety is achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent collaborative control and fault tolerance technology for multiple unmanned aerial vehicles (UAVs), and more specifically, to a fault handling decision-making method, equipment, and medium for a dual-UAV collaborative hoisting system. Background Technology

[0002] Multi-UAV collaborative hoisting systems have shown great application potential in civilian and military fields such as large-scale material transportation, disaster relief, and high-altitude operations due to their advantages of strong load capacity and high flexibility. However, they face severe challenges in practical applications due to sudden failures (such as motor failure, rope breakage, and sudden load changes).

[0003] The control methods of traditional multi-UAV collaborative hoisting systems rely heavily on accurate system dynamics models. When sudden failures such as motor failure, rope breakage, or sudden load changes cause drastic changes in system dynamics, model inaccuracies can lead to a sharp decline in control performance.

[0004] Most existing fault handling solutions are designed for single faults, rely on preset rules, and lack the intelligent decision-making ability to assess the situation online and dynamically weigh multiple objectives such as "protecting the machine" and "protecting the load".

[0005] Furthermore, common centralized control architectures may suffer from communication delays and computational bottlenecks when handling sudden failures, making it difficult to meet millisecond-level response requirements in terms of real-time performance. Additionally, traditional methods are poorly adaptable to unforeseen failure modes.

[0006] Recent research, such as Multi-Agent Reinforcement Learning (MARL) and fault-tolerant control combining deep learning with Control Barrier Functions (CBF), has provided new insights. Q-learning, a classic model-free reinforcement learning (RL) algorithm, can autonomously learn optimal policies through interaction with the environment; however, when applied to such complex problems, it faces challenges such as a large state-action space and slow convergence speed. Summary of the Invention

[0007] The present invention aims to provide a fault handling decision-making method, equipment and medium for a dual-UAV collaborative hoisting system, in order to solve the problems of strong model dependence, unintelligent decision-making and poor real-time performance of traditional methods under sudden failures.

[0008] In a first aspect, the present invention provides a fault handling decision-making method for a dual-UAV collaborative hoisting system, comprising: Construct a simulation environment capable of simulating the dynamics and various faults of a dual-UAV collaborative lifting system; Define the key elements for reinforcement learning in a dual-UAV collaborative hoisting system; the key elements include state, action, and reward. In the simulation environment, offline training is performed using the Q-learning algorithm based on the key elements. The Q-table obtained from training is used to make online fault handling decisions for a dual-UAV collaborative hoisting system.

[0009] In a preferred embodiment, the state includes the state of the two UAVs, the state of the payload, discrete flags representing the fault type, and the relative position of the payload to the target point.

[0010] In a preferred embodiment, the state needs to be discretized, dividing the continuous state into discrete intervals or levels.

[0011] In a preferred embodiment, the actions include emergency coordinated landing, health drone towing, active sway-eliminating forced landing, and safe dropping and evacuation.

[0012] In a preferred embodiment, each action is mapped to a low-level controller; the low-level controller generates control commands corresponding to each action and sends them to the two drones in the dual-drone collaborative hoisting system for execution.

[0013] In a preferred embodiment, the reward includes: A reward will be given when the payload is successfully delivered to the target point; Penalty will be imposed if any drone crashes; Penalty is imposed when the load falls or swings beyond the safe physical limits; A reward is given when the hoisting system moves toward the target point; When the energy consumption of the hoisting system exceeds the energy consumption threshold, a penalty is imposed; when the control is smooth, a reward is given. A penalty is imposed when the load swings beyond the swing threshold.

[0014] In a preferred embodiment, the offline training using the Q-learning algorithm includes: The Q table is initialized as Q(S,A), where the behavior state S of the Q table Q(S,A) is represented by the action A. The exploration and training are conducted in a simulation environment, with each round's time step denoted as t; The current state is represented as s_t; Use an ε-greedy strategy to select action a_t: explore randomly with probability ε and select the action with the largest current Q value with probability 1-ε; Execute action a_t, the simulation environment transitions to the next state, represented as s_{t+1}, and receives a reward r_t; Update the Q-table according to the update rules of the Q-learning algorithm: Q(s_t,a_t)←Q(s_t,a_t)+α*[r_t+γ*max {a_t} Q(s_{t+1},a_t)-Q(s_t,a_t)] Where Q(s_t, a_t) represents the estimated Q value of the agent at time t after selecting and executing action a_t in state s_t, max {a_t} Q(s_{t+1},a_t) represents the maximum expected return of the next state s_{t+1}, α represents the learning rate, and γ represents the discount factor. Repeat the exploration and training process until the Q-table converges.

[0015] In a preferred embodiment, the online fault handling decision-making for the dual-UAV collaborative hoisting system using the trained Q-table includes: The trained Q-table is deployed to the dual-UAV collaborative hoisting system for online operation; Real-time monitoring of the status of the dual-drone collaborative hoisting system; The state of the dual-UAV collaborative hoisting system is discretized and represented as state s_current; Query the Q-table obtained from training to find the action a_best with the maximum Q value under state s_current; The underlying controller corresponding to action a_best is triggered. This underlying controller generates control commands for action a_best and sends them to the two drones in the dual-drone collaborative hoisting system for execution. Repeat the above process until the fault is resolved.

[0016] In a second aspect, the present invention provides an electronic device, comprising: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the instructions stored in the memory to perform the above-described method.

[0017] Thirdly, the present invention provides a computer-readable storage medium for storing instructions that, when executed, enable the above-described method to be implemented.

[0018] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. Model-free and highly adaptable: This invention utilizes the model-free nature of reinforcement learning, eliminating the dependence on precise mathematical models. It can learn through training to cope with unknown sudden failure modes that are not pre-programmed, resulting in stronger generalization ability and robustness.

[0019] 2. Intelligent dynamic trade-off: Through the design of a multi-objective reward function, this invention enables the intelligent agent to autonomously learn complex trade-off strategies between conflicting objectives such as "preserving the machine" and "preserving the payload," making decisions that are closer to the level of human experts rather than mechanically executing preset rules.

[0020] 3. High real-time performance: The core of the online phase of this invention is state discretization and table lookup operation, which has extremely high computational efficiency and meets the millisecond-level response requirements for fault handling.

[0021] 4. Collaborative decision optimization: This invention learns the optimal collaborative strategy at the overall system level, achieving a fault-tolerant control effect of 1+1>2.

[0022] 5. Safety and economy: The high-risk, high-cost trial-and-error learning process in this invention is completed in a simulation environment, which greatly reduces training risks and costs. Attached Figure Description

[0023] Figure 1 The flowchart illustrates a fault handling decision-making method for a dual-UAV collaborative hoisting system provided in this embodiment of the invention.

[0024] Figure 2 This is a schematic diagram of the state discretization process in an embodiment of the present invention.

[0025] Figure 3 This is a flowchart of the Q-learning algorithm training process in an embodiment of the present invention.

[0026] Figure 4 This is a flowchart illustrating the execution of the online decision-making stage in an embodiment of the present invention.

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0030] To address the problems of traditional methods, such as strong model dependence, unintelligent decision-making, and poor real-time performance under sudden failures, this invention provides a fault handling decision-making method for a dual-UAV collaborative hoisting system. This method enables the system to make collaborative decisions autonomously, quickly, and intelligently after a sudden failure, dynamically balancing multiple objectives such as "preserving the UAV" and "preserving the payload," ultimately maximizing system safety.

[0031] Therefore, in this fault handling decision-making method for a dual-UAV collaborative hoisting system, an intelligent decision-making framework based on the Q-learning algorithm is designed. Through state discretization and hierarchical action selection, the complex fault handling problem of a dual-UAV collaborative hoisting system is transformed into a learnable decision-making process. The intelligent agent interacts with the system in a high-fidelity simulation environment, autonomously learning the optimal collaborative handling action to take under specific fault states, ultimately achieving autonomous and safe fault handling. Figure 1 As shown, the fault handling decision-making method of the dual-UAV collaborative hoisting system includes the following steps: S100 is used to build a simulation environment that can simulate the dynamics and various faults of a dual-UAV collaborative hoisting system.

[0032] In this embodiment of the invention, the simulation environment is a high-fidelity simulation environment, and the faults include motor failure, rope breakage, sudden load changes, etc., and this simulation environment serves as a training ground for the reinforcement learning agent. In some other scenarios, environmental disturbances (such as wind disturbances) can also be simulated.

[0033] S200 defines key elements for reinforcement learning in a dual-UAV collaborative hoisting system, including state, action, and reward.

[0034] In this embodiment of the invention, the state refers to the state of the system perceived by the agent at each time step, including: (1) The status of the two UAVs, including their position, speed, attitude, angular velocity, and remaining motor thrust; (2) The state of the load, including one or more of the following: swing angle (swing angle in both pitch and yaw directions), swing angular velocity, and altitude; (3) Discrete flags representing fault types; (4) The relative position of the load and the target point.

[0035] The state needs to be discretized, such as Figure 2 As shown, continuous states are divided into discrete intervals or levels. For example, the swing angle is discretized into [negative large, negative medium, negative small, zero, positive small, positive medium, positive large], and the velocity is discretized into [high negative, low negative, zero, low positive, high positive], etc. Through discretization, the infinite-dimensional continuous state space is mapped to a finite set of states.

[0036] In this embodiment of the invention, the action refers to employing a hierarchical action space to reduce the decision-making dimensionality. The action output by the agent is a high-level decision instruction, rather than a direct low-level control quantity. The action includes: (1) Emergency coordinated landing: With the primary goal of preserving the payload, the two UAVs coordinate to land together as smoothly as possible; (2) Healthy drone towing: The faulty drone attempts to hover or land, while the healthy drone takes on the entire load and tows the load to a safe point; (3) Active anti-sway forced landing: The two UAVs perform specific coordinated maneuvers (such as reverse movement) to suppress load swing and create conditions for forced landing; (4) Safe drop and evacuation: In extreme cases, actively release the payload to save both drones.

[0037] Each action is mapped to a pre-designed low-level controller, such as a trajectory tracking controller based on PID (Proportional Integral Derivative). This low-level controller generates control commands corresponding to each action and sends them to the two drones in the dual-drone collaborative lifting system for execution, including controlling the drones' throttle, pitch, roll, etc.

[0038] In this embodiment of the invention, the reward is a reward that integrates multiple objective trade-offs and is used to evaluate the merits of an action. The reward includes: (1) A reward will be given when the payload is successfully delivered to the target point: R_task=X1 R_task represents the reward for successfully delivering the payload to the target point, and X1 is the specific reward score for successfully delivering the payload to the target point. It can be set according to needs or actual application, such as +1000.

[0039] (2) Penalize the user when any drone crashes: R_crash=X2 R_crash represents the penalty for any drone crashing, and X2 is the specific penalty score for any drone crashing. This score can be set according to needs or actual application conditions, such as -1000.

[0040] (3) Penalty will be imposed when the load falls or swings beyond the safe physical limits: R_payload=X3 R_payload represents the penalty when the load falls or swings beyond the safe physical limit, and X2 is the specific penalty score when the load falls or swings beyond the safe physical limit. It can be set according to needs or actual application, for example, -800.

[0041] (4) A reward is given when the hoisting system moves toward the target point: R_progress=k1*(d_prev-d_current) R_progress represents the reward when the hoisting system moves towards the target point, k1 represents the ratio coefficient of the movement reward, which can be set according to needs or actual application, d_prev represents the distance from the load to the target point at the previous moment, and d_current represents the distance from the load to the target point at the current moment. The faster the movement, the higher the reward score.

[0042] (5) Penalize the hoisting system when its energy consumption exceeds the energy consumption threshold, and reward it when the control is smooth: R_energy=k2*(|U1|+|U2|) Where R_energy represents the reward or penalty for energy consumption and smooth control of the hoisting system, k2 represents the ratio coefficient of reward or penalty for energy consumption and smooth control, which can be set according to needs or actual application, U1 represents the energy consumption control amount, and U2 represents the smooth control amount, which punishes excessive energy consumption and encourages smooth control.

[0043] (6) Penalize when the load swings beyond the swing threshold: R_stability=k3*(|θ|+|φ|) Where R_stability represents the penalty for load sway, k3 represents the proportional coefficient of the load sway penalty, which can be set according to needs or actual application conditions, θ represents the polar angle of load sway, and φ represents the azimuth angle of load sway.

[0044] S300, in the simulation environment, based on the key elements, offline training is performed using the Q-learning algorithm, such as... Figure 3 As shown, the main steps are as follows: S301, initialize the Q table as Q(S,A), the behavior state S of the Q table Q(S,A) is the action A.

[0045] S302, explores and trains in a simulation environment, with each round's time step represented as t.

[0046] S303, observe the current state as s_t, which is a discretized state.

[0047] S304, using the ε-greedy strategy to select action a_t: explore randomly with probability ε, and select the action with the largest current Q value with probability 1-ε.

[0048] S305, execute action a_t, the simulation environment transitions to the next state, represented as s_{t+1}, and receives a reward r_t.

[0049] S306, Update the Q-table according to the update rules of the Q-learning algorithm: Q(s_t,a_t)←Q(s_t,a_t)+α*[r_t+γ*max {a_t} Q(s_{t+1},a_t)-Q(s_t,a_t)] Where Q(s_t, a_t) represents the estimated Q value (long-term cumulative reward) of the agent at time t after selecting and executing action a_t in state s_t, s_t represents the current state, a_t represents the selected action, and max {a_t} Q(s_{t+1}, a_t) represents the maximum expected return of the state at the next time step, r_t represents the immediate reward, α represents the learning rate, and γ represents the discount factor.

[0050] S307, Repeat steps 302 to S306 until the Q-table converges, that is, the agent has learned which optimal collaborative handling strategy should be adopted under what fault state.

[0051] The S400 uses a trained Q-table to make online fault handling decisions for a dual-UAV collaborative hoisting system, such as... Figure 4 As shown, the main steps are as follows: S401 deploys the trained Q-table (the converged Q-table) to the dual-UAV collaborative hoisting system for online operation, for example, it can be deployed to the onboard computer or ground control station of the two UAV flight systems.

[0052] S402 can perceive the status of a dual-drone collaborative hoisting system in real time. It can obtain information about drones, loads, and faults in real time through a sensor system, thereby obtaining the status of the dual-drone collaborative hoisting system.

[0053] S403 discretizes the state of the dual-UAV collaborative hoisting system, and the discretized state is represented as s_current.

[0054] S404, query the Q-table obtained from training, and find the action a_best with the maximum Q value under state s_current.

[0055] S405 triggers the underlying controller corresponding to action a_best. The underlying controller generates control commands corresponding to action a_best and sends them to the two drones in the dual-drone collaborative hoisting system for execution.

[0056] S406, repeat the above process until the fault handling is completed.

[0057] Specific application examples: Take, for example, a collaborative lifting system in which two quadcopter drones lift a single load. The two quadcopter drones are each connected to the load by a fixed-length sling, and the load is suspended below the center of the two quadcopter drones.

[0058] The specific implementation steps of the fault handling decision-making method for the dual-UAV collaborative hoisting system provided by this invention are as follows: The S100 uses the MATLAB / Simulink platform to build a dynamic model of a dual-UAV collaborative hoisting system and simulates faults such as motor failure (thrust suddenly drops to 50% or 0%) and single hoisting rope breakage.

[0059] S200 defines key elements for reinforcement learning in a dual-UAV collaborative hoisting system, including state, action, and reward.

[0060] The states include: (1) The states of the two drones are represented as follows: (x,y,z,vx,vy,vz,roll,pitch,yaw,p,q,r,thrust_ratio) Where x, y, z represent the position coordinates of the UAV in the X, Y, Z directions; vx, vy, vz represent the velocities of the UAV in the X, Y, Z directions; roll, pitch, yaw represent the attitudes of the UAV, which are the roll angle in the X direction, the pitch angle in the Y direction, and the yaw angle in the Z direction, respectively; p, q, r represent the angular velocities of the UAV in the X, Y, Z directions; and thrust_ratio represents the remaining motor thrust of the UAV.

[0061] (2) The state of the load is represented as: (sway_angle, swing_rate) Where sway_angle represents the swing angle of the load in the pitch direction; swing_rate represents the swing angle of the load in the yaw direction.

[0062] (3) A discrete flag representing the fault type is denoted as fault_flag.

[0063] For example: fault_flag=0 indicates no fault; fault_flag=1 indicates a fault in the left drone motor; fault_flag=2 indicates a broken sling between the right drone and the payload...

[0064] (4) The relative position of the load and the target point is expressed as: (target_Xpos, target_Ypos) Where target_Xpos represents the horizontal distance of the load relative to the target point; target_Ypos represents the vertical distance of the load relative to the target point.

[0065] Discretization processing: For example, z (height) can be discretized into [too low, low, normal, high, too high]; sway_angle (sway angle) can be discretized into [left large swing, left small swing, no swing, right small swing, right large swing].

[0066] The actions are represented as action set A = {"cooperative landing to point A", "healthy aircraft towed to point B", "emergency drop"}.

[0067] The reward is represented by a reward function: R=R_task+R_crash+R_payload+R_progress+R_energy+R_stability The specific reward points and percentage coefficients may be adjusted depending on the circumstances.

[0068] S300, in the simulation environment, based on the key elements, offline training is performed using the Q-learning algorithm: the learning rate α = 0.1, the discount factor γ = 0.9, and the exploration rate ε is initially 0.9 and decays over time. Training is conducted for millions of time steps until the Q-table converges.

[0069] The S400 utilizes a trained Q-table for online fault handling decisions in its dual-UAV collaborative lifting system: the trained Q-table is burned into the onboard computer. During actual flight, if a right rope breakage fault (fault_flag=2) is detected and the load swings significantly, the system immediately queries the Q-table, may select the "healthy UAV towing" action, and calls the corresponding underlying controller to generate control commands. This commands the left UAV (healthy UAV) to increase its pull and fly towards the predetermined emergency landing point, while the right UAV (faulty UAV) performs hovering or landing to avoid the collision.

[0070] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the fault handling decision-making method flow of the dual-UAV collaborative hoisting system provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 5 As shown, the electronic device may include: At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 5 The example used is the connection between the processor and memory via a bus. The bus... Figure 5 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 5 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0071] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute the fault handling decision-making method for a dual-UAV collaborative hoisting system described above.

[0072] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0073] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0074] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the fault handling decision-making method for a dual-UAV collaborative hoisting system disclosed in the embodiments of this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0075] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0076] By designing and programming the processor, the code corresponding to the fault handling decision-making method of the dual-UAV collaborative hoisting system described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the steps of the method described in the above embodiments during operation. How to design and program the processor is a technique well known to those skilled in the art, and will not be elaborated here.

[0077] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to execute the aforementioned fault handling decision-making method for a dual-UAV collaborative hoisting system.

[0078] In some alternative embodiments, the present invention also provides a method for fault handling decision-making in a dual-UAV collaborative hoisting system, which can also be implemented as a program product including program code. When the program product is run on a device, the program code is used to cause the control device to perform the steps in the fault handling decision-making method for a dual-UAV collaborative hoisting system according to various exemplary embodiments of the present invention as described above.

[0079] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0080] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0083] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0085] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for handling failure of a dual-UAV cooperative hoisting system, characterized in that, The method comprises the following steps: building a simulation environment capable of simulating the dynamics of a dual-unmanned aerial vehicle (UAV) cooperative load-lifting system and various types of faults; defining key elements for reinforcement learning in the dual-UAV cooperative load-lifting system; the key elements include states, actions, and rewards; based on the key elements in the simulation environment, performing offline training using a Q-learning algorithm; using the trained Q table to make online decisions for fault handling of the dual-UAV cooperative load-lifting system. 2.The dual-UAV cooperative hoisting system fault handling decision method according to claim 1, characterized in that, The states include the states of the two UAVs, the state of the load, a discrete flag representing the type of fault, and the relative position of the load and the target point. 3.The method of claim 2, wherein, The states need to be discretized, dividing the continuous states into discrete intervals or levels. 4.The dual-UAV coordinated hoisting system fault handling decision method of claim 1, wherein, The actions include emergency cooperative landing, healthy UAV towing, active anti-swing forced landing, and safe jettisoning and evacuation.

5. The dual-UAV coordinated hoisting system fault handling decision method of claim 4, wherein, Each action is mapped to a bottom controller, which is used to generate control instructions corresponding to each action and send them to the two UAVs in the dual-UAV cooperative load-lifting system for execution.

6. The dual drone coordinated hoisting system failure handling decision method of claim 1, wherein, The rewards include: rewarding when the load successfully reaches the target point; punishing when any of the UAVs crashes; punishing when the load falls or swings beyond the safety physical limit; rewarding when the load-lifting system moves towards the target point; punishing when the energy consumption of the load-lifting system exceeds the energy consumption threshold, and rewarding when the control is smooth; punishing when the load swings beyond the swing threshold.

7. The dual drone coordinated hoisting system failure handling decision method of claim 1, wherein, The offline training using the Q-learning algorithm includes: initializing the Q table represented as Q(S,A), with the behavior state S of the Q table Q(S,A) and the column as the action A; exploring and training in the simulation environment, with each time step of a round represented as t; observing the current state represented as s_t; using an ε-greedy strategy to select the action a_t: with an ε probability of random exploration and a 1-ε probability of selecting the action with the maximum current Q value; executing the action a_t, with the simulation environment moving to the next state represented as s_{t+1} and obtaining the reward r_t; updating the Q table according to the update rule of the Q-learning algorithm: Q(s_t, a_t) ← Q(s_t, a_t) + a * [r_t + γ * max {a_t} Q(s_{t+1},a_t) - Q(s_t,a_t)] wherein Q(s t, a t) represents an estimated Q value after the agent selects and executes the action a t in the state s t at the t time, max {a_t} Q(s_{t+1},a_t) represents the maximum expected return of the next state s_{t+1}, and a represents a learning rate and g represents a discount factor. repeating the exploration and training until the Q table converges.

8. The dual-UAV coordinated hoisting system fault handling decision method of claim 7, wherein, The online fault handling decision-making of the dual-UAV cooperative load-lifting system using the trained Q table includes: deploying the trained Q table to the dual-UAV cooperative load-lifting system for online operation; real-time sensing of the state of the dual-UAV cooperative load-lifting system; discretizing the sensed state of the dual-UAV cooperative load-lifting system, with the discretized state represented as s_current; querying the trained Q table to find the action a_best corresponding to the maximum Q value under the state s_current; triggering the bottom controller corresponding to the action a_best, which generates control instructions corresponding to the action a_best and sends them to the two UAVs in the dual-UAV cooperative load-lifting system for execution; repeating the above process until the fault handling is completed.

9. An electronic device, comprising: The method comprises the following steps: at least one processor; and a memory connected in communication with the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method according to any one of claims 1-8 by executing the instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store instructions that, when executed, cause the method according to any one of claims 1-8 to be implemented.

Images