Unmanned aerial vehicle landing method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle landing method based on deep reinforcement learning, and relates to the technical field of control. The method comprises the following steps that S1, the relative position between the unmanned aerial vehicle and the unmanned ship is obtained; s2, a landing visual system of the unmanned aerial vehicle tracks and positions the unmanned ship; s3, setting and obtaining two reward functions, namely a step length reward function and a time reward function, for the controller of the unmanned aerial vehicle to restrain the landing time of the unmanned aerial vehicle; s4, using a TD3 algorithm to introduce noise so as to avoid convergence to a local optimal solution; when the reviewer network is updated, the target strategy network is updated discontinuously; the TD3 algorithm updates a parameter phi through a deterministic strategy gradient and introduces a hyper-parameter to carry out soft updating; s5, establishing an automatic course learning mechanism, and decomposing a complex task into a series of simpler subtasks; according to the method, efficient landing of the boat is achieved through the steps S1 to S5 and the like.

Description

Technical Field

[0001] The present invention relates to the field of control technology, and in particular to a UAV landing method based on deep reinforcement learning. Background Art

[0002] Currently, with the increasing development of marine resources, air-sea collaborative systems are showing great potential. In maritime search and rescue, exploration, and other scenarios, drones can provide important assistance to unmanned boats by providing positioning data and acting as communication relay stations.

[0003] To address the issue of limited drone endurance, existing methods include replacing drones with batteries with higher energy density and using lighter body materials. However, these methods have limited improvements in endurance, resulting in drones being able to search only near the coast.

[0004] In the existing technology, the short flight time, difficulty and slow landing of drones have become technical problems that need to be solved urgently. Summary of the Invention

[0005] The present invention provides a UAV landing method based on deep reinforcement learning, which solves the technical problems of difficult and slow UAV landing.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0007] A UAV landing method based on deep reinforcement learning includes the following steps:

[0008] Step S1: Obtain the relative position between the UAV and the unmanned boat;

[0009] Step S2: The UAV's landing vision system tracks and locates the unmanned boat;

[0010] Step S3: setting and obtaining two reward functions, namely the step reward function and the time reward function, which are used by the drone controller to constrain the landing time of the drone;

[0011] Step S4: Use the TD3 algorithm to introduce noise to avoid convergence to a local optimal solution; when updating the critic network, the target policy network is updated intermittently; the TD3 algorithm updates the parameter φ through deterministic policy gradients and introduces hyperparameters for soft updates;

[0012] Step S5: Establish an automatic curriculum learning mechanism to decompose complex tasks into a series of simpler subtasks.

[0013] A further technical solution is: in step S1, the kinematic model of the quadrotor UAV is obtained, i.e., formula (3), and the relative position between the UAV and the unmanned boat is calculated according to formula (4);

[0014] Formula (3)

[0015] In formula (3), is the linear velocity component of the UAV in the coordinate system, is the position of the drone in the coordinate system, Indicates the change rate of the drone's Euler angle, To surround the body The angular velocity of the axis of rotation, To surround the body The angular velocity of the axis of rotation, To surround the body Angular velocity of axis rotation; Move on the plane, Indicates the position of the unmanned boat in the world coordinate system;

[0016] Formula (4)

[0017] Formula (4) is the calculation formula for the relative position between the UAV and the unmanned boat.

[0018] A further technical solution is: in step S1, the kinematic equation of the quadrotor drone is obtained, that is, formula (1),

[0019] Formula (1)

[0020] In formula (1), is the quality of the drone, express The acceleration in the axial direction, is the acceleration due to gravity, Indicates the thrust generated by the four propellers;

[0021] Axis and The horizontal motion of the axis is determined by equation (2):

[0022] Formula (2)

[0023] In formula (2), and Respectively Axis and Acceleration in the axial direction; is the total thrust produced by all propellers, is the pitch angle, is the roll angle.

[0024] A further technical solution is that in step S2, an image of the unmanned boat is obtained based on the camera on the drone and sent to a landing vision module running on the drone's controller. The controller, camera and landing vision module on the drone form a landing vision system. The landing vision module is used to use a target recognition algorithm to identify the unmanned boat from the image of the unmanned boat and obtain the position of the unmanned boat.

[0025] A further technical solution is that: in step S3, the landing time of the drone is constrained to prevent the drone from hovering instead of landing, and the drone is encouraged to approach the landing point on the unmanned boat within each time step; the time reward function is used to prevent the drone from delaying time, and the step reward function is used to shorten the total landing distance of the drone as much as possible; the time reward function That is, formula (5), the step reward function That is, formula (6),

[0026] Formula (5)

[0027] Formula (6)

[0028] In formula (6), is the number of pixels exceeding the threshold, express The total number of pixels in ; The value range is [0,1].

[0029] A further technical solution is that: in step S4, the TD3 algorithm reduces the possibility of overestimation of the Q value by taking the minimum value of the output of the two reviewer networks; by introducing noise , which increases the robustness of the target policy and avoids over-reliance on precise policy estimation; the target policy network is intermittently updated to reduce the overestimation problem; the TD3 algorithm updates the parameter φ through deterministic policy gradients and introduces hyperparameters for soft updates to alleviate the overfitting problem.

[0030] A further technical solution is that: in step S4, a double-delay strategy update mechanism is adopted, and the update frequency of the strategy network is lower than that of the Q value network. The critic network is used to evaluate the expected cumulative reward under a given state and action, where the given state is such as the position, speed and visual information of the drone, and the action is such as the speed instruction. The expected cumulative reward is the Q value. By learning the long-term value of the state-action, the policy network is guided to select a better action. are the parameters of the critic network; the target value is calculated using the Bellman equation , the two outputs of the target critic network The smaller one is fed into the Bellman equation to avoid overestimation of the Q value. The target value is calculated as follows:

[0031] Formula (7)

[0032] In formula (7), is the target value; The current reward represents the immediate reward obtained after a strategy takes an action in a certain state. is the discount factor, and its value range is ; Indicates that between the two choices, choose the smaller Q value; Indicates that the status Next, take action The Q-value function after Indicates the introduced fine-tuning value;

[0033] Update the critic network as shown in formula (8):

[0034] Formula (8)

[0035] In formula (8), is the parameter of the Q function, which is The weights of the networks; Indicates the update operation, assigning the optimization result on the right to ; Indicates searching for parameters The value of , which minimizes the objective function; Indicates taking the average of a batch of data.

[0036] A further technical solution is that: in step S4, in the TD3 algorithm, the parameter After a specific training iteration, the deterministic policy gradient is updated, and the policy update equation is as follows:

[0037] Formula (9)

[0038] In formula (9), Find the gradient of the objective function in the policy network, Represents the Q function for action The gradient, Representation Policy Network Parameters gradient;

[0039] Introducing hyperparameters Used for soft updates to alleviate the problem of overfitting during training. The target network is updated as follows:

[0040]

[0041] Formula (10)

[0042] In formula (10), and is the updated state value, and is the current state value.

[0043] A further technical solution is: in step S5, the training process is started under manageable conditions, and the difficulty is gradually increased, and subtasks with increasing difficulty are designed in stages; the subtasks include basic tasks, default tasks and advanced tasks.

[0044] A further technical solution is that in step S5, the basic task includes a single variable, namely the linear speed of the unmanned boat at sea. ,The purpose of introducing the basic tasks is to promote the UAV to learn the basic landing strategy.,The action strategy learned in the basic tasks is used to help the UAV transition from the basic tasks to the subsequent default tasks,,thus filling the gap that occurs during the task switching;

[0045] The default task is used to promote the UAV to learn to land in complex situations. The default task adds two variables on the basis of the basic task, namely angular velocity and acceleration , change regularly The direction is used to enhance the UAV's ability to respond quickly to unexpected environmental changes; by adjusting acceleration and steering, the robustness of the landing process is improved;

[0046] Advanced tasks are used to further enhance the adaptability of the strategy in complex environments and promote the UAV to learn to land under sea breeze disturbance. The disturbance is simulated by adding continuous Gaussian noise to the UAV control command. The amplitude is limited to , the longest duration is ;Will Enhanced to simulate more severe conditions so that the strategy can cope with environmental disturbances during real flight.

[0047] The beneficial effects of adopting the above technical solution are:

[0048] A deep reinforcement learning-based UAV landing method includes the following steps: Step S1: Obtaining the relative position between the UAV and the unmanned boat; Step S2: The UAV's landing vision system tracks and locates the unmanned boat; Step S3: Setting and obtaining two reward functions, namely a step-size reward function and a time reward function, which are used by the UAV controller to constrain the landing time of the UAV; Step S4: Using the TD3 algorithm, noise is introduced to avoid convergence to a local optimal solution; When updating the critic network, the target policy network is discontinuously updated; The TD3 algorithm updates the parameter φ via deterministic policy gradients and introduces hyperparameters for soft updates; Step S5: Establishing an automatic curriculum learning mechanism to decompose complex tasks into a series of simpler subtasks. Through Steps S1 to S5, the method achieves efficient landing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the present invention;

[0050] Figure 2 A vector diagram of a drone;

[0051] Figure 3 This is a diagram of the drone landing status. DETAILED DESCRIPTION

[0052] The purpose of this application is to provide a method for landing a UAV, so as to overcome the technical problems of the prior art UAVs such as short flight time, difficult landing and slow landing.

[0053] This application proposes a drone landing method based on deep reinforcement learning, using the unmanned boat as a mobile platform for charging and maintaining the drone. The collaboration between the aircraft and the boat greatly improves the drone's endurance and solves the problem that drones can only search near the coast.

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0056] Example 1:

[0057] like Figure 1 As shown, the present invention discloses a UAV landing method based on deep reinforcement learning, comprising the following steps:

[0058] Step S1: Set the coordinate system for force analysis, such as Figure 2 As shown, the UAV is a quad-rotor UAV, with four micro motors installed vertically on the frame. Provide lift in the reverse direction, assuming that the UAV only The drone is not affected by gravity and will not deform during flight.

[0059] The kinematic equation of the UAV is shown in formula (1):

[0060] Formula (1)

[0061] In formula (1), is the quality of the drone, express The acceleration in the axial direction, is the acceleration due to gravity, represents the thrust generated by the four propellers. This equation takes into account gravity as well as the thrust generated by the propellers to maintain the drone's altitude.

[0062] Axis and The horizontal motion of the axis is determined by equation (2):

[0063] Formula (2)

[0064] In formula (2), and Respectively Axis and The acceleration in the direction of the axis. is the total thrust produced by all propellers, is the pitch angle, is the roll angle.

[0065] The kinematic model of the UAV can be obtained as shown in formula (3):

[0066] Formula (3)

[0067] In formula (3), is the linear velocity component of the UAV in the coordinate system, is the position of the drone in the coordinate system, Indicates the change rate of the drone's Euler angle, To surround the body The angular velocity of the axis of rotation, To surround the body The angular velocity of the axis of rotation, To surround the body The angular velocity of the axis of rotation. Move on the plane, Indicates the position of the unmanned boat in the world coordinate system.

[0068] Therefore, the relative position between the UAV and the unmanned boat can be calculated according to formula (4):

[0069] Formula (4)

[0070] Formula (4) is the calculation formula for the relative position between the UAV and the unmanned boat.

[0071] Step S2: Design a landing vision system to enable the UAV to identify and locate the target UAV. First, the UAV uses a downward camera attached to the bottom to capture The RGB image of the pixels is then used for object recognition and visual tracking using the landing vision module. This is done by comparing the markers with the provided template. A bounding box is drawn based on the matching results, highlighting the area within the bounding box and darkening the surrounding area.

[0072] If the UAV cannot be detected at the current time, the current state is replaced with the latest state containing the UAV position. Then, a depth-separable convolution is used to locate the UAV within a single image frame, and the output image is normalized to provide the UAV with the motion trend of the UAV. Finally, the RGB image is converted to a grayscale image and then reshaped into , which constitutes the time step The state vector of .

[0073] Step S3: During the landing process of the UAV, Figure 3 As shown in the figure, the drone must land precisely within the specified range on top of the unmanned boat. Secondly, the drone needs to continuously track the unmanned boat. In order to ensure accurate state estimation during the entire landing process, the visibility of the markings on the unmanned boat must be improved. A reward function is also set to shorten the time it takes for the drone to complete the landing. The reward received at each time step is The design is as follows:

[0074] Formula (5)

[0075] In addition, the introduction of reward items , helping the drone to keep up with the moving target at each time step. This reward is related to the state space Associated, drive the UAV close to the UAV landing platform. Set the step reward to , to prevent the drone from hovering in the center of the landing area instead of landing. The definition is as follows:

[0076] Formula (6)

[0077] In formula (6), is the number of pixels exceeding the threshold, express The total number of pixels in . The value range of is [0,1], which prompts the UAV to approach the landing point of the unmanned boat in each time step.

[0078] Step S4: The UAV uses the TD3 algorithm for deep reinforcement learning, which improves the efficiency and stability of the UAV's autonomous landing. A double-delay policy update mechanism is used, in which the update frequency of the policy network is lower than that of the Q-value network. The critic network is used to evaluate the expected cumulative reward under a given state and action. The given state is such as the position, speed, and visual information of the drone, and the action is such as the speed instruction. The expected cumulative reward is the Q value. Its core is to guide the policy network to select a better action by learning the long-term value of the state-action. are the parameters of the critic network. In addition, noise is introduced during the learning process to prevent convergence to unfavorable local optimal solutions. The target value is calculated using the Bellman equation The two outputs of the target critic network The smaller one is fed into the Bellman equation to avoid overestimation of the Q value. The target value is calculated as follows:

[0079] Formula (7)

[0080] In formula (7), is the target value. is the current reward, which represents the immediate reward obtained after a certain action of the strategy in a certain state. is the discount factor, and its value range is . It means that between the two choices, the smaller Q value is selected. Indicates that the status Next, take action The Q-value function after Indicates the introduced fine-tuning value.

[0081] Based on the above formula, update the reviewer network as shown in formula (8):

[0082] Formula (8)

[0083] In formula (8), is the parameter of the Q function, which is The weight of the network. Indicates the update operation, assigning the optimization result on the right to . Indicates searching for parameters value that minimizes the objective function. Indicates taking the average of a batch of data.

[0084] In the TD3 algorithm, the parameters After a specific training iteration, the deterministic policy gradient is updated, and the policy update equation is as follows:

[0085] Formula (9)

[0086] In formula (9), Find the gradient of the objective function in the policy network, Represents the Q function for action The gradient, Representation Policy Network Parameters gradient.

[0087] Introducing hyperparameters Used for soft updates to alleviate the problem of overfitting during training. The target network is updated as follows:

[0088]

[0089] Formula (10)

[0090] In formula (10), and is the updated state value, and is the current state value.

[0091] Step S5: Establish an automatic curriculum learning mechanism to decompose complex tasks into a series of simpler subtasks, start the training process under manageable conditions, and gradually increase the difficulty, designing subtasks with increasing difficulty in stages.

[0092] Basic task: The basic task includes a single variable: the linear speed of the unmanned boat at sea The purpose of introducing the base mission is to facilitate the UAV to learn basic landing strategies. In addition, the action strategies learned in the base mission are intended to help the UAV transition from the base mission to the subsequent default mission, thereby bridging the gap that may occur during task switching.

[0093] Default mission: The default mission is designed to facilitate the drone to learn to land in complex situations. The default mission adds two variables to the basic mission: angular velocity and acceleration , change regularly The goal is to enhance the UAV's ability to respond quickly to unexpected environmental changes. In addition, by adjusting acceleration and steering, the robustness of the landing process can be improved.

[0094] Advanced tasks: To further enhance the adaptability of the strategy in complex environments, an advanced task is introduced to promote the UAV to learn to land under sea breeze interference. The interference is simulated by adding continuous Gaussian noise to the UAV control instructions. The amplitude is limited to , the longest duration is In addition, Enhanced to simulate more severe conditions so that the strategy can cope with environmental disturbances during real flight.

[0095] This approach achieves efficient learning by gradually increasing the difficulty of the learning task to ensure that the drone masters the basic skills required for successful landing.

[0096] The beneficial technical effects of this application are described in detail as follows.

[0097] 1. Strong Adaptability: Deep reinforcement learning enables drones to dynamically adjust their flight strategies based on environmental changes, especially in complex or unknown environments. Drones can learn through interaction with their environment and gradually optimize their landing strategies. This adaptability is particularly important in situations such as changing terrain, weather conditions, and obstacles.

[0098] 2. No human intervention required: In automated and autonomous systems, deep reinforcement learning enables drones to operate unmanned, eliminating the need for extensive manual debugging. Through continuous learning, drones can continuously improve their autonomous landing capabilities, reducing the risks and costs of manual operation.

[0099] 3. Efficiently handle complex tasks: Autonomous drone landing involves multiple factors, such as flight attitude, speed, and landing position. Deep reinforcement learning can simultaneously process this multi-dimensional information, optimize multiple mission objectives, and achieve an efficient landing process.

[0100] In summary, this application analyzes the force and motion relationships of a quadrotor drone in dynamic modeling. Using a drone-mounted camera to capture images, the application combines target detection and tracking algorithms to identify the position of an unmanned vehicle (UAV) and calculate the relative positions of the two. A reward function is designed to enable the drone to quickly and accurately track and land on a moving UAV. By combining the TD3 algorithm, dual-Q network, target strategy smoothing, and delayed update technologies, the efficiency and stability of the drone's landing are improved. Finally, through an automatic course learning mechanism, the tasks are divided into three stages: basic, default, and advanced. The training difficulty is gradually increased, enhancing the drone's adaptability and robustness in complex environments and ensuring successful landings.

[0101] Example 2:

[0102] The present invention discloses a method for landing a UAV based on deep reinforcement learning, comprising the following steps:

[0103] S1: Set the coordinate system for force analysis. The UAV is a quad-rotor UAV. Four micro motors are installed perpendicular to the frame. Provide lift in the reverse direction, assuming that the UAV only The drone is not affected by gravity and will not deform during flight.

[0104] S2: Use the landing vision system to identify and locate the target unmanned vehicle, then use the landing vision module for target recognition and visual tracking. Finally, use depthwise separable convolution to locate the unmanned vehicle within a single image frame, normalize the output image, and provide the drone with the motion trend of the unmanned vehicle.

[0105] S3: First introduce the reward item , the UAV needs to continuously track the UAV to ensure accurate state estimation during the landing process and to maximize the visibility of the marker. In order to shorten the landing time of the UAV, a reward item is introduced. , the reward and state space Associated, drive the UAV close to the landing platform of the unmanned boat. Set the step reward to , to prevent the drone from hovering in the center of the landing area instead of landing.

[0106] S4: Use TD3 algorithm for deep reinforcement learning, adopt target policy smoothing technology, introduce noise in the learning process to prevent convergence to unfavorable local optimal solutions. Use Bellman equation to calculate the target value The two outputs from the target critic network are The smaller one is fed into the Bellman equation to avoid overestimation of the Q value. While updating the critic network, the target policy network is updated intermittently. In TD3, the parameter After a specific training iteration, the policy gradient is updated by the deterministic policy. Finally, the hyperparameters are introduced. Used for soft updates to alleviate the problem of overfitting during training.

[0107] S5: Establish an automatic curriculum learning mechanism to break down complex tasks into a series of simple subtasks. Start the training process under manageable conditions and gradually increase the difficulty. Design subtasks with increasing difficulty in stages. This method ensures that the drone masters the basic skills required for successful landing by gradually increasing the difficulty of the learning tasks, thereby achieving efficient learning.

[0108] The similarities between Example 2 and Example 1 are not repeated here.

[0109] Based on Example 2, preferably, the speed of the UAV and its position relative to the UAV are analyzed, and an equation is established, wherein: is the quality of the drone, express The acceleration in the axial direction, is the acceleration due to gravity, Indicates the This equation takes into account gravity as well as the thrust generated by the propellers to keep the drone aloft.

[0110] Based on Example 2, preferably, a UAV speed equation is established, wherein, and Respectively Axis and The acceleration in the direction of the axis. is the total thrust produced by all propellers, is the pitch angle, is the roll angle. A UAV kinematic model is established, where is the position of the UAV in the coordinate system. Move on the plane, Represents its position in the world coordinate system.

[0111] Based on Example 2, preferably, in step S3, two reward functions are set: a time reward function and a step reward function, which constrain the time and path selection of the UAV landing.

[0112] Based on Example 2, preferably, in step S4, the TD3 algorithm integrates some key features, improves its efficiency and stability, introduces noise, and prevents convergence to an unfavorable local optimal solution.

[0113] Based on Example 2, an automated curriculum learning mechanism is preferably established, breaking down complex tasks into a series of simpler subtasks. The training process begins with manageable conditions and gradually increases the difficulty, designing subtasks of increasing difficulty in stages. This approach ensures that the drone masters the basic skills required for successful landing by gradually increasing the difficulty of the learning tasks, thereby achieving efficient learning.

[0114] Example 3:

[0115] The present invention discloses a method for landing a UAV based on deep reinforcement learning, comprising the following steps:

[0116] S1: Set the coordinate system for force analysis. The UAV is a quad-rotor UAV. Four micro motors are installed perpendicular to the frame. The reverse direction provides lift, setting the drone to only The drone is not affected by gravity and will not deform during flight.

[0117] S2: Use the landing vision system to locate the unmanned boat, normalize the output image, and provide the UAV with the movement trend of the unmanned boat.

[0118] S3: Two reward functions are set, namely the step reward function and the time reward function, which constrain the landing time of the drone, prevent the drone from hovering instead of landing, and encourage the drone to approach the landing point of the unmanned boat within each time step.

[0119] S4: Using the TD3 algorithm, this approach employs target policy smoothing techniques to introduce noise to prevent convergence to local optima. When updating the critic network, the target policy network is updated intermittently, minimizing overestimation. TD3 updates the parameter φ using deterministic policy gradients and introduces hyperparameters for soft updates to mitigate overfitting.

[0120] S5: Establish an automatic course learning mechanism to decompose complex tasks into a series of simpler subtasks. By gradually increasing the difficulty of the learning tasks, the UAV can master the basic skills required for successful landing, thereby achieving efficient learning.

[0121] The similarities between Example 3 and Example 1 are not repeated here.

[0122] Based on Example 3, preferably, in step S1, a speed decomposition formula of the UAV is constructed, and the relative positions of the UAV and the unmanned boat are calculated using a distance formula.

[0123] Based on Example 3, preferably, a landing vision system is used to identify and locate the target unmanned boat, and then a landing vision module is used to perform target recognition and visual tracking. Finally, a depthwise separable convolution is used to locate the unmanned boat within a single image frame, and the output image is normalized to provide the drone with the motion trend of the unmanned boat.

[0124] Based on Example 3, preferably, two reward functions are set: a time reward function and a step-size reward function. The time reward function is set to prevent the drone from taking meaningless actions to delay time, and the step-size reward function makes the total landing distance of the drone as short as possible.

[0125] Based on Example 3, preferably, compared to the DDPG algorithm, TD3 reduces the possibility of overestimation of the Q value by taking the minimum value of the output of the two critic networks. , which increases the robustness of the target policy and avoids over-reliance on precise policy estimates.

[0126] Based on Example 3, preferably, an automatic course learning mechanism is established to decompose complex tasks into a series of simpler subtasks, start the training process under manageable conditions, and gradually increase the difficulty. Subtasks with increasing difficulty are designed in stages. This method ensures that the drone masters the basic skills required for successful landing by gradually increasing the difficulty of the learning tasks, thereby achieving efficient learning.

Claims

1. A UAV landing method based on deep reinforcement learning, characterized by: The following steps are included: Step S1: Obtain the relative position between the UAV and the unmanned boat; Step S2: The UAV's landing vision system tracks and locates the unmanned boat; Step S3: setting and obtaining two reward functions, namely the step reward function and the time reward function, which are used by the drone controller to constrain the landing time of the drone; Step S4: Use the TD3 algorithm to introduce noise to avoid convergence to a local optimal solution; when updating the critic network, the target policy network is updated intermittently; the TD3 algorithm updates the parameter φ through deterministic policy gradients and introduces hyperparameters for soft updates; Step S5: Establish an automatic curriculum learning mechanism to decompose complex tasks into a series of simpler subtasks.

2. The UAV landing method based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the kinematic model of the quadrotor UAV is obtained, i.e., formula (3), and the relative position between the UAV and the unmanned boat is calculated according to formula (4); Formula (3) In formula (3), is the linear velocity component of the UAV in the coordinate system, is the position of the drone in the coordinate system, Indicates the change rate of the drone's Euler angle, To surround the body The angular velocity of the axis of rotation, To surround the body The angular velocity of the axis of rotation, To surround the body Angular velocity of axis rotation; Move on the plane, Indicates the position of the unmanned boat in the world coordinate system; Formula (4) Formula (4) is the calculation formula for the relative position between the UAV and the unmanned boat.

3. The method for landing a drone based on deep reinforcement learning according to claim 2, characterized in that: In step S1, the kinematic equation of the quadrotor drone is obtained, that is, equation (1): Formula (1) In formula (1), is the quality of the drone, express The acceleration in the axial direction, is the acceleration due to gravity, Indicates the thrust generated by the four propellers; Axis and The horizontal motion of the axis is determined by equation (2): Formula (2) In formula (2), and Respectively Axis and Acceleration in the axial direction; is the total thrust produced by all propellers, is the pitch angle, is the roll angle.

4. The method for landing a drone based on deep reinforcement learning according to claim 1, characterized in that: In step S2, an image of the unmanned boat is obtained based on the camera on the drone and sent to a landing vision module running on the controller of the drone. The controller, camera and landing vision module on the drone form a landing vision system. The landing vision module is used to use a target recognition algorithm to identify the unmanned boat from the image of the unmanned boat and obtain the position of the unmanned boat.

5. The method for landing a drone based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the landing time of the UAV is constrained to prevent the UAV from hovering instead of landing, and to encourage the UAV to approach the landing point on the UAV within each time step; the time reward function is used to prevent the UAV from delaying time, and the step reward function is used to shorten the total landing distance of the UAV as much as possible; the time reward function That is, formula (5), the step reward function That is, formula (6), Formula (5) Formula (6) In formula (6), is the number of pixels exceeding the threshold, express The total number of pixels in ; The value range is [0,1].

6. The method for landing a drone based on deep reinforcement learning according to claim 1, characterized in that: In step S4, the TD3 algorithm reduces the possibility of overestimation of the Q value by taking the minimum value of the output of the two reviewer networks; by introducing noise , which increases the robustness of the target policy and avoids over-reliance on precise policy estimation; the target policy network is intermittently updated to reduce the overestimation problem; the TD3 algorithm updates the parameter φ through deterministic policy gradients and introduces hyperparameters for soft updates to alleviate the overfitting problem.

7. The method for landing a drone based on deep reinforcement learning according to claim 1, characterized in that: In step S4, a double-delay strategy update mechanism is adopted, and the update frequency of the strategy network is lower than that of the Q value network. The critic network is used to evaluate the expected cumulative reward under a given state and action, where the given state is such as the position, speed and visual information of the drone, and the action is such as the speed instruction. The expected cumulative reward is the Q value. By learning the long-term value of the state-action, the policy network is guided to select a better action. are the parameters of the critic network; the target value is calculated using the Bellman equation , the two outputs of the target critic network The smaller one is fed into the Bellman equation to avoid overestimation of the Q value. The target value is calculated as follows: Formula (7) In formula (7), is the target value; The current reward represents the immediate reward obtained after a strategy takes an action in a certain state. is the discount factor, and its value range is ; Indicates that between the two choices, the smaller Q value is selected; Indicates that the status Next, take action The Q-value function after Indicates the introduced fine-tuning value; Update the critic network as shown in formula (8): Formula (8) In formula (8), is the parameter of the Q function, which is The weights of the networks; Indicates the update operation, assigning the optimization result on the right to ; Indicates searching for parameters The value of , which minimizes the objective function; Indicates taking the average of a batch of data.

8. The method for landing a drone based on deep reinforcement learning according to claim 7, characterized in that: In step S4, in the TD3 algorithm, the parameter After a specific training iteration, the deterministic policy gradient is updated, and the policy update equation is as follows: Formula (9) In formula (9), Find the gradient of the objective function in the policy network, Represents the Q function for action The gradient, Representation Policy Network Parameters gradient; Introducing hyperparameters Used for soft updates to alleviate the problem of overfitting during training. The target network is updated as follows: Formula (10) In formula (10), and is the updated state value, and is the current state value.

9. The method for landing a drone based on deep reinforcement learning according to claim 1, characterized in that: In step S5, the training process is started under manageable conditions, and the difficulty is gradually increased, and subtasks with increasing difficulty are designed in stages; the subtasks include basic tasks, default tasks and advanced tasks.

10. The method for landing a UAV based on deep reinforcement learning according to claim 9, characterized in that: In step S5, the basic task includes a single variable, namely the linear speed of the unmanned boat at sea. ,The purpose of introducing the basic tasks is to promote the UAV to learn the basic landing strategy.,The action strategy learned in the basic tasks is used to help the UAV transition from the basic tasks to the subsequent default tasks,,thus filling the gap that occurs during the task switching; The default task is used to promote the UAV to learn to land in complex situations. The default task adds two variables on the basis of the basic task, namely angular velocity and acceleration , change regularly The direction is used to enhance the UAV's ability to respond quickly to unexpected environmental changes; by adjusting acceleration and steering, the robustness of the landing process is improved; Advanced tasks are used to further enhance the adaptability of the strategy in complex environments and promote the UAV to learn to land under sea breeze disturbance. The disturbance is simulated by adding continuous Gaussian noise to the UAV control command. The amplitude is limited to , the longest duration is ;Will Enhanced to simulate more severe conditions so that the strategy can cope with environmental disturbances during real flight.

Images