Unmanned aerial vehicle two-stage target searching and tracking method based on deep reinforcement learning

Abstract

The invention provides an unmanned aerial vehicle two-stage target searching and tracking method based on deep reinforcement learning. The method comprises the steps that a searching and rescue task is divided into a searching stage and a tracking stage; a corresponding decision model DDPG-3C is carried in each stage, and switching between a searching stage and a tracking stage is carried out by judging whether a detection target is in a radar detection range of the unmanned aerial vehicle or not. The control strategy and algorithm can be optimized in a targeted manner in each stage, the search range is expanded and the discovery speed is improved in the search stage, the tracking precision and the response speed of a dynamic target are improved in the tracking stage, and the efficiency of each stage is improved; the DDPG-3C model does not depend on an accurate mathematical model to make decisions, so that the method can show stronger adaptability when facing a complex and dynamically changing environment, learns and optimizes a strategy through interaction with the environment instead of depending on predefined model parameters, and can better cope with unknown or changing conditions.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle rescue technology, and in particular to a two-stage target search and tracking method for unmanned aerial vehicles based on deep reinforcement learning. Background Art

[0002] With the continuous advancement of technology, drones have achieved significant improvements in intelligence, endurance, payload capacity, anti-interference, etc., enabling them to adapt to more diverse application scenarios.

[0003] However, in disaster rescue missions, such as after earthquakes, floods or mudslides, the disaster areas are often complex in terrain and communication is interrupted. UAVs may face problems such as damaged infrastructure, harsh natural environment, and interference from human factors, which puts the UAV in a communication denial environment. At this time, the perception ability of the UAV may be seriously affected, making it difficult for the UAV to find trapped people or important clues, reducing the success rate of search and rescue. Secondly, in a communication denial environment, the UAV may not be able to receive ground control instructions in real time, which places great demands on the UAV's autonomous control decision-making ability.

[0004] In recent years, some advanced technologies such as model predictive control, optimization-based methods and intelligent optimization algorithms have been successfully applied to the decision-making process of UAVs. These methods provide UAVs with the ability to efficiently navigate and perform tasks in complex environments. However, these methods have obvious limitations. On the one hand, many methods rely on precise mathematical models, but these models are difficult to fully apply to actual, complex and changing environments. On the other hand, these methods are heavily dependent on sensor information, and the accuracy and reliability of sensors are often interfered by environmental factors. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a two-stage target search and tracking method for an unmanned aerial vehicle based on deep reinforcement learning. The present invention divides the search and rescue mission into a search phase and a tracking phase to improve the ability of the unmanned aerial vehicle to detect and track targets in a post-disaster rescue environment.

[0006] The technical solution of the present invention is: a two-stage target search and tracking method for unmanned aerial vehicles based on deep reinforcement learning, comprising the following steps:

[0007] Step 1: Divide the search and rescue mission into the search phase and the tracking phase;

[0008] Step 2: Construct a deep deterministic policy gradient decision model DDPG-3C and train the decision model DDPG-3C accordingly according to the search phase and tracking phase;

[0009] Step 3: Load the trained corresponding deep deterministic policy gradient decision model DDPG-3C into the drone controllers in the search phase and tracking phase respectively;

[0010] Step 4: Determine whether the detected target is within the radar detection range of the drone to switch between the search phase and the tracking phase.

[0011] Preferably, the deep deterministic policy gradient decision model DDPG-3C includes an Actor module and a Critic module; wherein the Actor module includes an Actor network and a target Actor network; the Critic module includes three Critic networks and three target Critic networks; in the process of interaction between the drone and the environment, the drone is regarded as an intelligent agent equipped with the decision model DDPG-3C. When the intelligent agent receives state information s from the environment i When the Actor network is based on the state information s i Output the corresponding action a i ; The agent then performs action a i ; After the action is performed, the environment will respond to it i ′ and returns the corresponding reward r i to the agent; then the agent sends the state information s i 、Action a i , Responses i ′, reward r i Stored in the experience replay buffer; the intelligent body will periodically randomly extract a small batch of data samples from the experience replay buffer to update the parameters of the Actor network and the Critic network.

[0012] Preferably, the deep deterministic policy gradient decision model DDPG-3C adopts an experience replay buffer mechanism, which removes the maximum value of the three estimated Q values ​​and takes the average of the remaining two estimated Q values ​​as the target estimated Q value.

[0013] Preferably, the Actor network is responsible for converting the environment state into actions, and its goal is to learn a strategy to maximize the expected return in the future, namely:

[0014]

[0015] Where R1 represents the reward value obtained from state s0; τ μ represents the state distribution under the strategy μ; θ μ is the parameter of the Actor network; J(θ μ ) represents the average reward value obtained from state s0 under strategy μ; represents mathematical expectation;

[0016] The following update gradient formula is obtained through the policy gradient theorem:

[0017]

[0018] In the formula, Represents the strategy relative to the Actor network parameter θ μ The gradient of

[0019] represents the gradient of the action-value function with respect to action a; N represents the batch size, that is, the amount of experience data sampled from the playback buffer during each training iteration; when calculating the loss of the Actor network, only the first Critic network is used to predict the Q value; represents the parameters of the Critic network, and Q represents the output value of the Critic network.

[0020] Preferably, the parameters of the target Actor network are updated via a soft update strategy:

[0021] θ μ′ =τθ μ +(1-τ)θ μ′ ;

[0022] In the formula, θ μ′ represents the parameters of the target Actor network; τ is the soft update coefficient; θ μ Parameters for the Actor network.

[0023] Preferably, the Q value is estimated independently by three Critic networks and three target Critic networks, the three Critic networks have the same structure, and each Critic network estimates the Q value according to a given state and action pair (state information s, action a); the largest Q value is removed in each iteration, and the other two Q values ​​are averaged as the current target Q value; that is:

[0024]

[0025] In the formula, y t is the update target of the Critic network; r t Represents the reward value obtained from the environment; γ represents the discount factor, which is used to calculate the present value of future rewards and helps determine the long-term optimization strategy; a′ represents the action output by the target Critic network according to the next state s′; Represents the lower two target Q values ​​output by the three-target Critic network.

[0026] Preferably, the loss function of the Critic network is expressed as:

[0027]

[0028] In the formula, MSE represents the root mean square error, Represents the Q value output by the corresponding Critic network;

[0029] After the loss value is calculated by the loss function, the parameters of the Critic network are updated by the following update gradient formula:

[0030]

[0031] In the formula, Indicates the loss value Relative to the Critic network parameters The gradient of ; N represents the batch size;

[0032] The parameters of the target critic network are Update via soft update strategy:

[0033]

[0034] τ is the soft update coefficient.

[0035] Preferably, the Actor network and the Critic network each include a corresponding input layer, a first Dense layer, a second Dense layer, a third Dense layer, and an output layer.

[0036] Preferably, in the search phase, the optimization goal of the drone controller is to enable the drone to cover the entire environment and quickly locate the target; if the target is detected by the drone's radar, the drone will switch to tracking mode; in the tracking phase, the goal of the drone controller is to continuously track dynamic targets.

[0037] Preferably, in the search phase, the state space of the reinforcement learning of the drone is:

[0038] S=drone x ,drone y ,drone vx ,drone vy ,obstacle dx ,obstacle dy ]

[0039] If the obstacle is not within the drone's radar range, the obstacle's distance is marked as 1.

[0040] In the formula, drone xIndicates the position of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0041] drone y Indicates the position of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0042] drone vx Indicates the speed of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0043] drone vy Indicates the speed of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0044] obstacle dx Indicates the distance of the obstacle detected by the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [0,1];

[0045] obstacle dy Indicates the distance of the obstacle detected by the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [0,1].

[0046] As a preferred method, the reward function in the search scenario includes process rewards and terminal rewards. Each step generates a process reward r step , to encourage the agent to quickly complete the coverage task. At the same time, when the agent covers a new area, it will receive a process reward r coverage ; If the range covered by the drone (agent) reaches 95% of the entire rescue area, the terminal will be rewarded r complete ; If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion , in the search scenario reward function R is as follows:

[0047] R=∑r step +p∑r coverage +r complete +r collosion ;

[0048] In the formula, p is the reward ratio;

[0049] Preferably, in the tracking stage, the state space of the reinforcement learning of the drone is:

[0050] S=drone x ,drone y ,drone vx ,drone vy ,target dx ,target dy,target vx ,target vy ,obstacle dx ,obstacle dy ]

[0051] In the formula, target dx ,target dy Respectively represent the distance of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane; target vx ,target vy They represent the speed of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane respectively; their normalized range is [0,1]; if the target is not within the radar range of the drone, the distance of the target is marked as 1 and the speed is marked as 0.

[0052] Preferably, during the tracking phase, each step of the drone receives a reward r for approaching the target. pro ; The reward r for approaching the target pro Expressed as the Euclidean distance between the drone and the target:

[0053]

[0054] In the formula, the 0.5 in the denominator is used to avoid the denominator becoming zero when the distance is very close or zero, ensuring the stability of the formula. If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion ; Therefore, the reward function of the tracking phase is as follows:

[0055] R=∑r pro +r collosion .

[0056] The beneficial effects of the present invention are:

[0057] 1. The present invention divides the search and rescue mission into two stages: target search and target tracking. Each stage can optimize its control strategy and algorithm in a targeted manner. This phased approach allows the search range to be expanded and the discovery speed to be improved in the search stage, while the tracking stage focuses on improving the tracking accuracy and response speed of dynamic targets. This not only improves the efficiency of each stage, but also makes the entire system more flexible and can adjust strategies according to different operational requirements.

[0058] 2. The present invention reduces the reliance on complex reward functions, simplifies the model training process, and reduces computational complexity by optimizing reward design;

[0059] 3. The DDPG-3C model of the present invention does not rely on precise mathematical models to make decisions, so that the present invention can show stronger adaptability when facing complex and dynamically changing environments, and learn and optimize strategies through interaction with the environment instead of relying on predefined model parameters, so as to better cope with unknown or changing conditions;

[0060] 4. DDPG-3C of the present invention significantly reduces the over-estimation problem and improves the accuracy of value estimation by introducing three Critic networks and removing the maximum value among the three Q values, and using the average value of the remaining two as the target Q value, thereby making the strategy learning process more stable, accelerating the convergence speed, and improving the reliability of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a structural framework diagram of the deep deterministic policy gradient decision model DDPG-3C of Example 1 of the present invention;

[0062] Figure 2 A flowchart of a two-stage target search and tracking method according to Embodiment 1 of the present invention;

[0063] Figure 3 This is a structural block diagram of the Actor network and the Critic network of Example 1 of the present invention;

[0064] Figure 4 This is a schematic diagram of the implementation process of the UAV using the decision model DDPG-3C in the search phase of Example 1 of the present invention;

[0065] Figure 5 is a schematic diagram of a two-dimensional search and rescue environment according to Embodiment 2 of the present invention;

[0066] Figure 6 The training results of the DDPG-3C model in the search phase of Example 2 of the present invention are as follows: (a) is a curve graph of the round reward value; (b) is a curve graph of the target Q value;

[0067] Figure 7 The training results of the DDPG-3C model in the tracking phase of Example 2 of the present invention are as follows: (a) is a curve graph of the round reward value; (b) is a curve graph of the target Q value;

[0068] Figure 8 This is a schematic diagram of experimental results of Example 2 of the present invention using the method of Example 1 in a search and rescue (SAR) mission. DETAILED DESCRIPTION

[0069] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0070] Example 1

[0071] like Figure 1 As shown, this embodiment provides a two-stage target search and tracking method for a drone based on deep reinforcement learning.

[0072] Step 1: Divide the search and rescue mission into the search phase and the tracking phase;

[0073] Step 2: Construct a deep deterministic policy gradient decision model DDPG-3C and train the decision model DDPG-3C accordingly according to the search phase and tracking phase;

[0074] Step 3: Load the trained corresponding deep deterministic policy gradient decision model DDPG-3C into the drone controllers in the search phase and tracking phase respectively;

[0075] Step 4: Determine whether the detected target is within the radar detection range of the drone to switch between the search phase and the tracking phase.

[0076] As a preferred embodiment of the present invention, the deep deterministic policy gradient decision model DDPG-3C includes an Actor module and a Critic module; wherein the Actor module includes an Actor network and a target Actor network; the Critic module includes three Critic networks and three target Critic networks; in the process of interaction between the drone and the environment, the drone is regarded as an intelligent agent equipped with the decision model DDPG-3C. When the intelligent agent receives state information s from the environment i When the Actor network is based on the state information s i Output the corresponding action a i ; The agent then performs action a i ; After the action is performed, the environment will respond to it i ′ and returns the corresponding reward r i to the agent; then the agent sends the state information s i 、Action a i , Responses i ′, reward r i Stored in the experience replay buffer; the intelligent body will periodically randomly extract a small batch of data samples from the experience replay buffer to update the parameters of the Actor network and the Critic network.

[0077] As a preferred embodiment of this embodiment, the deep deterministic policy gradient decision model DDPG-3C adopts an experience replay buffer mechanism, which removes the maximum value of the three estimated Q values ​​and takes the average of the remaining two estimated Q values ​​as the target estimated Q value.

[0078] As a preferred embodiment of this invention, the Actor network is responsible for converting the environment state into actions, and its goal is to learn a strategy to maximize the expected return in the future, namely:

[0079]

[0080] Where R1 represents the reward value obtained from state s0; ρ μ represents the state distribution under the strategy μ; θ μ is the parameter of the Actor network; J(θ μ ) represents the average reward value obtained from state s0 under strategy μ; represents mathematical expectation;

[0081] The following update gradient formula is obtained through the policy gradient theorem:

[0082]

[0083] In the formula, Represents the strategy relative to the Actor network parameter θ μ The gradient of

[0084] represents the gradient of the action-value function with respect to action a; N represents the batch size, that is, the amount of experience data sampled from the playback buffer during each training iteration; when calculating the loss of the Actor network, only the first Critic network is used to predict the Q value; represents the parameters of the Critic network, and Q represents the output value of the Critic network.

[0085] As a preferred embodiment of this embodiment, the parameters of the target Actor network are updated through a soft update strategy:

[0086] θ μ′ =τθ μ +(1-τ)θ μ′ ;

[0087] In the formula, θ μ′ represents the parameters of the target Actor network; τ is the soft update coefficient; θ μ Parameters for the Actor network.

[0088] As a preferred embodiment of this invention, the Q value is estimated independently by three Critic networks and three target Critic networks, the three Critic networks have the same structure, and each Critic network estimates the Q value according to a given state and action pair (state information s, action s); the largest Q value is removed in each iteration, and the other two Q values ​​are averaged as the current target Q value; that is:

[0089]

[0090] In the formula, y t is the update target of the Critic network; r t Represents the reward value obtained from the environment; γ represents the discount factor, which is used to calculate the present value of future rewards and helps determine the long-term optimization strategy; a′ represents the action output by the target Critic network according to the next state s′; Represents the lower two target Q values ​​output by the three-target Critic network.

[0091] As a preferred embodiment of this embodiment, the expression of the loss function of the Critic network is:

[0092]

[0093] In the formula, MSE represents the root mean square error, Represents the Q value output by the corresponding Critic network;

[0094] After the loss value is calculated by the loss function, the parameters of the Critic network are updated by the following update gradient formula:

[0095]

[0096] In the formula, Indicates the loss value Relative to the Critic network parameters The gradient of ; N represents the batch size;

[0097] The parameters of the target critic network are Update via soft update strategy:

[0098]

[0099] τ is the soft update coefficient.

[0100] As a preferred embodiment of this embodiment, the Actor network and the Critic network both include a corresponding input layer, a first Dense layer, a second Dense layer, a third Dense layer, and an output layer. For details, see Figure 3 As shown in the figure, s i Indicates the state of the drone, a i represents the action, Q represents the output value of the Critic network, input_dim represents the size of the network input layer dimension, fc1_dim and fc2_dim represent the size of the network hidden layer dimension, and output_dim represents the size of the network output layer dimension.

[0101] As a preferred embodiment of this invention, in the search phase, the training of the deep deterministic policy gradient decision model DDPG-3C includes the following steps:

[0102] S211), initializing the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network;

[0103] S212), obtaining the state observation of the UAV from the search scene;

[0104] S213), input the state observation into the Actor network, and the Actor network outputs action a t ;

[0105] S214), the drone performs action a t , get the reward value r from the search scene t and the next state observation s of the drone t+1 ;

[0106] S215) Experience value (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer;

[0107] S216), sampling a small batch of data from the experience playback buffer;

[0108] S217), updating the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network according to the gradient formula and the soft update strategy;

[0109] S218), repeating steps S213)-S217), until the round ends in the search scene;

[0110] S219) Reset the search scene and repeat steps S212)-S218) until the set number of training times is completed; save the structure and parameters of the Actor network.

[0111] As a preferred embodiment of this invention, in the tracking stage, the training of the deep deterministic policy gradient decision model DDPG-3C includes the following steps:

[0112] S221), initializing the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network;

[0113] S222), obtaining the state observation of the UAV from the tracking scene;

[0114] S223), input the state observation into the Actor network, and the Actor network outputs action a t ;

[0115] S224), the drone performs action a t , get the reward value r from the search scene t and the next state observation s of the drone t+1 ;

[0116] S225) Experience value (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer;

[0117] S226), sampling a small batch of data from the experience playback buffer;

[0118] S227), updating the parameters of the Actor network, the target Actor network, the Critic network, and the target Critic network according to the gradient formula and the soft update strategy;

[0119] S228), repeating steps S223)-S227), until the round ends in the search scene;

[0120] S229) Reset the tracking scene and repeat steps S222)-S228) until the set number of training times is completed; save the structure and parameters of the Actor network.

[0121] As a preferred embodiment of the present invention, in the search phase, the optimization goal of the drone controller is to enable the drone to cover the entire environment and quickly locate the target; if the target is detected by the drone's radar, the drone will switch to tracking mode; in the tracking phase, the goal of the drone controller is to continuously track dynamic targets.

[0122] As a preferred embodiment of this invention, in the search phase, the state space S of the reinforcement learning of the drone is:

[0123] S=drone x ,drone y ,drone vx ,drone vy ,obstacle dx ,obstacle dy ]

[0124] If the obstacle is not within the drone's radar range, the obstacle's distance is marked as 1.

[0125] In the formula, drone xIndicates the position of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0126] drone y Indicates the position of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0127] drone vx Indicates the speed of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0128] drone vy Indicates the speed of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1,1];

[0129] abstacle dx Indicates the distance of the obstacle detected by the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [0,1];

[0130] obstacle dy Indicates the distance of the obstacle detected by the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [0,1].

[0131] The action space of the drone is a two-dimensional vector [F x ,F y ]; F y are the thrusts generated by the propellers in the x-axis and y-axis directions respectively; considering the maneuverability of the drone, the action value is not greater than the maximum thrust F generated by the propellers max .

[0132] As a preferred embodiment of this invention, the reward function in the search scenario includes a process reward and a terminal reward. Each step generates a process reward r step = -0.1 to encourage the agent to quickly complete the coverage task. At the same time, when the agent covers a new area, it will receive a process reward r coverage ; If the drone (agent) covers 95% of the entire rescue area, a terminal reward will be given

[0133] r complete =300; if the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion = -200, the reward function R in the search scenario is as follows:

[0134] R=∑r step +p∑r coverage +r complete +r collosion ;

[0135] Wherein, p is the reward ratio, and in this embodiment, p=0.001.

[0136] As a preferred embodiment of this invention, in the tracking stage, the state space of the reinforcement learning of the drone is:

[0137] S=drone x ,drone y ,drone vx ,drone vy ,target dx ,target dy ,target vx ,target vy ,obstacle dx ,obstacle dy ]

[0138] In the formula, target dx ,target dy Respectively represent the distance of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane; target vx ,target vy They represent the speed of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane respectively; their normalized range is [0,1]; if the target is not within the radar range of the drone, the distance of the target is marked as 1 and the speed is marked as 0.

[0139] The action space of the drone is a two-dimensional vector [F x ,F y ]; F y are the thrusts generated by the propellers in the x-axis and y-axis directions respectively; considering the maneuverability of the drone, the action value is not greater than the maximum thrust F generated by the propellers max .

[0140] As a preferred embodiment of the present invention, during the tracking phase, each step of the drone will receive a reward r for approaching the target. pro ; The reward r for approaching the target pro Expressed as the Euclidean distance between the drone and the target:

[0141]

[0142] In the formula, the 0.5 in the denominator is used to avoid the denominator becoming zero when the distance is very close or zero, ensuring the stability of the formula. If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion ; Therefore, the reward function of the tracking phase is as follows:

[0143] R=∑r pro +r collosion .

[0144] As a preferred embodiment of the present invention, in step 4, it is determined whether the detected target is within the radar detection range of the drone to switch between the search phase and the tracking phase, such as Figure 2 and 4 As shown, the specific steps include:

[0145] S41), the drone controller loads the structure and parameters of the Actor network saved in the search phase and the tracking phase;

[0146] S42), obtaining state observations of the drone from the search and rescue environment;

[0147] S43), determine whether the target is within the radar range of the drone, if the drone radar does not detect the target, go to step S44), if the drone radar detects the target, go to step S46);

[0148] S44), inputting the state observation of the UAV into the decision model DDPG-3C in the search phase;

[0149] S45), the decision model DDPG-3C in the search phase outputs the action to the drone for execution, and goes to step S42);

[0150] S46), inputting the state observation of the UAV into the decision model DDPG-3C in the tracking phase;

[0151] S47), the decision model DDPG-3C in the tracking phase outputs the action to the drone for execution, and goes to step S42);

[0152] S48) Repeat S42)-S43) until the round ends.

[0153] Example 2

[0154] In order to verify the effectiveness of the method and model proposed in Example 1, this embodiment first establishes a two-dimensional simulation search and rescue environment. Figure 5 The figure shows a simplified two-dimensional search and rescue (SAR) environment; the scene is represented by a two-dimensional rectangular area with a length of L and a width of W. The dark blue drone icon represents the drone (agent); the blue circular icon represents the target waiting for rescue; the dark green circular area represents an obstacle or dangerous area with a radius of r. In this scene, the drone is equipped with a radar device that senses the speed and distance of approaching objects, and its sensing range is represented by a red circle.

[0155] In this two-dimensional simulation search and rescue environment, the initial positions and velocities of the drone and the search target are random. The search target maintains uniform linear motion, and it is assumed that the collision between the target and the obstacle is a completely elastic collision, and it continues to maintain uniform linear motion after the collision. The motion of the drone changes its speed through the thrust generated by the propellers in the x-axis and y-axis directions, and its motion model is:

[0156]

[0157] In the formula, F x 、F y are the thrusts generated by the propellers in the x-axis and y-axis directions respectively; their magnitude range is F x 、F y ∈[-Fmax max ;v x 、v y are the speeds of the drone in the x-axis and y-axis directions respectively; their magnitude range is v x 、v y ∈[-vmax max ; Δt represents the sampling interval; m represents the mass of the UAV.

[0158] When training the decision model DDPG-3C, the hyperparameters of the decision model DDPG-3C are shown in Table 1.

[0159] Table 1 Hyperparameters of decision model DDPG-3C

[0160]

[0161]

[0162] During the search phase, the decision model DDPG-3C was trained with no targets in the search scene. The goal of the drone was to quickly and efficiently cover the entire 2D rescue area. The 2D SAR scene was 800 meters long and wide, and included a drone with a sensor range of 200 meters and two obstacles with a radius of 25 meters. The obstacles were stationary, while the drone changed speed through the thrust generated by the propellers in the x-axis and y-axis directions, with a maximum action value of 400N. The decision model DDPG-3C was trained under the same conditions using the hyperparameters in Table 1.

[0163] The training results of the DDPG-3C decision model in the search phase of the search and rescue (SAR) mission in a denied environment are as follows: Figure 6 (a) and (b). To visualize the trend more clearly, the exponential moving average (EMA) is used to calculate the results:

[0164]

[0165] Among them, s=0.95 is the smoothing factor.

[0166] As the number of training steps increases, the round reward value of the decision model DDPG-3C using three critic networks gradually increases and finally stabilizes at about 600, showing good learning effect and performance stability. After removing the highest value of the three Q values, the target Q value is calculated using the average of the remaining two. This effectively reduces the common problem of overestimation of Q values ​​in reinforcement learning.

[0167] The test was performed in 10,000 rounds, and the average area coverage and average collision rate of the model were calculated to evaluate its effectiveness and safety, as shown in Table 2.

[0168] Table 2 Average area coverage and collision rate of decision model DDPG-3C in the search phase

[0169]

[0170] The goal of the tracking phase is to keep tracking the target. The length and width of the two-dimensional SAR scene are both 800 meters. The scene includes a drone that can fully perceive the state of the environment and two stationary obstacles, each with a radius of 25 meters. The drone changes its speed through the thrust generated by the propellers in the x-axis and y-axis directions, with a maximum action value of 400N.

[0171] The training results of the DDPG-3C decision model in the SAR mission tracking phase in a denied environment are as follows: Figure 7 (a) and (b). To visualize the trend more clearly, the results are calculated using the exponential moving average (EMA).

[0172] The test was performed in 10,000 rounds, and the average area coverage and average collision rate of the model were calculated to evaluate its effectiveness and safety, as shown in Table 3.

[0173] Table 3 Average area coverage and collision rate of decision model DDPG-3C in the tracking phase

[0174]

[0175] Figure 8 The experimental results of the proposed two-stage target search and tracking method in the search and rescue (SAR) mission are shown, including the process of avoiding obstacles, searching for targets, and tracking targets. Figure 8 As can be seen in (a) and (d), the initial position of the drone is randomly generated and it searches for potential targets. Once the drone detects the target, Figure 8 As shown in (b) and (e), the drone switches to tracking mode. At this point, the drone adjusts its path to get closer to the target while avoiding obstacles. Figure 8 As shown in (c) and (f), the drone not only maintains its heading toward the target, but also adapts to changes in the target’s speed. Figure 8 (c) The drone shown can successfully avoid obstacles throughout the entire process, demonstrating the effectiveness of this method.

[0176] And in 10,000 rounds, the average number of steps required for the drone to find the target, the average number of steps required to successfully track the target, and the collision rate based on the decision model DDPG-3C of Example 1 are shown in Table 4, where the drone finding the target is defined as the situation where the drone sensor detects the target for the first time; successful target tracking is defined as the drone keeping the target within 20 meters.

[0177] Table 4 Average number of steps required for the drone to find the target, average number of steps required to successfully track the target, and collision rate based on the decision model DDPG-3C of Example 1

[0178]

[0179] In the search phase, the method of Example 1 takes an average of 171.27 steps to find the target, which is less than the number of steps required by the traditional single-stage method. This is due to the special optimization and training of the two-stage method in the search phase, which makes its search strategy more efficient. Once the drone's radar detects the target, the two-stage target search and tracking method takes an average of 354.54 steps to track the target in the tracking phase. The collision rate of the method of Example 1 is 4.07%, which is a substantial improvement over existing solutions.

[0180] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A two-stage target search and tracking method for UAV based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Divide the search and rescue mission into the search phase and the tracking phase; Step 2: Construct a deep deterministic policy gradient decision model DDPG-3C and train the decision model DDPG-3C accordingly according to the search phase and tracking phase; The deep deterministic policy gradient decision model DDPG-3C includes an Actor module and a Critic module; wherein the Actor module includes an Actor network and a target Actor network; the Critic module includes three Critic networks and three target Critic networks; Step 3: Load the trained corresponding deep deterministic policy gradient decision model DDPG-3C into the drone controllers in the search phase and tracking phase respectively; Step 4: Determine whether the detected target is within the radar detection range of the drone to switch between the search phase and the tracking phase.

2. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: In the process of interaction between the UAV and the environment, the UAV is regarded as an intelligent agent equipped with the decision model DDPG-3C. When the intelligent agent receives state information s from the environment i When the Actor network is based on the state information s i Output the corresponding action a i ; The agent then performs action a i ; After the action is performed, the environment will respond to it i ′ And return the corresponding reward r i to the agent; then the agent sends the state information s i 、Action a i , Responses i ′ , Reward i Stored in the experience replay buffer; the intelligent body will periodically randomly extract a small batch of data samples from the experience replay buffer to update the parameters of the Actor network and the Critic network.

3. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 2, characterized in that: The deep deterministic policy gradient decision model DDPG-3C adopts an experience replay buffer mechanism, which removes the maximum value of three estimated Q values ​​and takes the average of the remaining two estimated Q values ​​as the target estimated Q value.

4. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 2, characterized in that: The Actor network is responsible for converting the state of the environment into actions, and its goal is to learn a strategy to maximize the expected return in the future, namely: Where R1 represents the reward value obtained from state s0; ρ μ represents the state distribution under strategy μ; θμ is the parameter of the Actor network; J(θ μ ) represents the average reward value obtained from state s0 under strategy μ; represents mathematical expectation; The following update gradient formula is obtained through the policy gradient theorem: In the formula, represents the gradient of the strategy with respect to the Actor network parameter θμ; represents the gradient of the action-value function with respect to action a; N represents the batch size, that is, the amount of experience data sampled from the playback buffer during each training iteration; Q represents the output value of the Critic network; when calculating the loss of the Actor network, only the first Critic network is used to predict the Q value; Represents the parameters of the Critic network.

5. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 4, characterized in that: The parameters of the target Actor network are updated using a soft update strategy: θμ′=τθμ+(1-τ)θμ′; Where θμ′ represents the parameters of the target Actor network; τ is the soft update coefficient; θμ is the parameter of the Actor network.

6. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: The Q value is estimated independently by the three Critic networks and the three target Critic networks. The three Critic networks have the same structure, and each Critic network estimates the Q value according to the given state and action pair [state information s, action a]. The largest Q value is removed in each iteration, and the other two Q values ​​are averaged as the current target Q value; that is: In the formula, y t is the update target of the Critic network; r t represents the reward value obtained from the environment; γ represents the discount factor, which is used to calculate the present value of future rewards and helps determine the long-term optimization strategy; a ′ Indicates that the target Critic network is based on the next state s ′ Output action; Represents the lower two target Q values ​​output by the three-target Critic network.

7. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 6, characterized in that: The expression of the loss function of the Critic network is: In the formula, MSE represents the root mean square error, i=1,2,3 represents the Q value output by the corresponding Critic network; After the loss value is calculated by the loss function, the parameters of the Critic network are updated by the following update gradient formula: In the formula, Indicates the loss value Relative to the Critic network parameters The gradient of , N represents the batch size; The parameters of the target critic network are Update via soft update strategy: τ is the soft update coefficient.

8. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: In the search phase, the optimization goal of the drone controller is to make the drone cover the entire environment and quickly locate the target; If the target is detected by the drone’s radar, the drone will switch to tracking mode; During the tracking phase, the goal of the UAV controller is to continuously track the dynamic target.

9. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 1, characterized in that: In the search phase, the state space S of the drone’s reinforcement learning is: S=[drone x ,drone y ,drone vx ,drone vy ,obstacle dx ,obstacle dy ] If the obstacle is not within the drone's radar range, the obstacle's distance is marked as 1. In the formula, drone x Indicates the position of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1, 1]; drone y Indicates the position of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1, 1]; drone vx Indicates the speed of the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [-1, 1]; drone vy Indicates the speed of the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [-1, 1]; obstacle dx Indicates the distance of the obstacle detected by the drone in the x-axis direction of the two-dimensional plane, and the normalized range is [0, 1]; obstacle dy Indicates the distance of the obstacle detected by the drone in the y-axis direction of the two-dimensional plane, and the normalized range is [0, 1]; In the tracking phase, the state space of the UAV's reinforcement learning is: S=[drone x ,drone y ,drone vx ,drone vy , target dx ,target dy ,target vx ,target vy ,obstacle dx ,obstacle dy ] In the formula, target dx , target dy Respectively represent the distance of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane; target vx , target vy Respectively represent the speed of the target detected by the drone in the x-axis and y-axis directions of the two-dimensional plane; the normalized range is [0, 1]; If the target is not within the drone's radar range, the target's distance is marked as 1 and its speed is marked as 0.

10. The method for two-stage target search and tracking of unmanned aerial vehicles based on deep reinforcement learning according to claim 9, characterized in that: The reward function in the search scenario includes process rewards and terminal rewards. Each step will generate a process reward r step , to encourage the agent to quickly complete the coverage task. At the same time, when the agent covers a new area, it will receive a process reward r coverage ; If the drone covers 95% of the entire rescue area, a terminal reward r will be given complete ; If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion , in the search scenario the reward function R is as follows: R=∑r step +p∑r coverage +r complete +r collosion ; In the formula, p is the reward ratio; During the tracking phase, the drone receives a reward r for each step it takes to get close to the target. pro ; The reward r for approaching the target pro Expressed as the Euclidean distance between the drone and the target: In the formula, the 0.5 in the denominator is used to avoid the denominator becoming zero when the distance is very close or zero, ensuring the stability of the formula. If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion ; Therefore, the reward function of the tracking phase is as follows: R＝∑r pro +r collosion 。

Images