Unmanned aerial vehicle two-stage target searching and tracking method based on deep reinforcement learning

Abstract

The application provides a kind of unmanned aerial vehicle two-stage target search and tracking method based on deep reinforcement learning, including the search and rescue task is divided into search stage and tracking stage;Each stage is equipped with the corresponding decision model DDPG-3C, by judging whether the detection target is in the radar detection range of unmanned aerial vehicle, to switch the search stage and tracking stage.Each stage can optimize its control strategy and algorithm, expand the search range and improve the discovery speed in the search stage, and focus on improving the tracking accuracy and response speed of dynamic targets in the tracking stage, improve the efficiency of each stage;DDPG-3C model does not depend on accurate mathematical model to make decisions, so that the application can show stronger adaptability when facing complex and dynamic environment, learn and optimize strategy through interaction with environment, instead of relying on pre-defined model parameters, so as to better cope with unknown or changing conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle rescue, and in particular to a two-stage target search and tracking method for unmanned aerial vehicles based on deep reinforcement learning. BACKGROUND

[0002] With the continuous progress of technology, unmanned aerial vehicles have made significant improvements in intelligence, endurance, payload capacity, and anti-interference, making them suitable for a wider range of application scenarios.

[0003] However, in disaster rescue missions such as earthquakes, floods, or mudslides, the disaster area often has complex terrain and communication disruptions. Unmanned aerial vehicles may face problems such as infrastructure damage, harsh natural environments, and human interference, making them in a communication denial environment. At this time, the perception ability of the unmanned aerial vehicle may be severely affected, making it difficult for the unmanned aerial vehicle to find trapped personnel or important clues, reducing the success rate of search and rescue. Secondly, in a communication denial environment, the unmanned aerial vehicle may not be able to receive real-time ground control instructions, which poses a great challenge to the autonomous control decision-making ability of the unmanned aerial vehicle.

[0004] In recent years, some advanced technologies such as model predictive control, optimization-based methods, and intelligent optimization algorithms have been successfully applied to the decision-making process of unmanned aerial vehicles. These methods provide unmanned aerial vehicles with the ability to efficiently navigate and perform tasks in complex environments. However, these methods have obvious limitations. On the one hand, many methods rely on precise mathematical models, but these models are difficult to completely adapt to actual, complex and variable environments. On the other hand, these methods rely heavily on sensor information, and the accuracy and reliability of sensors are often disturbed by environmental factors. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a two-stage target search and tracking method for unmanned aerial vehicles based on deep reinforcement learning. The present application divides the search and rescue task into a search phase and a tracking phase to improve the ability of unmanned aerial vehicles to find and track targets in post-disaster rescue environments.

[0006] The technical solution of the present application is a two-stage target search and tracking method for unmanned aerial vehicles based on deep reinforcement learning, comprising the following steps:

[0007] Step 1: dividing the search and rescue task into a search phase and a tracking phase;

[0008] Step 2: constructing a deep deterministic policy gradient decision model DDPG-3C and training the decision model DDPG-3C according to the search phase and the tracking phase;

[0009] Step 3: load the trained corresponding deep deterministic policy gradient decision model DDPG-3C into the drone controller of the search phase and the tracking phase respectively;

[0010] Step 4: switch the search phase and the tracking phase by judging whether the detection target is within the radar detection range of the drone.

[0011] As preferred, the deep deterministic policy gradient decision model DDPG-3C comprises an Actor module and a Critic module; wherein the Actor module comprises an Actor network and a target Actor network; the Critic module comprises three Critic networks and three target Critic networks; in the interaction process of the drone and the environment, the drone is regarded as an agent carrying the decision model DDPG-3C, when the agent receives state information s i from the environment, the Actor network outputs a corresponding action a i according to the state information s i ; the agent then executes the action a i ; after executing the action, the environment responds s i ' and returns a corresponding reward r i to the agent; then the agent stores the state information s i , the action a i , the response s i ', and the reward r i in the experience replay buffer; the agent regularly extracts a small batch of data samples from the experience replay buffer to update the parameters of the Actor network and the Critic network.

[0012] As preferred, the deep deterministic policy gradient decision model DDPG-3C adopts an experience replay buffer mechanism, and removes the maximum value of the three estimated Q values, and takes the average value of the remaining two estimated Q values as the target estimated Q value.

[0013] As preferred, the Actor network is responsible for converting the environment state into an action, and its goal is to learn a policy to maximize the expected return in the future, that is:

[0014]

[0015] wherein R1 represents the reward value obtained from state s0; τ μ represents the state distribution under the policy μ; θ μ is the parameter of the Actor network; J(θ μ ) represents the average reward value obtained from state s0 under the policy μ; represents mathematical expectation;

[0016] The following update gradient formula is obtained by the policy gradient theorem:

[0017]

[0018] wherein, represents the gradient of the policy with respect to the Actor network parameter θ μ ;

[0019] represents the gradient of the action value function with respect to the action a; N represents the batch size, i.e. the amount of experience data sampled from the replay buffer during each training iteration; only the first Critic network is used to predict the Q value when calculating the loss of the Actor network; represents the parameter of the Critic network, and Q represents the output value of the Critic network.

[0020] Preferably, the parameters of the target Actor network are updated by a soft update policy:

[0021] θ μ′ = τθ μ + (1-τ)θ μ′ ;

[0022] wherein, θ μ′ represents the parameter of the target Actor network; τ is a soft update coefficient; and θ μ represents the parameter of the Actor network.

[0023] Preferably, the Q value is independently estimated by three Critic networks and three target Critic networks, the three Critic networks have the same structure, and each Critic network estimates the Q value according to a given state and action pair (state information s, action a); the maximum Q value is removed each time, and the other two Q values are averaged as the current target Q value; that is:

[0024]

[0025] wherein, y t is the update target of the Critic network; r t represents the reward value obtained from the environment; γ represents a discount factor for calculating the present value of future rewards, which helps to determine the long-term optimization strategy; and a' represents the action output by the target Critic network according to the next state s'; represents the lower two target Q values output by the three-target Critic network.

[0026] Preferably, the loss function of the Critic network has the following expression:

[0027]

[0028] where MSE denotes the mean square error, Q denotes the Q value output by the corresponding Critic network;

[0029] After the loss value is calculated by the loss function, the parameters of the Critic network are updated by the following update gradient formula

[0030]

[0031] where denotes the loss value the gradient of the Critic network parameters ; N denotes the batch size;

[0032] The parameters of the target Critic network are updated by the soft update strategy:

[0033]

[0034] τ is the soft update coefficient.

[0035] As preferred, the Actor network and the Critic network each comprise a corresponding input layer, a first Dense layer, a second Dense layer, a third Dense layer, and an output layer.

[0036] As preferred, in the search phase, the optimization target of the UAV controller is to make the UAV cover the entire environment and quickly locate the target; if the target is detected by the radar of the UAV, the UAV will switch to the tracking mode; in the tracking phase, the target of the UAV controller is to continuously track the dynamic target.

[0037] As preferred, in the search phase, the state space of the reinforcement learning of the UAV is:

[0038] S = [drone x , drone y , drone vx , drone vy , obstacle dx , obstacle dy ]

[0039] If the obstacle is not within the radar range of the UAV, the distance of the obstacle is marked as 1,

[0040] where drone xrepresents the position of the drone in the x-axis direction of the two-dimensional plane, normalized to the range [-1, 1];

[0041] drone y represents the position of the drone in the y-axis direction of the two-dimensional plane, normalized to the range [-1, 1];

[0042] drone vx represents the velocity of the drone in the x-axis direction of the two-dimensional plane, normalized to the range [-1, 1];

[0043] drone vy represents the velocity of the drone in the y-axis direction of the two-dimensional plane, normalized to the range [-1, 1];

[0044] obstacle dx represents the distance of the obstacle detected by the drone in the x-axis direction of the two-dimensional plane, normalized to the range [0, 1];

[0045] obstacle dy represents the distance of the obstacle detected by the drone in the y-axis direction of the two-dimensional plane, normalized to the range [0, 1].

[0046] As a preferred, the reward function in the search scene includes process reward and terminal reward, a process reward r step is generated at each step to encourage the agent to quickly complete the coverage task, and at the same time, when the agent covers a new area, a process reward r coverage is given; if the range covered by the drone (agent) reaches 95% of the entire rescue area, a terminal reward r complete is given; if the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion In the search scene, the reward function R is as follows:

[0047] R = ∑r step + p∑r coverage + r complete + r collosion ;

[0048] where p is the reward ratio;

[0049] As a preferred, in the tracking phase, the state space of the reinforcement learning of the drone is:

[0050] S = [drone x , drone y , drone vx , drone vy , target dx , target dytarget vx target vy obstacle dx obstacle dy ]

[0051] target dx target dy target vx target vy target pro target pro target collosion

[0052] As preferred, in the tracking phase, each step of the UAV receives a reward r pro for approaching the target; the reward r pro for approaching the target is expressed by the Euclidean distance between the UAV and the target:

[0053]

[0054] In the formula, the 0.5 in the denominator is used to avoid the denominator becoming zero when the distance is very close or zero, ensuring the stability of the formula; if the UAV collides with the obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion ; therefore, the reward function of the tracking phase is as follows:

[0055] R = ∑ r pro + r collosion .

[0056] The beneficial effects of the present application are:

[0057] 1. The present application divides the search and rescue task into two phases of target search and target tracking, and each phase can optimize its control strategy and algorithm, the phased method allows to expand the search range and improve the discovery speed in the search phase, and focuses on improving the tracking accuracy and response speed of the dynamic target in the tracking phase, not only improves the efficiency of each phase, but also makes the whole system more flexible, and can adjust the strategy according to different operation requirements;

[0058] 2. The present application optimizes the reward design, reduces the dependence of the complex reward function, simplifies the training process of the model, and reduces the computational complexity;

[0059] 3、The DDPG-3C model of the application does not rely on an accurate mathematical model for decision-making, so that the application can exhibit stronger adaptability when facing complex and dynamic environments, and can learn and optimize strategies through interaction with the environment, rather than relying on pre-defined model parameters, thereby better coping with unknown or changing conditions;

[0060] 4、The DDPG-3C of the application significantly reduces the overestimation problem by introducing three Critic networks and removing the maximum value in the three Q values, using the average value of the remaining two as the target Q value, improves the accuracy of value estimation, thereby making the policy learning process more stable, speeding up the convergence speed, and improving the reliability of decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The structural framework diagram of the deep deterministic policy gradient decision-making model DDPG-3C of the embodiment 1 of the application is shown in the figure;

[0062] Figure 2 The method flowchart of the two-stage target search and tracking of the embodiment 1 of the application is shown in the figure;

[0063] Figure 3 The structural diagram of the Actor network and the Critic network of the embodiment 1 of the application is shown in the figure;

[0064] Figure 4 The implementation process schematic diagram of the decision-making model DDPG-3C used by the unmanned aerial vehicle in the search stage of the embodiment 1 of the application is shown in the figure;

[0065] Figure 5 The schematic diagram of the two-dimensional search and rescue environment of the embodiment 2 of the application is shown in the figure;

[0066] Figure 6 The training results of the search stage DDPG-3C model of the embodiment 2 of the application are shown in the figure: (a) is a round reward value curve; (b) is a target Q value curve;

[0067] Figure 7 The training results of the tracking stage DDPG-3C model of the embodiment 2 of the application are shown in the figure: (a) is a round reward value curve; (b) is a target Q value curve;

[0068] Figure 8 The experimental result schematic diagram of the embodiment 2 of the application in the search and rescue (SAR) task using the method of the embodiment 1 is shown in the figure. DETAILED DESCRIPTION

[0069] The specific embodiments of the application will be further described below in conjunction with the accompanying drawings:

[0070] Embodiment 1

[0071] As Figure 1 shown in the figure, the embodiment provides a two-stage target searching and tracking method for unmanned aerial vehicle based on deep reinforcement learning,

[0072] Step 1: dividing the search and rescue task into a searching stage and a tracking stage;

[0073] Step 2: constructing a deep deterministic policy gradient decision model DDPG-3C and training the decision model DDPG-3C according to the searching stage and the tracking stage;

[0074] Step 3: loading the trained deep deterministic policy gradient decision model DDPG-3C into the unmanned aerial vehicle controller of the searching stage and the tracking stage respectively;

[0075] Step 4: switching the searching stage and the tracking stage by judging whether the detected target is within the radar detection range of the unmanned aerial vehicle.

[0076] As preferred in the embodiment, the deep deterministic policy gradient decision model DDPG-3C comprises an Actor module and a Critic module; wherein the Actor module comprises an Actor network and a target Actor network; the Critic module comprises three Critic networks and three target Critic networks; in the interaction process between the unmanned aerial vehicle and the environment, the unmanned aerial vehicle is regarded as an intelligent agent carrying the decision model DDPG-3C, when the intelligent agent receives state information s i from the environment, the Actor network outputs a corresponding action a i according to the state information s i ; the intelligent agent then executes the action a i ; after executing the action, the environment responds s i ′ and returns a corresponding reward r i to the intelligent agent; then the intelligent agent stores the state information s i , the action a i , the response s i ′ and the reward r i in an experience replay buffer; the intelligent agent regularly updates the parameters of the Actor network and the Critic network by randomly extracting a small batch of data samples from the experience replay buffer.

[0077] As preferred in the embodiment, the deep deterministic policy gradient decision model DDPG-3C adopts an experience replay buffer mechanism, and removes the maximum value among the three estimated Q values, taking the average value of the remaining two estimated Q values as the target estimated Q value.

[0078] As preferred in the embodiment, the Actor network is responsible for converting the environment state into an action, and aims to learn a policy to maximize the expected return in the future, i.e.

[0079]

[0080] wherein R1represents a reward value obtained from the state s0; p μ represents a state distribution under the policy μ; θ μ is a parameter of the Actor network; J(θ μ ) represents an average reward value obtained from the state s0under the policy μ; represents a mathematical expectation;

[0081] The following update gradient formula is obtained by the policy gradient theorem:

[0082]

[0083] wherein represents a gradient of the policy with respect to the Actor network parameter θ μ ; Q represents a value of the action-value function.

[0084] represents a gradient of the action-value function with respect to the action a; N represents a batch size, i.e. an amount of experience data sampled from the replay buffer during each training iteration; only the first Critic network is used to predict the Q value when calculating the loss of the Actor network; represents a parameter of the Critic network, and Q represents an output value of the Critic network.

[0085] As preferred in the embodiment, the parameters of the target Actor network are updated by a soft update policy:

[0086] θ μ′ = τθ μ + (1-τ)θ μ′ ;

[0087] wherein θ μ′ represents a parameter of the target Actor network; τ is a soft update coefficient; and θ μ is a parameter of the Actor network.

[0088] As preferred in the embodiment, the Q value is independently estimated by three Critic networks and three target Critic networks, the three Critic networks are of the same structure, and each Critic network estimates the Q value according to a given state and action pair (state information s, action s); the maximum Q value is removed each time, and the other two Q values are averaged as the current target Q value; i.e.

[0089]

[0090] wherein y t is the update target of the Critic network; r t represents the reward value obtained from the environment; γ represents a discount factor for calculating the present value of future rewards, which helps to determine the long-term optimization strategy; a' represents the action output by the target Critic network according to the next state s'; represents the lower two target Q values output by the three-target Critic network.

[0091] As preferred in the present embodiment, the expression of the loss function of the Critic network is as follows:

[0092]

[0093] wherein MSE represents the root mean square error, represents the Q value output by the corresponding Critic network;

[0094] After the loss value is calculated by the loss function, the parameters of the Critic network are updated by the following update gradient formula

[0095]

[0096] wherein represents the loss value relative to the Critic network parameters ; N represents the batch size;

[0097] The parameters of the target Critic network are updated by the following formula:

[0098]

[0099] τ is the soft update coefficient.

[0100] As preferred in the present embodiment, the Actor network and the Critic network each include a corresponding input layer, a first Dense layer, a second Dense layer, a third Dense layer, and an output layer, as shown in Figure 3 wherein s i represents the state of the UAV, a i represents the action, Q represents the value output by the Critic network, input_dim represents the size of the network input layer dimension, fc1_dim and fc2_dim respectively represent the size of the network hidden layer dimension, and output_dim represents the size of the network output layer dimension. ​

[0101] As preferred in the embodiment, in the search phase, the training of the deep deterministic policy gradient decision model DDPG-3C includes the following steps:

[0102] S211), initialize the parameters of the Actor network, the target Actor network, the Critic network and the target Critic network;

[0103] S212), obtain the state observation of the UAV from the search scene;

[0104] S213), input the state observation into the Actor network, and the Actor network outputs the action a t ;

[0105] S214), the UAV executes the action a t , obtains the reward value r t and the next state observation s t+1 of the UAV from the search scene;

[0106] S215), store the experience value (s t , a t , r t , s t+1 ) in the experience replay buffer;

[0107] S216), sample a small batch of data from the experience replay buffer;

[0108] S217), update the parameters of the Actor network, the target Actor network, the Critic network and the target Critic network according to the gradient formula and the soft update strategy;

[0109] S218), repeat steps S213)-S217) until the round in the search scene is over;

[0110] S219), reset the search scene, repeat steps S212)-S218) until the set number of training is over; save the structure and parameters of the Actor network.

[0111] As preferred in the embodiment, in the tracking phase, the training of the deep deterministic policy gradient decision model DDPG-3C includes the following steps:

[0112] S221), initialize the parameters of the Actor network, the target Actor network, the Critic network and the target Critic network;

[0113] S222), obtain the state observation of the UAV from the tracking scene;

[0114] S223), inputting the state observation into the Actor network, and the Actor network outputs an action a t ;

[0115] S224), the UAV performs the action a t , obtaining a reward value r from the search scenario t and a next state observation s of the UAV t+1 ;

[0116] S225), storing the experience value (s t ,a t ,r t ,s t+1 ) in the experience replay buffer;

[0117] S226), sampling a small batch of data from the experience replay buffer;

[0118] S227), updating the parameters of the Actor network, the target Actor network, the Critic network and the target Critic network according to the gradient formula and the soft update strategy;

[0119] S228), repeating steps S223)-S227) until the round in the search scenario ends;

[0120] S229), resetting the tracking scenario, repeating steps S222)-S228) until the set number of training times ends; saving the structure and parameters of the Actor network.

[0121] As preferred in the embodiment, in the search phase, the optimization goal of the UAV controller is to cover the entire environment and quickly locate the target; if the target is detected by the radar of the UAV, the UAV will switch to the tracking mode; in the tracking phase, the goal of the UAV controller is to continuously track the dynamic target.

[0122] As preferred in the embodiment, in the search phase, the state space S of the reinforcement learning of the UAV is:

[0123] S=[drone x ,drone y ,drone vx ,drone vy ,obstacle dx ,obstacle dy ]

[0124] If the obstacle is not within the radar range of the UAV, the distance of the obstacle is marked as 1,

[0125] In the formula, drone xrepresents the position of the UAV in the x-axis direction of the two-dimensional plane, and the normalized range is [-1, 1];

[0126] drone y represents the position of the UAV in the y-axis direction of the two-dimensional plane, and the normalized range is [-1, 1];

[0127] drone vx represents the velocity of the UAV in the x-axis direction of the two-dimensional plane, and the normalized range is [-1, 1];

[0128] drone vy represents the velocity of the UAV in the y-axis direction of the two-dimensional plane, and the normalized range is [-1, 1];

[0129] abstacle dx represents the distance of the obstacle detected by the UAV in the x-axis direction of the two-dimensional plane, and the normalized range is [0, 1];

[0130] obstacle dy represents the distance of the obstacle detected by the UAV in the y-axis direction of the two-dimensional plane, and the normalized range is [0, 1].

[0131] The action space of the UAV is a two-dimensional vector [F x ,F y ]; F y and F max are the thrusts generated by the propellers in the x-axis and y-axis directions, respectively; considering the maneuverability of the UAV, the action value is not greater than the maximum thrust F step .

[0132] As preferred in this embodiment, the reward function in the search scenario includes a process reward and a terminal reward, a process reward r coverage =-0.1 is generated at each step to encourage the agent to quickly complete the coverage task, and at the same time, a process reward r complete is obtained when the agent covers a new area; if the UAV (agent) covers 95% of the entire rescue area, a terminal reward r

[0133] r collosion =-200 will be given to the agent if the UAV collides with an obstacle or leaves the rescue area, and the reward function R in the search scenario is as follows:

[0134] R = ∑r step + p∑r coverage + r complete + r collosion ;

[0135] where p is the reward ratio, and in this embodiment, p = 0.001.

[0136] As preferred in this embodiment, in the tracking phase, the state space of the reinforcement learning of the UAV is:

[0137] S = [drone x , drone y , drone vx , drone vy , target dx , target dy , target vx , target vy , obstacle dx , obstacle dy ]

[0138] where target dx and target dy respectively represent the distance of the target detected by the UAV in the x-axis and y-axis directions of the two-dimensional plane; target vx and target vy respectively represent the speed of the target detected by the UAV in the x-axis and y-axis directions of the two-dimensional plane; after normalization, the range is [0, 1]; if the target is not within the radar range of the UAV, the distance of the target is marked as 1, and the speed is marked as 0.

[0139] The action space of the UAV is a two-dimensional vector [F x , F y ]; F y is the thrust generated by the propeller in the x-axis direction and the y-axis direction, respectively; considering the maneuvering performance of the UAV, the action value is not greater than the maximum thrust F max generated by the propeller.

[0140] As preferred in this embodiment, in the tracking phase, the UAV receives a reward r pro for approaching the target at each step; the reward r pro for approaching the target is expressed by the Euclidean distance between the UAV and the target:

[0141]

[0142] where the denominator 0.5 is used to avoid the denominator becoming zero when the distance is very close or zero, ensuring the stability of the formula; if the UAV collides with the obstacle or leaves the rescue area, the agent will receive a negative terminal reward r collosion ; therefore, the reward function in the tracking phase is as follows:

[0143] R =∑r pro +r collosion .

[0144] As preferred in the present embodiment, in step 4, the switching between the search phase and the tracking phase is performed by determining whether the detection target is within the radar detection range of the UAV, as shown in Figure 2 and 4 , specifically comprising the following steps:

[0145] S41), the UAV controller loads the structure and parameters of the Actor network saved in the search phase and the tracking phase;

[0146] S42), the state observation of the UAV is obtained from the search and rescue environment;

[0147] S43), determine whether the target is within the radar range of the UAV, if the UAV radar does not detect the target, go to step S44), if the UAV radar detects the target, go to step S46);

[0148] S44), input the state observation of the UAV into the decision model DDPG-3C of the search phase;

[0149] S45), the decision model DDPG-3C of the search phase outputs the action to the UAV for execution, and goes to step S42);

[0150] S46), input the state observation of the UAV into the decision model DDPG-3C of the tracking phase;

[0151] S47), the decision model DDPG-3C of the tracking phase outputs the action to the UAV for execution, and goes to step S42);

[0152] S48), repeat S42)-S43) until the end of the round.

[0153] Embodiment 2

[0154] In order to verify the effectiveness of the method and model proposed in Embodiment 1, a two-dimensional simulation search and rescue environment is first established. As shown in Figure 5 , a simplified two-dimensional search and rescue (SAR) environment; the scene is represented by a two-dimensional rectangular area with a length of L and a width of W. Among them, the dark blue UAV icon represents the UAV (agent); the blue circular icon represents the target waiting for rescue; the dark green circular area represents the obstacle or dangerous area, with a radius of r. In this scene, the UAV is equipped with a radar device that can sense the speed and distance of approaching objects, and its sensing range is represented by a red circle.

[0155] In this two-dimensional simulation search and rescue environment, the initial position and velocity of the UAV and the search target are random. The search target maintains a constant linear motion and assumes that the collision between the target and the obstacle is a completely elastic collision, and continues to maintain a constant linear motion after the collision. The motion of the UAV changes the speed through the thrust generated by the x-axis and y-axis propellers, and its motion model is:

[0156]

[0157] where F x and F y are the thrusts generated by the x-axis and y-axis propellers, respectively; their size ranges are F x and F y ∈ [-Fmax max ; v x and v y are the velocities of the UAV in the x-axis and y-axis directions, respectively; their size ranges are v x and v y ∈ [-vmax max ; Δt represents the sampling interval; and m represents the mass of the UAV.

[0158] When training the decision model DDPG-3C, the hyperparameters of the decision model DDPG-3C are shown in Table 1.

[0159] Table 1 Hyperparameters of the decision model DDPG-3C

[0160]

[0161]

[0162] In the search phase, there is no target in the search scene when the decision model DDPG-3C is trained. The goal of the UAV is to quickly and effectively cover the entire two-dimensional rescue area. The length and width of the two-dimensional SAR scene are both 800 meters, including a UAV with a sensor range of 200 meters and two obstacles with a radius of 25 meters. The obstacles are stationary, while the UAV changes speed through the thrust generated by the x-axis and y-axis propellers, with a maximum action value of 400 N. The decision model DDPG-3C is trained under the same conditions using the hyperparameters in Table 1.

[0163] The training results of the DDPG-3C decision model in the search phase of the search and rescue (SAR) task in the rejection environment are shown in FIGS. Figure 6 (a) and (b). To more clearly visualize the trend, the results are calculated using the exponential moving average (EMA):

[0164]

[0165] where s = 0.95 is the smoothing factor.

[0166] With the increase of training steps, the episode reward value of the decision model DDPG-3C with three Critic networks gradually rises and eventually stabilizes at a level of about 600, showing good learning effect and performance stability. After removing the highest value among the three Q values, the average value of the remaining two is used to calculate the target Q value. This effectively reduces the common Q value overestimation problem in reinforcement learning.

[0167] The model was tested for 10,000 episodes, and the average area coverage rate and average collision rate were calculated to evaluate its effectiveness and safety, as shown in Table 2.

[0168] Table 2 Average area coverage rate and collision rate of decision model DDPG-3C in the search phase

[0169]

[0170] The goal of the tracking phase is to maintain continuous tracking of the target. The two-dimensional SAR scenario has a length and width of 800 meters. The scenario includes a drone that can fully perceive the environment state, and two stationary obstacles, each with a radius of 25 meters. The drone changes its speed by the thrust generated by the x-axis and y-axis propellers, with a maximum action value of 400 N.

[0171] The training results of the DDPG-3C decision model in the tracking phase of the SAR task in the rejection environment are shown in Figure 7 (a) and (b). To more clearly visualize the trend, the results were calculated using the exponential moving average (EMA).

[0172] The model was tested for 10,000 episodes, and the average area coverage rate and average collision rate were calculated to evaluate its effectiveness and safety, as shown in Table 3.

[0173] Table 3 Average area coverage rate and collision rate of decision model DDPG-3C in the tracking phase

[0174]

[0175] Figure 8 The experimental results of the proposed two-stage target search and tracking method in the search and rescue (SAR) task are shown, including the process of avoiding obstacles, searching for targets, and tracking targets. From Figure 8 (a) and (d), it can be seen that the initial position of the drone is randomly generated and looks for potential targets. Once the drone detects a target, as shown in Figure 8 (b) and (e), the drone switches to tracking mode. At this time, the drone adjusts its path to approach the target while avoiding obstacles. In the tracking phase, as shown inFigure 8 (c) and (f), the UAV not only maintains the heading towards the target, but also adapts to the change of target speed. In addition, as ​ (c) shows that the UAV successfully avoids obstacles throughout the process, showing the high efficiency of the method.

[0176] And in 10000 rounds, the average number of steps required for the UAV based on the decision-making model DDPG-3C of embodiment 1 to find the target, the average number of steps required for the UAV to successfully track the target, and the collision rate are shown in Table 4, wherein the UAV finding the target is defined as the case where the UAV sensor first detects the target; successfully tracking the target is defined as the UAV keeping the target within a distance of 20 meters.

[0177] Table 4 Average number of steps required for the UAV based on the decision-making model DDPG-3C of embodiment 1 to find the target, the average number of steps required for the UAV to successfully track the target, and the collision rate

[0178]

[0179] In the search phase, the method of embodiment 1 requires an average of 171.27 steps to find the target, which is less than the number of steps required by the traditional single-stage method, which is due to the special optimization and training of the two-stage method in the search phase, making its search strategy more efficient. Once the radar of the UAV detects the target, the two-stage target search and tracking method requires an average of 354.54 steps to track the target in the tracking phase, and the collision rate of the method of embodiment 1 is 4.07%, which is a substantial improvement over existing solutions.

[0180] The above embodiments and descriptions described in the specification are only to illustrate the principles and best embodiments of the present application, and various changes and improvements can be made to the present application without departing from the spirit and scope of the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A two-stage target search and tracking method for unmanned aerial vehicles (UAVs) based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Divide the search and rescue mission into a search phase and a tracking phase; Step 2: Construct the deep deterministic policy gradient decision model DDPG-3C, and train the decision model DDPG-3C accordingly based on the search phase and the tracking phase; The deep deterministic policy gradient decision model DDPG-3C includes an Actor module and a Critic module; wherein, the Actor module includes an Actor network and a target Actor network; the Critic module includes three Critic networks and three target Critic networks; the expression for the loss function of the Critic network is: In the formula, MSE represents the root mean square error. i = 1, 2, 3 represents the Q value of the corresponding Critic network output; After calculating the loss value using the loss function, the parameters of the Critic network are updated using the following gradient update formula. In the formula, Indicates the loss value Relative to Critic network parameters The gradient, where N represents the batch size; The parameters of the target Critic network Update using a soft update strategy: τ is the soft update coefficient; Step 3: Load the trained deep deterministic policy gradient decision model DDPG-3C into the UAV controllers for the search and tracking phases respectively; Step 4: By determining whether the target is within the radar detection range of the UAV, the search phase and the tracking phase are switched.

2. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 1, characterized in that: During the interaction between the drone and its environment, the drone is considered an intelligent agent equipped with the DDPG-3C decision-making model. When the intelligent agent receives state information s from the environment... i At that time, the Actor network, based on the state information s i Output the corresponding action a i The agent then executes action a. i After the action is performed, the environment will respond accordingly. i And return the corresponding reward r i The agent is given the state information s; the agent then sends the state information s to the agent. i Action a i Response s i ′、Reward r i The data is stored in the experience replay buffer; the AI ​​periodically draws a small batch of data samples randomly from the experience replay buffer to update the parameters of the Actor network and the Critic network.

3. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 2, characterized in that: The deep deterministic policy gradient decision model DDPG-3C employs an empirical replay buffer mechanism, which removes the maximum value among the three estimated Q values ​​and uses the average of the remaining two estimated Q values ​​as the target estimated Q value.

4. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 2, characterized in that: The Actor network is responsible for transforming environmental states into actions, and its goal is to learn a policy to maximize future expected rewards, i.e.: In the formula, R1 represents the reward value obtained starting from state s0; ρ μ θ represents the state distribution under policy μ; μ The parameters of the Actor network; J(θ) μ ) represents the average reward value obtained starting from state s0 under policy μ; Represents the mathematical expectation; The following update gradient formula is obtained by using the policy gradient theorem: In the formula, The policy is represented relative to the Actor network parameters θ. μ The gradient; represents the gradient of the action-value function with respect to action a; N represents the batch size, i.e., the amount of empirical data sampled from the replay buffer during each training iteration; Q represents the output value of the Critic network; when calculating the loss of the Actor network, only the first Critic network is used to predict the Q value; This represents the parameters of the Critic network.

5. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 4, characterized in that: The parameters of the target Actor network are updated using a soft update strategy: i μ′ =tθ μ +(1-τ)θ μ′ ; In the formula, θ μ′ τ represents the parameters of the target Actor network; τ is the soft update coefficient; θ μ These are the parameters of the Actor network.

6. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 1, characterized in that: The Q-value is independently estimated using three Critic networks and three target Critic networks. The three Critic networks have identical structures, and each Critic network estimates the Q-value based on a given state and action pair [state information s, action a]. In each iteration, the largest Q-value is removed, and the average of the remaining two Q-values ​​is taken as the current target Q-value. That is: In the formula, y t It is the update target of the Critic network; r t γ represents the reward value obtained from the environment; γ represents the discount factor, used to calculate the present value of future rewards, which helps to determine long-term optimization strategies; a′ represents the action output by the target Critic network based on the next state s′. This indicates the lower Q-values ​​of the two targets output by the three-target Critic network.

7. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 1, characterized in that: During the search phase, the optimization goal of the drone controller is to enable the drone to cover the entire environment and quickly locate the target. If the target is detected by the drone's radar, the drone will switch to tracking mode; During the tracking phase, the goal of the drone controller is to continuously track dynamic targets.

8. The two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 1, characterized in that: During the search phase, the state space S of the drone's reinforcement learning is: S=[drone x ,drone y ,drone vx ,drone vy ,obstacle dx ,obstacle dy ] If the obstacle is not within the drone's radar range, the obstacle's distance is marked as 1. In the formula, drone x This represents the position of the UAV along the x-axis in a two-dimensional plane, with a normalized range of [-1, 1]. drone y This represents the position of the UAV in the y-axis direction of the two-dimensional plane, with a normalized range of [-1, 1]. drone vx This represents the velocity of the UAV along the x-axis in a two-dimensional plane, with a normalized range of [-1, 1]. drone vy This represents the velocity of the UAV in the y-axis direction of the two-dimensional plane, with a normalized range of [-1, 1]. obstacle dx This represents the distance of the obstacle detected by the drone along the x-axis in the two-dimensional plane, with a normalized range of [0,1]. obstacle dy This represents the distance of the obstacle detected by the drone along the y-axis in the two-dimensional plane, with a normalized range of [0,1]. During the tracking phase, the state space of the UAV's reinforcement learning is as follows: S=[drone x ,drone y ,drone vx ,drone vy ,target dx ,target dy ,target vx ,target vy ,obstacle dx ,obstacle dy ] In the formula, target dx ,target dy These represent the distances of the target detected by the drone along the x-axis and y-axis in the two-dimensional plane, respectively; target vx ,target vy These represent the velocities of the target detected by the UAV in the x-axis and y-axis directions of the two-dimensional plane, respectively; their normalized range is [0,1]. If the target is not within the drone's radar range, the target's distance is marked as 1 and its speed as 0.

9. A two-stage target search and tracking method for UAVs based on deep reinforcement learning according to claim 8, characterized in that: The reward function in a search scenario includes process reward and terminal reward, with a process reward r generated at each step. step This encourages agents to quickly complete coverage tasks, and they receive a process reward r when they cover new areas. coverage If the drone's coverage reaches 95% of the entire rescue area, a terminal reward will be given. complete If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward. collosion The reward function R in the search scenario is as follows: R=∑r step +p∑r coverage +r complete +r collosion ; In the formula, p is the reward ratio; During the tracking phase, the drone receives a reward r for each step it takes to approach the target. pro The reward r for approaching the target pro Expressed using the Euclidean distance between the drone and the target: In the formula, 0.5 in the denominator is used to prevent the denominator from becoming zero when the distance is very close or zero, ensuring the stability of the formula. If the drone collides with an obstacle or leaves the rescue area, the agent will receive a negative terminal reward r. collosion Therefore, the reward function for the tracking phase is as follows: R＝∑r pro +r collosion 。

Images