Target tracking method, device, drone and storage medium
Through the deep reinforcement learning model, the flight action parameters are calculated in real time and the drone cluster is controlled for intelligent tracking, which solves the problem of low success rate in dynamic unknown target tracking, and improves the tracking effect.
Patent Information
- Application Number
- CN202110822787.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-07-21
AI Technical Summary
The traditional goal tracking method has a low success rate in dynamic unknown target tracking tasks and cannot effectively adapt to the changes in the target motion trajectory.
The deep reinforcement learning model is adopted to obtain the flight status data and target position information of the drone cluster, calculate the flight action parameters in real time, and control the drone cluster for intelligent tracking.
The success rate of target tracking is improved, and through deep reinforcement learning environmental adaptability, the drone can flexibly adapt to environmental changes and achieve effective tracking of dynamic unknown goals.
Smart Images

Figure CN113554680B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target tracking technology, and in particular to a target tracking method, device, drone and storage medium. Background Art
[0002] In recent years, drones have achieved great success in military and civilian applications, such as data collection in smart cities and regional monitoring. Due to their flexibility and low cost, drones are favored in performing complex tasks in open environments. Target tracking is a typical example of the aforementioned tasks. Dynamic unknown target tracking, where the target position and motion state change over time and the target's motion trajectory cannot be known in advance by the tracker, is one of the recent research hotspots in target tracking.
[0003] Traditional target tracking methods often assume that the target's motion trajectory can be predicted in advance, and that it is only necessary to control the drone to fly along the target trajectory. However, this method is only applicable to tracking problems where the target is known.
[0004] In the dynamic unknown target tracking task, the success rate of target tracking is low using traditional techniques. Summary of the invention
[0005] Based on this, it is necessary to provide a target tracking method, device, drone and storage medium that can improve the target tracking success rate in response to the above technical problems.
[0006] A target tracking method, the method comprising:
[0007] Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0008] Inputting the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0009] The flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target.
[0010] In one embodiment, the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step are input into a trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step, including:
[0011] Input the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step;
[0012] The Q value corresponding to each candidate flight action parameter of each UAV at the current time step is input into the policy network in the trained deep reinforcement learning model to determine the flight action parameter of each UAV at the current time step.
[0013] In one embodiment, the method further comprises:
[0014] Obtaining a state reward corresponding to the flight state data of each UAV at the current time step according to the Q value corresponding to the flight action parameter of each UAV at the current time step;
[0015] Acquire flight state data of the next time step obtained by each UAV flying according to the flight action parameters;
[0016] According to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step and the flight state data of each UAV at the next time step, the trained deep reinforcement learning model is updated to obtain an updated deep reinforcement learning model.
[0017] In one embodiment, the flight status data includes location information; the method further includes:
[0018] Obtaining the position information of the tracking target at the next time step;
[0019] Determining whether the current strategy executed in the trained deep reinforcement learning model is correct according to the determined distance between the position information of the tracking target at the next time step and the position information of each drone at the next time step;
[0020] The method updates the trained deep reinforcement learning model according to the flight state data of each drone at the current time step, the flight action parameters of each drone at the current time step, the state reward corresponding to the flight state data of each drone at the current time step, and the flight state data of each drone at the next time step to obtain an updated deep reinforcement learning model, including:
[0021] If it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step, and the flight state data of each UAV at the next time step to obtain an updated deep reinforcement learning model.
[0022] In one embodiment, the Q network is updated by minimizing the following formula:
[0023]
[0024] Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a t represents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t , a t ) represents the soft Q function in the Q network, r(s t , a t ) represents the reward function, and γ is implicitly defined by the Q network.
[0025] In one embodiment, the parameters of the policy network are updated by minimizing the following formula:
[0026]
[0027] Among them, φ represents the parameters of the policy network, s t represents the flight status data of the drone at the current time step, D represents the experience playback pool, a t Represents the flight action parameters of the drone at the current time step, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network, Q θ (s t , a t ) represents the soft Q function in the Q network.
[0028] In one embodiment, the loss function of the temperature coefficient is:
[0029]
[0030] Among them, π t Implicitly defined by the Q network, H0 is a preset constant.
[0031] A target tracking device, comprising:
[0032] The data acquisition module is used to obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0033] A parameter determination module is used to input the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0034] The target tracking module is used to control the flight of each drone in the drone cluster according to the flight action parameters to track the tracking target.
[0035] A drone comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:
[0036] Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0037] Inputting the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0038] The flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target.
[0039] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:
[0040] Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0041] Inputting the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0042] The flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target.
[0043] The above-mentioned target tracking method, device, drone and storage medium first obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step, and then input the flight status data of each drone at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each drone at the current time step. Finally, the flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target. It can be understood that in the dynamic unknown target tracking task, the present application adopts deep reinforcement learning for drone control, and takes advantage of its environmental adaptability and powerful environmental feature characterization capabilities to make the optimal flight decision for the current time step in real time based on the position information of the target at the previous time step, so that the control of the drone can flexibly adapt to environmental changes, which is conducive to improving the success rate of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 A diagram showing an application environment of a target tracking method in an embodiment;
[0045] Figure 2 A schematic diagram of a target tracking method according to an embodiment of the present invention;
[0046] Figure 3 FIG. 1 is a diagram showing the internal structure of a drone in one embodiment. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0048] The target tracking method provided in this application can be applied to Figure 1 In the application environment shown in the figure, the drone cluster includes multiple drones, such as UAV1 and UAV2. The tracking target is also a drone. The drone cluster is used to intelligently track the tracking target according to the target tracking method involved in the following embodiment.
[0049] In one embodiment, Figure 2 As shown, a target tracking method is provided, which is applied to Figure 1 Take the drone in the video as an example to illustrate, including the following steps:
[0050] Step S202, obtaining the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step.
[0051] The drone cluster is composed of at least one drone. The flight status data refers to the status information of the drone during flight. Optionally, the flight status data includes at least one of position information or speed information. The tracking target is the target object being tracked.
[0052] Specifically, each drone in the drone cluster must obtain the flight status data of each drone at the current time step, that is, the flight status data of the drone and the flight status data of other drones in the drone cluster. Optionally, the drone can collect flight status data through its own installed sensors. The collected flight status data can be transmitted between drones. In addition, each drone must also obtain the position information of the tracking target at the previous time step.
[0053] More specifically, assume that the drone cluster is represented as: The position and speed information of each drone is used to represent the state of the entire system, that is, the system state vector can be expressed as formula (1):
[0054]
[0055] Among them, v i represents the velocity vector of drone i, p i Represents the position vector of drone i.
[0056] If we consider the UAV control situation in a three-dimensional coordinate system, we have v i =(v ix , v iy , v iz ) T and p i =(x i ,y i , z i ) T .
[0057] Step S204: input the flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step.
[0058] The flight action parameters refer to the parameters for controlling the flight of the drone. Optionally, the flight action parameters include at least one of acceleration or steering angle. Each drone can be set with a trained deep reinforcement learning model.
[0059] Specifically, each UAV inputs the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the respective trained deep reinforcement learning models to obtain the flight action parameters of each UAV at the current time step. Optionally, each UAV inputs the position information and speed information of each UAV at the current time step and the position information of the tracking target at the previous time step into the respective trained deep reinforcement learning models to obtain the acceleration and steering angle of each UAV at the current time step.
[0060] Optionally, before executing the actual target tracking method based on multi-UAV collaboration, the neural network in the deep reinforcement learning model, such as the Q network and the policy network, is first trained using simulated flight simulation. After the training is completed, it can be put into use. During a flight test, the flight state data samples of each UAV in the next time step and the position information samples of the tracking target in the previous time step are input into all the neural networks in the deep reinforcement learning model for training, and the parameters of the neural network are updated. After multiple rounds of training, the deep reinforcement learning model can make the next flight decision in real time according to the position information of the tracking target in the previous time step, thereby realizing intelligent tracking of the tracking target.
[0061] Step S206, controlling the flight of each drone in the drone cluster according to the flight action parameters to track the tracking target.
[0062] Specifically, each UAV flies according to its own flight action parameters to complete the tracking of the tracking target.
[0063] The above target tracking method first obtains the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step, and then inputs the flight status data of each drone at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each drone at the current time step. Finally, the flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target. It can be understood that this method uses deep reinforcement learning to control drones in dynamic unknown target tracking tasks, and takes advantage of its environmental adaptability and strong environmental feature representation capabilities. It can make the optimal flight decision for the current time step in real time based on the position information of the target at the previous time step, so that the control of the drone can flexibly adapt to environmental changes, which is conducive to improving the success rate of target tracking.
[0064] In one embodiment, a possible implementation method of the above step S204 "inputting the flight state data of each drone at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each drone at the current time step" is involved. Based on the above embodiment, step S204 can be specifically implemented by the following steps:
[0065] Step S2042, inputting the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the Q network in the trained deep reinforcement learning model, and obtaining the Q value corresponding to each candidate flight action parameter of each UAV at the current time step;
[0066] Step S2044, input the Q value corresponding to each candidate flight action parameter of each UAV at the current time step into the policy network in the trained deep reinforcement learning model to determine the flight action parameter of each UAV at the current time step.
[0067] Among them, the Q network is a neural network that represents the soft Q function Q θ (s t , a t ). The policy network is a neural network that represents the policy function π φ (a t |s t ) obtained.
[0068] Specifically, each UAV inputs the flight status data of the current time step of each UAV and the position information of the tracking target in the previous time step into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of the current time step of each UAV. Then, the Q value corresponding to each candidate flight action parameter of the current time step of each UAV is input into the policy network in the trained deep reinforcement learning model, so as to select the flight action parameters of each UAV at the current time step from the candidate flight action parameters.
[0069] In one embodiment, the method further comprises the following steps:
[0070] Step S212, obtaining a state reward corresponding to the flight state data of each drone at the current time step according to the Q value corresponding to the flight action parameter of each drone at the current time step;
[0071] Step S214, obtaining the flight status data of the next time step obtained by each UAV flying according to the flight action parameters;
[0072] Step S216, based on the flight status data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight status data of each UAV at the current time step, and the flight status data of each UAV at the next time step, the trained deep reinforcement learning model is updated to obtain an updated deep reinforcement learning model.
[0073] Specifically, each UAV obtains a state reward corresponding to the flight state data of each UAV at the current time step according to the Q value corresponding to the flight action parameters of each UAV at the current time step. Then, each UAV flies according to its own flight action parameters and collects the flight state data of the next time step. After that, the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step, and the flight state data of each UAV at the next time step are associated and stored in the experience replay pool. When the deep reinforcement learning model needs to be updated, the aforementioned data is obtained from the experience replay pool and input into the deep reinforcement learning model for dynamic updating of the model to obtain an updated deep reinforcement learning model to make it more perfect, and then the success rate of target tracking can be further improved according to the updated model.
[0074] In one embodiment, the method further comprises the following steps:
[0075] Step S222, obtaining the position information of the tracking target at the next time step;
[0076] Step S224, judging whether the current strategy executed in the trained deep reinforcement learning model is correct according to the determined distance between the position information of the tracking target at the next time step and the position information of each drone at the next time step;
[0077] Further, in one embodiment, step S216 may be implemented by the following steps:
[0078] Step S2162: If it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step, and the flight state data of each UAV at the next time step to obtain an updated deep reinforcement learning model.
[0079] Specifically, each UAV obtains the position information of the tracking target in the next time step, and calculates the distance between each UAV and the tracking target based on the position information of the tracking target in the next time step and the position information of each UAV in the next time step. If the distance is greater than the minimum distance threshold and less than the maximum distance threshold, it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, and then the model update is completed. Otherwise, it is determined that the current strategy executed in the trained deep reinforcement learning model is wrong.
[0080] In one embodiment, it involves the updating process of the Q network and the policy network. Specifically:
[0081] The Bellman equation followed by the update of the multi-UAV cooperative target tracking method based on deep reinforcement learning is shown in formula (2):
[0082]
[0083] And because the Bellman equation can be expressed as formula (3):
[0084]
[0085] The corresponding state reward is expressed as formula (4):
[0086]
[0087] Use neural network to represent soft Q function Q θ (s t , a t ) and the policy function π φ (a t |s t ). Q θ (s t , a t ) is replaced by the output of a multi-layer perceptron (MLP). Since a random strategy is used in this embodiment, π φ (a t |s t ) is a distribution, that is, the output of the policy network is a distribution. In this embodiment, Gaussian distribution is selected as the corresponding output. Since the policy distribution obeys the Boltzmann distribution, and formula (4) provides the relationship between the state reward, soft Q function and policy distribution, a policy is formalized as formula (5):
[0088]
[0089] Since the strategy distribution needs to be normalized, see formula (6):
[0090] ∫a π(a|s)da =1. (6)
[0091] Therefore, the state reward needs to satisfy formula (7):
[0092]
[0093] The form satisfied by formula (7) is the log-sum softmax function, that is, the definition formula (8):
[0094]
[0095] Then the dynamic programming form of the soft Q function can be expressed as formula (9):
[0096]
[0097] The update of the Q network can be achieved by minimizing the soft Bellman residual, as shown in formula (10):
[0098]
[0099] Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a t represents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t , a t ) represents the softQ function in the Q network, r(s t , a t ) represents the reward function, and γ is implicitly defined by the Q network.
[0100] The Bellman residual is optimized using the stochastic gradient method, as shown in formula (11):
[0101]
[0102] in, is a soft Q network with parameters, which is obtained by sliding the weight of the soft Q network through an exponential weight. Furthermore, in order to update the policy distribution, it is necessary to find a policy distribution that can approximate the policy distribution given by formula (5). In actual engineering, the policy distribution will be found in a given distribution family ∏. In order to measure the difference between the distribution being sought and the target distribution, this embodiment selects KL divergence (Kullback-Leibler Divergence) as the quantitative standard. The reason for this selection is that the research on KL divergence is very mature, and there are many ready-made mathematical tools to help with analysis. On the other hand, KL divergence has a certain convexity, which is conducive to analysis. The update formula is formula (12):
[0103]
[0104] The update of the policy network parameters is directly achieved by minimizing the KL divergence given by formula (12), as shown in formula (13):
[0105]
[0106] Among them, φ represents the parameters of the policy network, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network.
[0107] In order to minimize formula (13), neural network transformation is used to reparameterize, that is, as shown in formula (14):
[0108] a t =f φ (∈ t ;s t ), (14)
[0109] Among them, ∈ t is the input Gaussian noise vector. Then formula (13) can be rewritten as formula (15):
[0110]
[0111] Among them, π φ is implicitly defined by a neural network, then the gradient of formula (15) is approximated by the following formula, that is, formula (16):
[0112]
[0113] Among them, a t Depend on Estimated.
[0114] At this point, the update method of the Q network and the policy network has been solved. However, the choice of the temperature coefficient α is still fixed. Since the reward value will change continuously during the training process, the fixed temperature coefficient is unreasonable and will make it difficult for the entire training to converge. When the strategy is updated to a new area (or stage), the drone needs to try various paths extensively to obtain the optimal strategy, that is, the temperature coefficient is appropriately increased to increase the proportion of entropy. However, if a better strategy has been found and the feasible paths have been basically explored, the temperature coefficient needs to be reduced to ensure that the training can gradually stabilize. In summary, the optimization target of the temperature coefficient is as shown in formula (17):
[0115]
[0116] Finally, the loss function of the temperature coefficient is obtained as formula (18):
[0117]
[0118] Among them, π t Implicitly defined by the Q network, H0 is a preset constant.
[0119] At this point, the update formulas for all parameters have been determined.
[0120] An embodiment of the present application is introduced below in conjunction with a specific application scenario, specifically:
[0121] The operation of the algorithm in this embodiment is carried out alternately in two processes: one is to obtain feedback from the environment based on the current strategy, and the other is to use samples sampled from the experience replay pool to update the weights of the neural network in the deep reinforcement learning model. This embodiment assumes that the environment evolution has Markov properties and follows the Markov evolution equation, that is, s t+1 ~ξ(s t+1 |s t , a t After the training is completed, real-time UAV cooperative tracking can be performed. The cooperative target tracking of this embodiment is mainly divided into two stages: environmental feedback and strategy update.
[0122] When launching a flight experiment, include:
[0123] Step S302, the drone cluster detects the position information of the tracking target in the previous time step and the flight status data of the surrounding drones in the current time step;
[0124] Step S304, the UAV clusters transmit the flight status data of the current time step to each other and check the channel quality between the UAVs;
[0125] Step S306, each UAV inputs the position information of the tracking target at the previous time step and the flight state data of each UAV at the current time step into its own deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step, namely, acceleration and steering angle;
[0126] Step S308, detecting the position information of the tracking target at the next time step, and updating the distances between the UAVs and the distances between the UAVs and the tracking target to determine whether the current flight decision is correct;
[0127] Step S310, based on the judgment result, update the neural network parameters in the deep reinforcement learning model and return to step S302 until the flight experiment is completed.
[0128] In this embodiment, deep reinforcement learning is used for drone control, taking advantage of its ability to adapt to the environment and powerful environmental feature representation capabilities, which is beneficial to improving the success rate of target tracking.
[0129] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps in may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time step, but can be executed at different time steps. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.
[0130] In one embodiment, Figure 3 As shown, a target tracking device is provided, comprising: a data acquisition module, a parameter determination module and a target tracking module, wherein:
[0131] The data acquisition module is used to obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0132] The parameter determination module is used to input the flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0133] The target tracking module is used to control the flight of each drone in the drone cluster according to the flight action parameters to track the tracking target.
[0134] In the above-mentioned target tracking device, the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step are first obtained, and then the flight status data of each drone at the current time step and the position information of the tracking target at the previous time step are input into the trained deep reinforcement learning model to obtain the flight action parameters of each drone at the current time step. Finally, the flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target. It can be understood that in the dynamic unknown target tracking task, this device uses deep reinforcement learning to control drones, and uses its advantages of environmental adaptability and strong environmental feature representation capabilities to make the optimal flight decision for the current time step in real time based on the position information of the target at the previous time step, so that the control of the drone can flexibly adapt to environmental changes, which is conducive to improving the success rate of target tracking.
[0135] In one embodiment, the parameter determination module is specifically used to input the flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step; input the Q value corresponding to each candidate flight action parameter of each UAV at the current time step into the policy network in the trained deep reinforcement learning model to determine the flight action parameters of each UAV at the current time step.
[0136] For the specific definition of the target tracking device, please refer to the definition of the target tracking method above, which will not be repeated here. Each module in the above target tracking device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the drone in hardware form, or stored in the memory of the drone in software form, so that the processor can call and execute the operations corresponding to the above modules.
[0137] In one embodiment, a drone is provided, whose internal structure diagram can be as follows: Figure 3As shown. The drone includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the drone is used to provide computing and control capabilities. The memory of the drone includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the drone is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a target tracking method is implemented. The display screen of the drone can be a liquid crystal display screen or an electronic ink display screen, and the input device of the drone can be a touch layer covering the display screen, or a button, trackball or touchpad set on the drone shell, or an external keyboard, touchpad or mouse, etc.
[0138] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the drone to which the solution of the present application is applied. A specific drone may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0139] In one embodiment, a drone is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0140] Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0141] The flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step are input into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0142] The flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target.
[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0144] The flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step are input into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step; the Q value corresponding to each candidate flight action parameter of each UAV at the current time step is input into the policy network in the trained deep reinforcement learning model to determine the flight action parameters of each UAV at the current time step.
[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0146] According to the Q value corresponding to the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step is obtained; the flight state data of each UAV at the next time step obtained by flying according to the flight action parameters is obtained; according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step and the flight state data of each UAV at the next time step, the trained deep reinforcement learning model is updated to obtain an updated deep reinforcement learning model.
[0147] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0148] The position information of the tracking target in the next time step is obtained; according to the distance determined between the position information of the tracking target in the next time step and the position information of each UAV in the next time step, whether the current strategy executed in the trained deep reinforcement learning model is correct; if it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV in the current time step, the flight action parameters of each UAV in the current time step, the state reward corresponding to the flight state data of each UAV in the current time step, and the flight state data of each UAV in the next time step to obtain an updated deep reinforcement learning model.
[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0150] The Q network is updated by minimizing the following formula:
[0151]
[0152] Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a t represents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t ,at ) represents the soft Q function in the Q network, r(s t ,a t ) represents the reward function, and γ is implicitly defined by the Q network.
[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0154] The parameters of the policy network are updated by minimizing the following formula:
[0155]
[0156] Among them, φ represents the parameters of the policy network, s t represents the flight status data of the drone at the current time step, D represents the experience playback pool, a t Represents the flight action parameters of the drone at the current time step, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network, Q θ (s t , a t ) represents the soft Q function in the Q network.
[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0158] The loss function of the temperature coefficient is:
[0159]
[0160] Among them, π t Implicitly defined by the Q network, H0 is a preset constant.
[0161] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0162] Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step;
[0163] The flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step are input into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step;
[0164] The flight of each drone in the drone cluster is controlled according to the flight action parameters to track the tracking target.
[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0166] The flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step are input into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step; the Q value corresponding to each candidate flight action parameter of each UAV at the current time step is input into the policy network in the trained deep reinforcement learning model to determine the flight action parameters of each UAV at the current time step.
[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0168] According to the Q value corresponding to the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step is obtained; the flight state data of each UAV at the next time step obtained by flying according to the flight action parameters is obtained; according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step and the flight state data of each UAV at the next time step, the trained deep reinforcement learning model is updated to obtain an updated deep reinforcement learning model.
[0169] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0170] The position information of the tracking target in the next time step is obtained; according to the distance determined between the position information of the tracking target in the next time step and the position information of each UAV in the next time step, whether the current strategy executed in the trained deep reinforcement learning model is correct; if it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV in the current time step, the flight action parameters of each UAV in the current time step, the state reward corresponding to the flight state data of each UAV in the current time step, and the flight state data of each UAV in the next time step to obtain an updated deep reinforcement learning model.
[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0172] The Q network is updated by minimizing the following formula:
[0173]
[0174] Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a trepresents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t , a t ) represents the soft Q function in the Q network, r(s t , a t ) represents the reward function, and γ is implicitly defined by the Q network.
[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0176] The parameters of the policy network are updated by minimizing the following formula:
[0177]
[0178] Among them, φ represents the parameters of the policy network, s t represents the flight status data of the drone at the current time step, D represents the experience playback pool, a t Represents the flight action parameters of the drone at the current time step, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network, Q θ (s t , a t ) represents the soft Q function in the Q network.
[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0180] The loss function of the temperature coefficient is:
[0181]
[0182] Among them, π t Implicitly defined by the Q network, H0 is a preset constant.
[0183] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0184] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0185] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A target tracking method, characterized in that: The method comprises: Obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step; Inputting the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step; Controlling the flight of each drone in the drone cluster according to the flight action parameters to track the tracking target; The step of inputting the flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step includes: inputting the flight status data of each UAV at the current time step and the position information of the tracking target at the previous time step into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step; inputting the Q value corresponding to each candidate flight action parameter of each UAV at the current time step into the policy network in the trained deep reinforcement learning model to determine the flight action parameters of each UAV at the current time step; The Q network is updated by minimizing the following formula: Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a t represents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t ,a t ) represents the soft Q function in the Q network, r(s t ,a t ) represents the reward function, γ is implicitly defined by the Q network, is a soft Q network with parameters, represents the mathematical expectation of the flight status data of the drone at the next time step randomly sampled from the distribution p, It represents the mathematical expectation of the UAV's flight state data at the current time step and the flight action parameters at the current time step randomly sampled in the experience replay pool D. Indicates that the flight status data of the drone in the next time step is s t+1 The average return value when the action is taken. The average here refers to the average of all possible actions. Among them, the parameters of the policy network are updated by minimizing the following formula: Among them, φ represents the parameters of the policy network, s t represents the flight status data of the drone at the current time step, D represents the experience playback pool, a t Represents the flight action parameters of the drone at the current time step, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network, Q θ (s t , a t ) represents the soft Q function in the Q network, represents the mathematical expectation of the UAV’s current time step flight state data randomly sampled from the experience replay pool D, Indicates that the flight action parameters of the drone at the current time step follow the distribution π φ The mathematical expectation of the sampling.
2. The method according to claim 1, characterized in that The method further comprises: Obtaining a state reward corresponding to the flight state data of each UAV at the current time step according to the Q value corresponding to the flight action parameter of each UAV at the current time step; Acquire flight state data of the next time step obtained by each UAV flying according to the flight action parameters; According to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step and the flight state data of each UAV at the next time step, the trained deep reinforcement learning model is updated to obtain an updated deep reinforcement learning model.
3. The method according to claim 2, characterized in that The flight status data includes location information; the method further includes: Obtaining the position information of the tracking target at the next time step; Determining whether the current strategy executed in the trained deep reinforcement learning model is correct according to the determined distance between the position information of the tracking target at the next time step and the position information of each drone at the next time step; The method updates the trained deep reinforcement learning model according to the flight state data of each drone at the current time step, the flight action parameters of each drone at the current time step, the state reward corresponding to the flight state data of each drone at the current time step, and the flight state data of each drone at the next time step to obtain an updated deep reinforcement learning model, including: If it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step, and the flight state data of each UAV at the next time step to obtain an updated deep reinforcement learning model.
4. The method according to claim 1, characterized in that: The loss function of the temperature coefficient is: Among them, π t Implicitly defined by the Q network, Η0 is a preset constant, Indicates that the flight action parameters of the drone at the current time step follow the distribution π t The mathematical expectation of the sampling.
5. A target tracking device, characterized in that: The device comprises: The data acquisition module is used to obtain the flight status data of each drone in the drone cluster at the current time step and the position information of the tracking target at the previous time step; A parameter determination module is used to input the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the trained deep reinforcement learning model to obtain the flight action parameters of each UAV at the current time step; A target tracking module, used to control the flight of each drone in the drone cluster according to the flight action parameters to track the tracking target; The parameter determination module is specifically used to: input the flight state data of each UAV at the current time step and the position information of the tracking target at the previous time step into the Q network in the trained deep reinforcement learning model to obtain the Q value corresponding to each candidate flight action parameter of each UAV at the current time step; input the Q value corresponding to each candidate flight action parameter of each UAV at the current time step into the policy network in the trained deep reinforcement learning model to determine the flight action parameter of each UAV at the current time step; The Q network is updated by minimizing the following formula: Among them, θ represents the parameters of the Q network, s t Represents the flight status data of the drone at the current time step, a t represents the flight action parameters of the drone at the current time step, D represents the experience playback pool, Q θ (s t ,a t ) represents the soft Q function in the Q network, r(s t ,a t ) represents the reward function, γ is implicitly defined by the Q network, is a soft Q network with parameters, represents the mathematical expectation of the flight status data of the drone at the next time step randomly sampled from the distribution p, It represents the mathematical expectation of the UAV's flight state data at the current time step and the flight action parameters at the current time step randomly sampled in the experience replay pool D. Indicates that the flight status data of the drone in the next time step is s t+1 The average return value when the action is taken. The average here refers to the average of all possible actions. Among them, the parameters of the policy network are updated by minimizing the following formula: Among them, φ represents the parameters of the policy network, s t represents the flight status data of the drone at the current time step, D represents the experience playback pool, a t Represents the flight action parameters of the drone at the current time step, π φ Implicitly defined by the Q network, α represents the temperature coefficient, π φ (a t |s t ) represents the policy function in the policy network, Q θ (s t , a t ) represents the soft Q function in the Q network, represents the mathematical expectation of the UAV’s current time step flight state data randomly sampled from the experience replay pool D, Indicates that the flight action parameters of the drone at the current time step follow the distribution π φ The mathematical expectation of the sampling.
6. The device according to claim 5, characterized in that The device also includes: A first acquisition module is used to obtain a state reward corresponding to the flight state data of each drone at the current time step according to the Q value corresponding to the flight action parameter of each drone at the current time step; A second acquisition module is used to acquire the flight state data of the next time step obtained by each UAV flying according to the flight action parameters; An updating module is used to update the trained deep reinforcement learning model according to the flight state data of each drone at the current time step, the flight action parameters of each drone at the current time step, the state reward corresponding to the flight state data of each drone at the current time step, and the flight state data of each drone at the next time step, so as to obtain an updated deep reinforcement learning model.
7. The device according to claim 6, characterized in that The flight status data includes location information; the device also includes: A third acquisition module is used to obtain the position information of the tracking target at the next time step; A judgment module, used to judge whether the current strategy executed in the trained deep reinforcement learning model is correct according to the distance determined between the position information of the tracking target at the next time step and the position information of each drone at the next time step; The update module is specifically used for: If it is determined that the current strategy executed in the trained deep reinforcement learning model is correct, the trained deep reinforcement learning model is updated according to the flight state data of each UAV at the current time step, the flight action parameters of each UAV at the current time step, the state reward corresponding to the flight state data of each UAV at the current time step, and the flight state data of each UAV at the next time step to obtain an updated deep reinforcement learning model.
8. The device according to claim 5, characterized in that The loss function of the temperature coefficient is: Among them, π t Implicitly defined by the Q network, Η0 is a preset constant, Indicates that the flight action parameters of the drone at the current time step follow the distribution π t The mathematical expectation of the sampling.
9. A drone, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
Intelligent real-time prediction method for high-speed large-range maneuvering target track
CN110309909A
Control method and system for moving and landing of quadrotor unmanned aerial vehicle based on hybrid strategy
CN111338375A
Tracking method of airborne unmanned aerial vehicle multi-target tracking system based on deep learning
CN111932588A
Unmanned aerial vehicle maneuvering target tracking method integrating Kalman filtering and DDQN algorithm
CN112435275A
Lidar site model to aid counter drone system
US20170261999A1
