A method and system for constructing a multi-task target tracking model for unmanned aerial vehicles
By building and integrating multiple basic models, the problem of inaccurate decision-making of the drone target tracking model when the target moving attributes change is solved, and the tracking effect of the drone for multiple target tracking tasks is improved. It is suitable for limited computing power Drones achieve accurate tracking when target tracking tasks change.
Patent Information
- Application Number
- CN202410560556.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-05-08
AI Technical Summary
The existing drone target tracking model makes inaccurate decisions when the target moving attributes change, and the multi-task reinforcement learning model has a large amount of computation, so it is not suitable for drones with limited computing power.
Building a multi-task target tracking model By building the state vector and action vector of the drone, training multiple basic models, and calculating the weight fusion model based on the average round total reward, forming a target tracking model suitable for multi-task.
It improves the generalization ability and tracking effect of the model, reduces the amount of computing, and is suitable for drones with limited computing power, achieving accurate tracking when target tracking tasks change.
Smart Images

Figure CN118569058B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicle control technology, and in particular to a method and system for constructing a multi-task target tracking model for unmanned aerial vehicles. Background Art
[0002] As drones become more diverse, their mission areas continue to expand, and using drones to track designated targets is a common application scenario. Drones obtain the location information of mission targets through their own sensors, such as images, radars, etc., or wireless communications, and continue to track the target in this direction so that it does not leave its monitoring range.
[0003] In more and more studies, reinforcement learning is used to solve the target tracking problem of drones. Different from common supervised learning and unsupervised learning, reinforcement learning is essentially a machine learning method that uses environmental feedback as input and adapts to the environment. It imitates the basic way of learning of humans or animals in nature, and uses evaluative feedback signals to optimize the behavioral decisions taken through continuous interaction with the environment and trial and error.
[0004] However, many current studies focus on a single task of UAVs. When the target's mobility properties (such as moving speed, direction of travel, etc.) change, the same effect cannot be achieved as before the change, limiting the application of UAVs in certain scenarios. Some existing multi-task reinforcement learning focuses on achieving knowledge sharing through model structure changes. In order to support multiple tasks at the same time, too many parameters are set, resulting in a large amount of calculation, which is not suitable for multi-task tracking of UAVs with limited computing power. Therefore, a target tracking model suitable for multiple tasks can enable UAVs to make flexible and accurate decisions, enabling them to cope with more complex and diverse mission requirements. Summary of the invention
[0005] In view of the above analysis, an embodiment of the present invention aims to provide a method and system for constructing a multi-task target tracking model for UAVs, so as to solve the problem that the decision provided by the target tracking model is inaccurate after the target tracking task changes, resulting in poor UAV tracking effect.
[0006] On the one hand, an embodiment of the present invention provides a method for constructing a multi-task target tracking model for a drone, comprising the following steps:
[0007] Construct the state vector and action vector of the drone, set different target tracking tasks, and train the reinforcement learning model through interaction with the simulation environment to obtain multiple basic models;
[0008] For each target tracking task, each basic model is used to interact with the simulation environment for multiple rounds, and the average round total reward of each basic model is calculated; according to the average round total reward of each basic model, the weight of each basic model in each target tracking task is obtained;
[0009] The basic models are fused according to their weights in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
[0010] Based on the further improvement of the above method, the target tracking task includes the target's moving speed and direction; the model structure of each basic model is the same, including a policy network and a value network; the input of the policy network is the state vector of the drone, and the output is the action vector; the input of the value network is the state vector of the drone, and the output is the value of the state under the current action.
[0011] Based on the further improvement of the above method, for each target tracking task, each base model interacts with the simulation environment the same number of times in each round.
[0012] Based on the further improvement of the above method, each basic model is used to interact with the simulation environment for multiple rounds, and the average round total reward of each basic model is calculated, including:
[0013] For each basic model, each time each drone interacts with the simulation environment in each round, the simulation environment feeds back a reward value, and the reward value of each interaction is summed up to get the total reward of each basic model in each round; then, according to the number of rounds, the average total reward of each basic model is obtained.
[0014] Based on the further improvement of the above method, the simulation environment feeds back a reward value, which is obtained by calculating the difference between the average distance between each drone and the target at the next moment and the average distance between each drone and the target at the current moment.
[0015] Based on the further improvement of the above method, the weight of each basic model in each target tracking task is obtained according to the average round total reward of each basic model, including:
[0016] The basic model corresponding to the minimum value of the average round total reward in each target tracking task is taken as the base model, and its initial weight is set to 1; the initial weights of other basic models are obtained by calculating the ratio of their respective average round total rewards to the average round total reward of the base model; the initial weights are normalized to obtain the weights of each basic model.
[0017] Based on the further improvement of the above method, the initial weights of other basic models are calculated by the following formula:
[0018]
[0019] in, is the initial weight of the i-th base model in the j-th target tracking task, is the average round total reward of the base model in the j-th target tracking task, is the average total round reward of the ith base model in the jth target tracking task.
[0020] Based on the further improvement of the above method, the target tracking model corresponding to each target tracking task is obtained by fusing the basic models according to the weights. The target tracking model having the same model structure as the basic model is constructed, and the sum of the products of the network parameters at the same position of each basic model in the same target tracking task and the weights of the basic model is used as the network parameters at the same position of the target tracking model to obtain the target tracking model corresponding to the target tracking task.
[0021] Based on the further improvement of the above method, the state vector of the UAV includes: the horizontal speed, vertical speed, horizontal position and vertical position of the current UAV, the horizontal distance and vertical distance between the current UAV and the target, and the horizontal distance and vertical distance between other UAVs and the current UAV; the action vector includes: the force applied to the current UAV in the positive horizontal direction, the force applied to the current UAV in the negative horizontal direction, the force applied to the current UAV in the positive vertical direction, and the force applied to the current UAV in the negative vertical direction.
[0022] On the other hand, an embodiment of the present invention provides a system for constructing a multi-task target tracking model for a drone, comprising:
[0023] The basic model training module is used to construct the state vector and action vector of the UAV. By setting different target tracking tasks in the simulation environment to train the reinforcement learning model, multiple basic models are obtained.
[0024] The weight acquisition module is used to calculate the average round total reward of each basic model after using each basic model to interact with the simulation environment for multiple rounds for each target tracking task; according to the average round total reward of each basic model, the weight of each basic model in each target tracking task is obtained;
[0025] The tracking model acquisition module is used to fuse the basic models according to the weights of the basic models in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
[0026] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0027] 1. Through the reinforcement learning model, multiple target tracking tasks are learned multiple times to obtain multiple basic models; then the weights are calculated according to the average round total reward of performing the same target tracking task, and the multiple basic models are integrated to improve the generalization ability of the model. The integrated model can make more flexible and accurate decisions when facing multiple tasks, improve the tracking effect of drones on various target tracking tasks, and realize accurate tracking when the target tracking task changes.
[0028] 2. By integrating multiple basic models, it avoids setting too many parameters to adapt to different target tracking tasks. The calculation amount is small and it is suitable for multi-task tracking scenarios of drones with limited computing power.
[0029] 3. The weight acquisition method is applicable to all reinforcement learning paradigms. Models with exactly the same structure except for network parameters speed up the learning and training speed of intelligent agents when facing multiple tasks.
[0030] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.
[0032] Figure 1 This is a flow chart of a method for constructing a multi-task target tracking model for a drone in Example 1 of the present invention;
[0033] Figure 2 This is a schematic diagram of the system structure for building a multi-task target tracking model for unmanned aerial vehicles in Example 2 of the present invention. DETAILED DESCRIPTION
[0034] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0035] Example 1
[0036] A specific embodiment of the present invention discloses a method for constructing a multi-task target tracking model for unmanned aerial vehicles, such as Figure 1 As shown, the following steps are included:
[0037] S1. Construct the state vector and action vector of the drone, set different target tracking tasks respectively, and train the reinforcement learning model by interacting with the simulation environment to obtain multiple basic models.
[0038] It should be noted that the target tracking task in this embodiment includes the moving speed and moving direction of the target, such as the target moving at a uniform speed and in a directional manner, and the target moving randomly and in an undirectional manner. In other words, the moving speed of the target in each target tracking task may be different, and the changing rules of the moving direction may be different. Multiple drones need to track the moving target and avoid collisions between drones.
[0039] Based on this scenario, the state vector of the drone includes: the horizontal speed, vertical speed, horizontal position and vertical position of the current drone, the horizontal distance and vertical distance between the current drone and the target, and the horizontal distance and vertical distance between other drones and the current drone, which belongs to the continuous state space; the action vector of the drone includes: the force applied to the current drone in the positive horizontal direction, the force applied to the current drone in the negative horizontal direction, the force applied to the current drone in the positive vertical direction, and the force applied to the current drone in the negative vertical direction, which belongs to the continuous action space.
[0040] The reinforcement learning model includes a policy network and a value network. This embodiment does not limit the network model structure. For example, both the policy network and the value network can adopt a two-layer fully connected network. Among them, the input of the policy network is the state vector of the drone, and the output is the action vector; the input of the value network is the state vector of the drone, and the output is the value of the state under the current action.
[0041] After determining the target tracking task, the reinforcement learning algorithm is used to update the policy network parameters in the process of interacting with the simulation environment, so that the probability of selecting high-reward actions increases and the probability of selecting low-reward actions decreases. In this way, the strategy is improved step by step until the optimal policy network is trained.
[0042] It should be noted that the model structures of the basic models trained in this embodiment are the same, but the network parameters are different. The basic models are divided into two categories: one is for the same target tracking task, by not setting a random seed, and training under the condition that the target attributes, action space, state space, and reward function are exactly the same; the second is for different target tracking tasks, by not setting a random seed, and training under the condition that the action space, state space, and reward function are the same but the target attributes are different.
[0043] It should be noted that setting the same random seed value during training can make the training process repeatable and obtain the same network parameters. Conversely, not setting the random seed can obtain different network parameters.
[0044] S2. For each target tracking task, after using each basic model to interact with the simulation environment for multiple rounds, calculate the average round total reward of each basic model; according to the average round total reward of each basic model, obtain the weight of each basic model in each target tracking task.
[0045] It should be noted that the target tracking tasks for which the trained basic models are facing are the tasks that each of them is good at. In order to accurately complete the target tracking tasks that they are good at and to cope with changes in tasks, this embodiment calculates a set of different weights for each basic model for each target tracking task, so that the fused integrated model can not only handle the target tracking task well, but also successfully complete other different target tracking tasks.
[0046] Considering that the reward value is an indicator for evaluating the quality of a strategy or an action in reinforcement learning, and the training process of reinforcement learning is based on rounds, this embodiment fuses the basic models based on the weights calculated based on the average round total reward.
[0047] It should be noted that an episode is a sequence of interactions between the agent and the environment, which ends with a final state. In this embodiment, reaching the number of interactions is used as the end mark, that is, for each target tracking task, each basic model interacts with the simulation environment the same number of times in each episode. However, the number of interactions with the simulation environment in each episode may be different for different target tracking tasks, and the total number of episodes may also be different.
[0048] For example, in the tracking task of "target uniform directional movement", each basic model interacts with the simulation environment 500 times in each round, and a total of 30 rounds are executed; in the tracking task of "target random non-directional movement", each basic model interacts with the simulation environment 600 times in each round, and a total of 50 rounds are executed.
[0049] Furthermore, after using each basic model to interact with the simulation environment for multiple rounds, the average round total reward of each basic model is calculated, including:
[0050] For each basic model, in each round, each time each drone interacts with the simulation environment, the simulation environment feeds back a reward value, and the reward value of each interaction is summed up to get the total reward of each basic model in each round; then according to the number of rounds, the average total round reward of each basic model is obtained; among them, the simulation environment feeds back a reward value, which is obtained by calculating the difference between the average distance between each drone and the target at the next moment and the average distance between each drone and the target at the current moment.
[0051] Next, according to the average round total reward of each basic model, the weight of each basic model in each target tracking task is obtained, including:
[0052] The basic model corresponding to the minimum value of the average round total reward in each target tracking task is taken as the base model, and its initial weight is set to 1; the initial weights of other basic models are obtained by calculating the ratio of their respective average round total rewards to the average round total reward of the base model; the initial weights are normalized to obtain the weights of each basic model.
[0053] Specifically, the initial weights of other basic models are calculated using the following formula:
[0054]
[0055] in, is the initial weight of the i-th base model in the j-th target tracking task, is the average round total reward of the base model in the j-th target tracking task, is the average total round reward of the ith base model in the jth target tracking task.
[0056] Furthermore, each initial weight is normalized to between [0, 1] by the following formula, and the sum of the normalized weights under the unified target tracking task is 1:
[0057]
[0058] in, for The normalized weights, n represents the total number of basic models.
[0059] For example, there are 2 basic models, the average round total reward of the first basic model is -1125.93991, and the average round total reward of the second basic model is -944.48179. Then the first basic model 1 is the base model, and the initial weights of the two basic models are 1 and 1.19212453 respectively, and the normalized weights are 0.4562 and 0.5438 respectively.
[0060] It should be emphasized that this embodiment performs multiple learning and training for different target tracking tasks to obtain multiple basic models, and then calculates the weight of each basic model in each target tracking task according to the average total round reward of performing the same target tracking task, so that the basic models have a larger weight and contribute more in the target tracking tasks they are good at, while increasing their generalization ability in other target tracking tasks.
[0061] It can be understood that the method for obtaining weights in this embodiment is applicable to all reinforcement learning paradigms, and models with exactly the same structure except for network parameters accelerate the learning and training speed of the intelligent agent when facing multiple tasks.
[0062] S3. The basic models are fused according to the weights of the basic models in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
[0063] Specifically, the basic models are fused according to the weights to obtain the integrated model corresponding to each target tracking task. This is done by constructing a target tracking model with the same model structure as the basic model, and taking the sum of the products of the network parameters at the same position of each basic model in the same target tracking task and the weights of the basic model as the network parameters at the same position of the target tracking model to obtain the target tracking model corresponding to the target tracking task.
[0064] It should be noted that by fusing multiple basic models, we avoid setting too many parameters to adapt to different target tracking tasks, and the amount of calculation is small, which is suitable for multi-task tracking scenarios of drones with limited computing power. The fused integrated model performs better than any basic model in the target tracking task when completing its corresponding target tracking task, and can make accurate decisions flexibly after the task changes, which improves the tracking effect of drones for multiple target tracking tasks and realizes accurate tracking when the target tracking task changes.
[0065] In practical applications, the corresponding target tracking model is determined according to the target tracking task to be completed in the real environment, the state vector generated by the interaction between each UAV and the real environment is passed into the target tracking model, and each UAV is controlled to track the target according to the output of the strategy network of the target tracking model.
[0066] It should be noted that during tracking, the state vector generated by the interaction between each UAV and the real environment is passed into the target tracking model. The mean and variance of each action corresponding to each UAV are output according to the strategy network of the target tracking model. The action vector of each UAV is randomly selected according to the probability distribution of the mean and variance, and each UAV is controlled to perform action tracking target, and then the real environment interaction is iterated until the target is tracked.
[0067] Compared with the prior art, the present embodiment provides a method for constructing a multi-task target tracking model for unmanned aerial vehicles. Multiple target tracking tasks are learned multiple times through a reinforcement learning model to obtain multiple basic models. Then, the weights are calculated according to the average round total reward of executing the same target tracking task, and then the multiple basic models are integrated, thereby improving the generalization ability of the model. The integrated integrated model can make more flexible and accurate decisions when facing multiple tasks, thereby improving the tracking effect of the unmanned aerial vehicle on multiple target tracking tasks, and realizing accurate tracking when the target tracking task changes. By fusing multiple basic models, it is avoided to set too many parameters to adapt to different target tracking tasks, and the calculation amount is small, which is suitable for unmanned aerial vehicle multi-task tracking scenarios with limited computing power. The method for obtaining weights is applicable to all reinforcement learning paradigms, and models with exactly the same structure except for network parameters accelerate the learning and training speed of the intelligent agent when facing multiple tasks.
[0068] Example 2
[0069] Another embodiment of the present invention discloses a system for constructing a multi-task target tracking model for drones, thereby realizing a method for constructing a multi-task target tracking model for drones in Embodiment 1. The specific implementation of each module refers to the corresponding description in Embodiment 1. Figure 2 As shown, the system includes:
[0070] The basic model training module 101 is used to construct the state vector and action vector of the UAV, and to obtain multiple basic models by setting different target tracking tasks to train the reinforcement learning model in the simulation environment;
[0071] The weight acquisition module 102 is used to calculate the average round total reward of each basic model after using each basic model to interact with the simulation environment for multiple rounds for each target tracking task; and obtain the weight of each basic model in each target tracking task according to the average round total reward of each basic model;
[0072] The tracking model acquisition module 103 is used to fuse the basic models according to the weights of the basic models in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
[0073] Since the system for constructing a multi-task target tracking model for drones in this embodiment and the method for constructing a multi-task target tracking model for drones in the above embodiment are related, they can be used for reference, and are repeated here, so they will not be repeated here. Since the principles of this system embodiment and the above method embodiment are the same, this system embodiment also has the corresponding technical effects of the above method embodiment.
[0074] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0075] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. A method for constructing a multi-task target tracking model for unmanned aerial vehicles, characterized in that: The following steps are involved: The state vector and action vector of the drone are constructed, and different target tracking tasks are set respectively. A plurality of basic models are obtained by interactively training the reinforcement learning model with the simulation environment; the target tracking task includes the moving speed and moving direction of the target; the model structure of each basic model is the same, and both include a policy network and a value network; the input of the policy network is the state vector of the drone, and the output is the action vector; the input of the value network is the state vector of the drone, and the output is the value of the state under the current action; For each target tracking task, each basic model is used to interact with the simulation environment for multiple rounds, and the average round total reward of each basic model is calculated; According to the average round total reward of each basic model, the weight of each basic model in each target tracking task is obtained, including: for each basic model, each time each drone interacts with the simulation environment in each round, the simulation environment feeds back a reward value, and the reward value of each interaction is summarized to obtain the total reward of each basic model in each round; then according to the number of rounds, the average round total reward of each basic model is obtained; the basic model corresponding to the minimum value of the average round total reward in each target tracking task is used as the base model, and its initial weight is set to 1; the initial weights of other basic models are obtained by calculating the ratio of their respective average round total rewards to the average round total reward of the base model; each initial weight is normalized to obtain the weight of each basic model; the simulation environment feeds back a reward value, which is obtained by calculating the difference between the average distance of each drone and the target at the next moment and the average distance of each drone and the target at the current moment; The basic models are fused according to their weights in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
2. The method for constructing a multi-task target tracking model for unmanned aerial vehicles according to claim 1, characterized in that: For each target tracking task, each base model interacts with the simulation environment the same number of times in each round.
3. The method for constructing a multi-task target tracking model for unmanned aerial vehicles according to claim 1, characterized in that: The initial weights of the other basic models are calculated by the following formula: in, is the initial weight of the i-th base model in the j-th target tracking task, is the average round total reward of the base model in the j-th target tracking task, r i j is the average total round reward of the ith base model in the jth target tracking task.
4. The method for constructing a multi-task target tracking model for unmanned aerial vehicles according to claim 1, characterized in that: The target tracking model corresponding to each target tracking task is obtained by fusing the basic models according to the weights of the basic models in each target tracking task, by constructing a target tracking model with the same model structure as the basic model, and taking the sum of the products of the network parameters at the same position of each basic model in the same target tracking task and the weights of the basic model as the network parameters at the same position of the target tracking model to obtain the target tracking model corresponding to the target tracking task.
5. The method for constructing a multi-task target tracking model for unmanned aerial vehicles according to claim 1, characterized in that: The state vector of the UAV includes: the horizontal speed, vertical speed, horizontal position and vertical position of the current UAV, the horizontal distance and vertical distance between the current UAV and the target, and the horizontal distance and vertical distance between other UAVs and the current UAV; the action vector includes: the force applied to the current UAV in the positive horizontal direction, the force applied to the current UAV in the negative horizontal direction, the force applied to the current UAV in the positive vertical direction, and the force applied to the current UAV in the negative vertical direction.
6. A system for constructing a multi-task target tracking model for unmanned aerial vehicles, characterized in that: include: The basic model training module is used to construct the state vector and action vector of the UAV, and to obtain multiple basic models by setting different target tracking tasks in the simulation environment to train the reinforcement learning model; the target tracking task includes the moving speed and moving direction of the target; the model structure of each basic model is the same, including a policy network and a value network; the input of the policy network is the state vector of the UAV, and the output is the action vector; the input of the value network is the state vector of the UAV, and the output is the value of the state under the current action; The weight acquisition module is used to calculate the average round total reward of each basic model after using each basic model to interact with the simulation environment for multiple rounds for each target tracking task; According to the average round total reward of each basic model, the weight of each basic model in each target tracking task is obtained, including: for each basic model, each time each drone interacts with the simulation environment in each round, the simulation environment feeds back a reward value, and the reward value of each interaction is summarized to obtain the total reward of each basic model in each round; then according to the number of rounds, the average round total reward of each basic model is obtained; the basic model corresponding to the minimum value of the average round total reward in each target tracking task is used as the base model, and its initial weight is set to 1; the initial weights of other basic models are obtained by calculating the ratio of their respective average round total rewards to the average round total reward of the base model; each initial weight is normalized to obtain the weight of each basic model; the simulation environment feeds back a reward value, which is obtained by calculating the difference between the average distance of each drone and the target at the next moment and the average distance of each drone and the target at the current moment; The tracking model acquisition module is used to fuse the basic models according to the weights of the basic models in each target tracking task to obtain the target tracking model corresponding to each target tracking task.
Citation Information
Patent Citations
Unmanned aerial vehicle trajectory tracking state compensation depth deterministic strategy gradient control method
CN114967472A
Multi-unmanned aerial vehicle multi-target cooperative tracking control method based on MAPPO algorithm
CN115509251A
Unmanned aerial vehicle many-to-many pursuit game method based on PSO-M3D DDPG
CN116796843A
Unmanned aerial vehicle target tracking method for realizing self-obstacle avoidance based on reinforcement learning
CN117369502A
KR20230080179A
