Multi-unmanned aerial vehicle target exploration and tracking method and system based on multi-agent ant colony reinforcement learning

Abstract

The invention discloses a multi-unmanned aerial vehicle target exploration and tracking method and system based on multi-agent ant colony reinforcement learning. The method comprises the following steps: dividing an actual environment for target tracking of unmanned aerial vehicles into grid units; performing target state initialization based on the grid units to obtain an initial observation position of the unmanned aerial vehicle; based on an ant colony algorithm and the initial observation position of the unmanned aerial vehicle, the transition probability of the unmanned aerial vehicle among the grid units is calculated; establishing a reward function based on the transition probability and the grid unit; based on the reward function and the existing exploration information and tracking information, performing centralized training on the multiple unmanned aerial vehicles by using a multi-agent near-end strategy optimization algorithm to obtain an optimal joint decision; and based on the optimal joint decision, enabling the unmanned aerial vehicle to complete exploration and tracking of multiple targets to be explored. According to the technical scheme, the possibility that the unmanned aerial vehicle finds the target in the unknown area is increased, and the target can be effectively followed.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-UAV (Unmanned Aerial Vehicle, UAVs) intelligent management, and specifically relates to a multi-UAV target exploration and tracking method and system based on multi-agent ant colony reinforcement learning. Background Art

[0002] With the rapid development of today's intelligent systems and UAV technology, target exploration and tracking in a specific space has become an important direction of UAV applications. In fields such as autonomous navigation, environmental monitoring, and disaster relief, how to effectively use UAVs for target exploration and tracking has important practical significance. However, due to the limited field of view (FOV) of UAVs and the uncertainty of targets in unknown environments, traditional exploration and tracking methods face challenges. How to improve the efficiency of UAV target exploration and tracking in unknown areas has become an urgent problem to be solved.

[0003] The target exploration and tracking of UAVs can be regarded as a multi-sensor dynamic path planning problem. UAVs need to search for targets and effectively track targets in the shortest possible time, so they need to plan the best path at every moment to achieve their goals. So far, many scholars have conducted extensive research in the direction of intelligent agent path planning and proposed many effective algorithms. For example, traditional path planning algorithms such as Dijkstra algorithm, A* algorithm and artificial potential field (APF) are simple and easy to implement, deterministic and efficient, but they are limited to static environments and simple graph structures. They have limited processing capabilities for solving dynamic environments, large-scale problems or complex graph structures, may fall into local optimal solutions, and are sensitive to parameters. In order to solve these problems, more advanced meta-heuristic algorithms have emerged, such as genetic algorithms (GA), particle swarm optimization (PSO) and ant colony algorithms (ACO). GA algorithm has the characteristics of strong global search ability and wide applicability, but slow convergence speed and difficult parameter adjustment; PSO is simple and easy to implement, with strong global search ability, but it is easy to fall into local optimal solution and is not suitable for discrete action space problems; ACO is suitable for solving discrete problems, has good global search ability and stability, but has more algorithm parameters. With the continuous advancement of science and technology, more advanced path planning algorithms have emerged, and in order to achieve better path planning effects, some new algorithms may combine multiple technologies. These new algorithms are based on methods such as deep learning, reinforcement learning (RL) and hybrid meta-heuristics, with higher efficiency, better performance and wider applicability. Han et al. proposed a new method: the five-neighborhood search A* algorithm, which introduces dynamic weighting in the heuristic function and uses the second-order Bezier curve to smooth the path, solving the problems of too many redundant nodes, slow search speed, and sharp turns in the path of the A* algorithm. Song et al. proposed an improved PSO algorithm to solve the problems of local capture and premature convergence. Adaptive fractional speed was introduced into the original PSO algorithm, and certain disturbances were applied to the particle swarm according to its evolutionary state, thereby enhancing the ability of the particle swarm to jump out of the local minimum point and exploring the search space more thoroughly. In response to the path planning problem of unmanned underwater vehicles at sea, Chen et al. proposed an ant colony optimization-artificial potential field method based on grid graph improvement for local and global path planning of unmanned underwater vehicles in dynamic environments. The improved ACO optimization mechanism was used to search for the global optimal path of the unmanned underwater vehicle from the starting point to the end point in the grid environment, and the improved APF algorithm was combined to avoid unknown obstacles during the navigation of the unmanned underwater vehicle.Tabrizi et al. proposed an algorithm that can process path planning in parallel through Q-learning and ACO algorithms to control nanorobot drugs injected into the body, realize their autonomous path planning, and improve the accuracy and speed of drug treatment.

[0004] In summary, although the improved path planning algorithms have shown good experimental results in solving common problems, for the dynamic path planning of UAVs, the task allocation between them needs to be considered. This is a complex multi-objective optimization problem, and the existing improved heuristic path planning method cannot effectively solve such problems. For the multi-target exploration and tracking task of UAVs, it is modeled as a partially observable Markov decision process of a multi-agent, which requires solving many problems. Problem 1: At the initial moment of task execution, what strategy should UAVs use to explore the area to improve the coverage of the space; Problem 2: When the UAV discovers a single or multiple targets, how to effectively track the target and transmit information to other UAVs to achieve UAVs collaboration; Problem 3: How does the UAV pursue the target when the target is lost or about to escape the UAV's field of view. Summary of the invention

[0005] In order to solve the decision-making optimization problems involved in target exploration and tracking of UAVs (multi-UAVs) in unknown areas, the present invention provides a multi-UAV target exploration and tracking method and system based on multi-agent ant colony reinforcement learning, wherein IACO (improved ant colony algorithm) expands the function of pheromones in the original ACO, and combines MAPPO (multi-agent proximal strategy optimization) to parallelly realize the control problem of UAVs. The combined IACO-MAPPO algorithm can improve the exploration efficiency of UAVs in grid space, increase the possibility of UAVs finding targets in unknown areas, and can also effectively follow targets.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The multi-UAV target exploration and tracking method based on multi-agent ant colony reinforcement learning includes the following steps:

[0008] Divide the actual environment where the UAV performs multi-target tracking into grid cells;

[0009] Based on the grid cells, target state initialization is performed to obtain an initial observation position of the UAV;

[0010] Based on the ant colony algorithm and the initial observation position of the drone, the transfer probability of the drone between each grid cell is calculated;

[0011] Based on the transition probability and the grid unit, a reward function is established to guide the UAV to perform area exploration and target tracking;

[0012] Based on the reward function and the existing exploration information and tracking information, a multi-agent proximal strategy optimization algorithm is used to centrally train multiple drones to obtain an optimal joint decision; wherein the optimal joint decision includes a joint action decision for exploration and tracking of multiple targets;

[0013] Based on the optimal joint decision, the UAV completes the exploration and tracking of multiple targets to be explored.

[0014] Preferably, the target state initialization includes initializing the position of the target at the initial moment, the state transfer matrix, the observation noise, the measurement noise and the covariance matrix.

[0015] Preferably, the ant colony algorithm is an ant colony algorithm for improving the pheromone application method, and the improvement method includes:

[0016] Pheromones are divided into exploration pheromones and tracking pheromones. Among them, exploration pheromones are used to guide drones to explore unfamiliar environments; tracking pheromones are used to guide drones to approach the grid cells of explored targets and to collaborate in exploring and tracking other surrounding targets.

[0017] Before the drone detects the target, the exploration pheromone is secreted in the grid cells along the drone's movement trajectory and evaporates over time;

[0018] After the UAV explores the target, the exploration pheromone stops being secreted, and the tracking pheromone is secreted in the corresponding grid cell where the UAV explores the target, and diffuses in a decreasing manner to the adjacent grid cells within the preset range; among them, only one type of pheromone is effective in each grid cell at the same time.

[0019] Preferably, the method for calculating the transfer probability of the drone between each grid unit includes:

[0020] Presetting the initial value of the exploration pheromone concentration and the initial value of the tracking pheromone concentration;

[0021] When the UAV is exploring the current grid cell, the local exploration pheromone concentration of the current grid cell is updated based on the initial value of the exploration pheromone concentration and the exploration pheromone increment left by the current grid cell in each exploration;

[0022] When the drone discovers a target in the grid cell where it is located, the local tracking pheromone concentration of the current grid cell is updated based on the initial value of the tracking pheromone concentration of the grid cell and the tracking pheromone increment left by the drone tracking the target;

[0023] Calculate path heuristic information based on the Euclidean distance from the current grid cell to the adjacent grid cells;

[0024] Based on the local exploration pheromone concentration and the local tracking pheromone concentration of the current grid cell, the local exploration pheromone concentration and the local tracking pheromone concentration of the adjacent grid cell, as well as the path heuristic information and the expected heuristic factor, the transfer probability of the UAV from the current grid cell to the adjacent grid cell is obtained.

[0025] Preferably, the reward function includes exploration reward, collision reward, tracking reward and pursuit reward.

[0026] Preferably, the method for obtaining the optimal joint decision includes:

[0027] When the drone finds the target, it stops secreting exploration pheromones and starts releasing and spreading tracking pheromones.

[0028] The drone initializes and estimates the target within the field of view; wherein the initialization includes target state initialization, state transfer matrix initialization and covariance initialization; the estimation includes state estimation and state covariance estimation;

[0029] When the same target enters the field of view of multiple UAVs, the target trajectory is fused based on the initialization and estimation results of the target to obtain a fused trajectory;

[0030] A pseudo-measurement equation is established based on the fused trajectory, and multiple UAVs are centrally trained in a multi-agent proximal strategy optimization algorithm based on the reward function and existing exploration information and tracking information. After the training is completed, the optimal joint decision of the multiple UAVs is obtained.

[0031] The present invention also provides a multi-UAV target exploration and tracking system based on multi-agent ant colony reinforcement learning, which is used to implement the method, including:

[0032] A grid division module is used to divide the actual environment of the UAV for multi-target tracking into grid units;

[0033] A target initialization module is used to initialize the target state based on the grid unit to obtain the initial observation position of the UAV;

[0034] The transfer probability calculation module is used to calculate the transfer probability of the UAV between each grid cell based on the ant colony algorithm and the initial observation position of the UAV;

[0035] A reward function establishment module, used to establish a reward function for guiding the UAV to perform area exploration and target tracking based on the transition probability and the grid unit;

[0036] A joint decision module is used to centrally train multiple UAVs using a multi-agent proximal strategy optimization algorithm based on the reward function and existing exploration information and tracking information to obtain an optimal joint decision; wherein the optimal joint decision includes a joint action decision for exploration and tracking of multiple targets;

[0037] The exploration and tracking module is used to enable the UAV to complete the exploration and tracking of multiple targets to be explored based on the optimal joint decision.

[0038] Preferably, the transition probability calculation module includes an ant colony algorithm improvement unit, which is used to improve the existing ant colony algorithm; the ant colony improved algorithm includes:

[0039] The pheromone division subunit is used to divide pheromones into exploration pheromones and tracking pheromones. Among them, the exploration pheromones are used to guide the UAV to explore unfamiliar environments; the tracking pheromones are used to guide the UAV to approach the grid unit of the explored target and to collaborate in exploring and tracking other surrounding targets.

[0040] The exploration pheromone secretion subunit is used to explore the secretion of pheromones in the grid cells that the drone passes along its movement trajectory before the drone detects the target, and volatilizes over time;

[0041] The tracking pheromone secretion subunit is used to stop the secretion of exploration pheromones after the UAV explores the target, and secrete the tracking pheromone in the corresponding grid unit where the UAV explores the target, and diffuse it to the adjacent grid units within the preset range in a decreasing manner; among them, only one type of pheromone is effective in each grid unit at the same time.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: This patent adopts the improved ACO and combines it with the MAPPO algorithm to control UAVs to achieve target exploration and tracking in unknown areas. First, an improved ACO algorithm IACO is proposed, which expands the pheromones in the original ACO into two categories: exploration pheromones and tracking pheromones. Exploration pheromones can prevent UAVs from repeatedly exploring a certain area, while tracking pheromones guide nearby UAVs to move closer to the target appearance area. IACO's guidance of the exploration process combined with the centralized training and decentralized execution of the MAPPO algorithm can effectively improve the exploration efficiency of UAVs for unknown areas and increase the probability of discovering targets. Then, by designing a reasonable reward function, the target can be continuously kept within the field of view of the UAVs, effectively completing the exploration and tracking tasks of the UAVs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a gridded UAVs exploration and tracking scenario according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of a multi-target exploration and tracking process based on IACO-MAPPO in an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the UAVs exploration and tracking process k=50 according to an embodiment of the present invention;

[0047] Figure 4 Schematic diagram of the UAVs exploration and tracking process k=100 according to an embodiment of the present invention;

[0048] Figure 5 Schematic diagram of pheromone secretion k=50 during the exploration and tracking process of UAVs in an embodiment of the present invention;

[0049] Figure 6 Schematic diagram of pheromone secretion k=100 during the exploration and tracking process of UAVs in an embodiment of the present invention;

[0050] Figure 7 A schematic diagram showing comparison of coverage rates under different control algorithms in an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of comparing target discovery times under different control algorithms in an embodiment of the present invention;

[0052] Fig. 9 This is a flow chart of a multi-UAV target exploration and tracking method based on multi-agent ant colony reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Embodiment 1

[0056] like Fig. 9 As shown in FIG. 1 , a multi-UAV target exploration and tracking method based on multi-agent ant colony reinforcement learning includes the following steps:

[0057] S1: Divide the actual environment of the UAV for multi-target tracking into grid cells; In multi-target tracking, dividing the actual environment into grid cells can describe and model the spatial structure of the environment. At the same time, the environment gridding is used to represent the UAV's perception range, the location of the target, and other environmental information, thereby helping the UAV to perceive, plan, and make decisions. Specifically, Figure 1 As shown in the figure, in order to simplify the complexity of the problem, the exploration and tracking problem of UAVs is handled in a two-dimensional plane, and the entire environment is divided into a two-dimensional 20*20 grid. Each grid cell represents a small area in the environment, and all grids have their own state representation. Figure 1 As shown: the grid state is represented as 1 to 400, the black solid circle represents the UAVs position, the black ring represents the detectable range of the corresponding UAV, and the hollow circle represents the target position. In the environment setting, the size of a single grid area depends on the limited visual range of the UAV, so that the UAV can effectively observe all targets in the area as long as it is in the area. During the exploration process of the UAV, the pheromone secretion of IACO is also carried out according to the area.

[0058] S2: Initialize the target state based on the grid cells to obtain the initial observation position of the UAV; specifically, the initial observation position of multiple UAVs. In a further implementation, the target state initialization includes initializing the position of the target at the initial moment, the state transfer matrix, the observation noise, the measurement noise, and the covariance matrix.

[0059] During the exploration process of UAVs, IACO's pheromone secretion will be carried out according to the area. During the mission, in order to facilitate the learning of the MARL network, the state of the UAV position needs to be converted:

[0060]

[0061] Among them, state i Indicates the status of UAV i, Posx i and Posy i They represent the horizontal and vertical coordinates of UAV i respectively, L is the side length of the grid, ceil() means rounding down, and fix() means rounding up.

[0062] In the set scenario, UAVs are homogeneous and use perception to track targets. There is no clear assignment between UAVs and targets, that is, each UAV can track any target that enters the field of view, and the same target can be tracked by one or more UAVs. This allows for a more flexible and extensive form of cooperation between UAVs. UAVs fly at a fixed altitude and explore the area in the form of discrete actions. The motion process of UAV is regarded as a two-dimensional plane motion, and the dynamic model of UAV at time k can be expressed as:

[0063]

[0064] Where s = 1, ..., m, m represents the number of UAVs, χ s,k+1 represents the position of UAV s at time k+1, υ step represents the motion step length of the UAV, a s,k = [1, 2, …, 8] represents the actions that the UAV can take to move to the adjacent grid.

[0065] For complex nonlinear systems, the present invention uses a combination of Unscented Kalman Filter (UKF) and Joint Probabilistic Data Association (JPDA) to achieve multi-target state estimation. UKF does not require Taylor expansion of the state transfer function and the observation function, so it is more efficient in calculation. Assume that the discrete-time nonlinear dynamic model is as follows:

[0066] x k+1 =f k (x k ,w k )

[0067] z s,k+1 =h s,k+1 (x k+1 ,v s,k+1 )

[0068] Where k∈N is the time index and k=1,...,T, T represents the length of the event, x k is the n×1-dimensional state vector of the system at time k, z s,k+1 is the measurement vector of the system state by UAV s at time k+1, w k and v s,k+1 Represented as uncorrelated Gaussian white noise, f k and h s,k+1 are the nonlinear state function and measurement function respectively.

[0069] In the process of target exploration and tracking by UAVs, when the target enters the UAV field of view, the UAV can obtain measurement information, and the state estimation and error covariance estimation of the target can be obtained through nonlinear filtering calculation. Where t=1,...,c, c represents the number of detected targets. Assume that the measurement change of the dynamic target t obtained by UAV s at time k+1 is:

[0070]

[0071] Among them, χ s,k =[x s,k ,y s,k ] Τ represents the location of UAV s at time k.

[0072] The distance r and azimuth θ measured by the UAV to the target can be written as:

[0073]

[0074] in, and They are the prior state estimates of target t at time k (horizontal and vertical coordinates), is the measurement noise of UAV s on target t at time k.

[0075] S3: Based on the ant colony algorithm and the initial observation position of the UAV, the transfer probability of the UAV between each grid cell is calculated; the ACO algorithm guides the ant colony to explore the optimal path by secreting pheromones. Introducing it into the multi-target tracking scenario can effectively guide UAVs to perform exploration and tracking tasks. In order to improve the exploration efficiency of the algorithm in the early stage, ACO was improved.

[0076] In the environment of UAVs in the FOV, pheromones are used as a medium for information exchange between UAVs, and their secretion can be obtained by all UAVs as global information. In the original ACO, the calculation of state transition probability is mainly based on the pheromones of ants and the heuristic information of each path, while the present invention improves the application method of pheromones.

[0077] A further implementation method is that the ant colony algorithm is an ant colony algorithm for improving the pheromone application method, and the improvement method includes:

[0078] Pheromones are divided into exploration pheromones and tracking pheromones. Among them, exploration pheromones are used to guide UAVs to explore unfamiliar environments; tracking pheromones are used to guide UAVs to approach grid cells of explored targets and to collaboratively explore and track other surrounding targets; exploration pheromones are mainly used to guide UAVs to effectively explore unfamiliar environments in the early stages of a mission, avoid UAVs from repeatedly exploring the same area, improve scene coverage, and thus increase the possibility of discovering targets.

[0079] Before the drone detects the target, the exploration pheromone is secreted in the grid cells along the drone's movement trajectory and evaporates over time;

[0080] After the UAV has discovered the target, the exploration pheromone stops being secreted, and the tracking pheromone is secreted in the corresponding grid cell where the UAV has discovered the target, and diffuses in a decreasing manner to the adjacent grid cells within the preset range; through this diffusion behavior, other UAVs are guided to approach the area where the target has been discovered, and conduct collaborative tracking and explore other surrounding targets. In the algorithm, the tracking pheromone is set to diffuse to the adjacent area in a decreasing manner, and the diffusion range is set according to the number of grid areas. In addition, when the UAV discovers the target, it stops secreting the exploration pheromone and starts secreting the tracking pheromone. That is, only one type of pheromone is effective in each grid cell at the same time.

[0081] A further implementation method is that the method for calculating the transfer probability of the drone between each grid unit includes:

[0082] Preset the initial value of exploration pheromone concentration and the initial value of tracking pheromone concentration; specifically, in order to prevent the state transition probability from being 0 in subsequent calculations, all exploration pheromones and tracking pheromones in all areas will have an initial value The initial value is close to 0. Each area (grid unit) can only have one type of pheromone at the same time. The higher concentration of pheromone works in this area, and the lower concentration of pheromone returns to the initial value. At the same time, in order to prevent the UAV from falling into the local optimum, when the exploration or tracking pheromone of area j is less than 0.1 due to evaporation, it is restored to the initial value.

[0083] When the drone is exploring in the current grid cell, the local exploration pheromone concentration of the current grid cell is updated based on the initial value of the exploration pheromone concentration and the exploration pheromone increment left by the current grid cell in each exploration; the global exploration pheromone attenuation concentration is obtained based on the global exploration pheromone evaporation coefficient and the local exploration pheromone concentration;

[0084] Specifically, when the drone explores a certain area i, the exploration pheromone of the area is locally updated:

[0085]

[0086] in, It represents the exploration pheromone concentration of region i at time k, and its initial value is Δτ explore Represents the pheromone increment left after each exploration.

[0087] In order to prevent pheromones from being too concentrated in a specific area, it is necessary to consider the attenuation of global pheromones:

[0088]

[0089] where α e To globally explore the evaporation coefficient of pheromones and control the decay rate of pheromones.

[0090] When the UAV explores the target in the grid cell where it is located, the local tracking pheromone concentration of the current grid cell is updated based on the initial value of the tracking pheromone concentration and the tracking pheromone increment left by the UAV when tracking the target; the global tracking pheromone attenuation concentration is obtained based on the global tracking pheromone evaporation coefficient and the local tracking pheromone concentration.

[0091] Specifically, when the UAV finds a target in area i at time k, it locally updates the tracking pheromone in that area:

[0092]

[0093] in, represents the tracking pheromone concentration in region i at time k, Δτ track Indicates the increment of pheromone left by the tracking target.

[0094] Similarly, considering the decay of the global tracking pheromone:

[0095]

[0096] where α t Global tracking of pheromone evaporation coefficients.

[0097] Calculate path heuristic information based on the Euclidean distance from the current grid cell to the adjacent grid cells;

[0098] Based on the local exploration pheromone concentration and the local tracking pheromone concentration of the current grid cell, the local exploration pheromone concentration and the local tracking pheromone concentration of the adjacent grid cell, as well as the path heuristic information and the expected heuristic factor, the transfer probability of the UAV from the current grid cell to the adjacent grid cell is obtained.

[0099] Specifically, in most ACO algorithm path planning applications, the UAV will decide the direction of the next action based on the concentration of pheromones when exploring or tracking. However, when using MAPPO to make decisions for UAVs, the transition probability of the ACO algorithm is used as an important influencing factor, which takes into account the role of exploration and tracking pheromones to determine the transition probability of the UAV moving from area i to the adjacent area j at time k:

[0100]

[0101] Among them, β is the desired heuristic factor, m∈neighbors represents the set of all neighboring regions of region i, Represents the exploration pheromone (explore) or tracking pheromone (track) contained in region j at time k. ij,k is the path heuristic information from region i to j, calculated as follows:

[0102]

[0103] where d ij represents the Euclidean distance from region i to region j.

[0104] S4: Based on the transition probability and the grid unit, a reward function is established to guide the UAV to perform area exploration and target tracking; a further implementation method is that the reward function includes an exploration reward, a collision reward, a tracking reward, and a pursuit reward.

[0105] In this embodiment, there are many factors that need to be considered in the process of UAVs multi-target tracking, and designing a reasonable reward function can guide the UAV to perform effective area exploration and target tracking.

[0106] Exploration reward: In the early stage of UAV mission execution, exploration is the primary task of UAVs. Its setting has a very important impact on the coverage of the scene and the response time of the target. Pheromones are an important factor in the exploration process. Whether it is exploration pheromones or tracking pheromones, their influence is taken into account in the state transition probability. In area i, the higher the exploration pheromone concentration, the lower the state transition probability, and the higher the tracking pheromone concentration, the higher the state transition probability. Therefore, when the exploration pheromone is in effect, it is set to a negative reward value, and the smaller the transition probability, the larger the negative value; when the tracking pheromone is in effect, it is set to a positive reward value, as shown below:

[0107]

[0108] Among them, R expl,s,k represents the exploration reward obtained by UAVs at time k.

[0109] Collision penalty: In order to prevent collisions between UAVs, a safe distance must be set. When they exceed the safe distance, there must be a corresponding penalty. On the one hand, the collision penalty ensures the safety of the UAV itself, and on the other hand, it prevents UAVs from tracking in the same target area, thereby improving resource utilization. The reward function is designed as follows:

[0110]

[0111] Among them, D i,j represents the distance between UAVs i and j, and i≠j, D safe Indicates the safe distance between UAVs.

[0112] Tracking reward: Both the target and UAVs are in dynamic motion, and the observation range of the UAV is limited by the distance, that is, the closer the target is to the UAV, the closer the detection probability is to 1, and the farther the target is, the smaller the detection probability is. If the detection probability is lower than the set value, the target is considered undetectable. The detailed setting parameters will be explained in the simulation link. Therefore, the distance between the UAV and the target affects its tracking effect on the target, so the tracking reward of UAV s for target t is set as:

[0113]

[0114] Where D max is the maximum detection range of the UAV, is the distance between UAV s and target t at time k.

[0115] Pursuit reward: In a complex environment, the UAV may lose the target at some point. When the target is lost from the field of view, the UAV needs to move to the position where the target is lost as soon as possible to pursue it. For a UAV that tracks multiple targets, if a target is lost within its field of view, the nearby UAV in exploration mode will be responsible for the pursuit. The pursuit reward must be triggered when the UAVs detected the target t before, but the target was lost at time k-1, so the pursuit is formed at time k, which requires the UAVs to quickly approach the position where the target is lost. The reward is set as follows:

[0116]

[0117] Therefore, the reward value that UAV s can obtain at time k is:

[0118]

[0119] If the UAV finds the target, step S5 is executed, otherwise the UAV is always in the exploration process.

[0120] S5: Based on the reward function and the existing exploration and tracking information, the multi-agent proximal strategy optimization algorithm is used to centrally train multiple drones to obtain the optimal joint decision; the optimal joint decision includes the joint action decision of exploration and tracking of multiple targets;

[0121] A further implementation method is that the method for obtaining the optimal joint decision includes:

[0122] When the drone finds the target, it stops secreting exploration pheromones and starts releasing and spreading tracking pheromones.

[0123] The drone initializes and estimates the target within the field of view; the initialization includes target state initialization, state transfer matrix initialization and covariance initialization; the estimation includes state estimation and state covariance estimation;

[0124] When the same target enters the field of view of multiple UAVs, the target trajectory is fused based on the initialization and estimation results of the target to obtain a fused trajectory; in this embodiment, for the same target that enters the field of view of multiple UAVs at the same time, track association processing needs to be performed in the state center, and the DBSCAN clustering algorithm is used for track clustering; after clustering, the GCI fusion algorithm is used to perform track fusion to obtain the fused state estimation and state covariance estimation;

[0125] Specifically, the multi-target tracking problem for UAVs usually consists of two parts: target state estimation and track fusion. When a target enters the field of view of the UAV, the motion state of the target is estimated, and the UAV makes a decision on the next action based on the state estimation result. In the multi-target tracking process, each UAV has its own filter for target tracking. When the same target enters the field of view of multiple UAVs at the same time, the trajectory of the target needs to be transmitted to the data processing center for data fusion. JPDA-UKF filtering is used as the target state estimator, and the GCI method is used for track fusion to achieve target tracking.

[0126] UKF combines the prediction update of the standard Kalman filter with the unscented transform to sequentially estimate the target state. When the target t enters the field of view of the UAV s at a certain time k, the UAV can only obtain the target's position information, but cannot obtain information such as the target's speed, so it is necessary to initialize the target's state, state transfer matrix, and covariance matrix. The distance r and angle θ measured by the UAV are used to calculate the target's position information. Used to initialize the position component in the state vector:

[0127]

[0128] in, It is the measurement information used for state initialization. x and y represent the position of the target, and the speed of the horizontal and vertical axes respectively. Since it cannot be obtained, it is assumed to be 0.

[0129] Target state transfer matrix Initialization: Assume that the target moves in a uniform straight line at the initial moment of detection. In this case, the state transfer matrix can be expressed as:

[0130]

[0131] Here, κ represents the sampling interval.

[0132] Covariance matrix (Same as above covariance initialization) also needs to be initialized:

[0133]

[0134] in, and represents the uncertainty of the position, and Represents the uncertainty of the velocity.

[0135] Due to the lack of sufficient measurement data, the initial estimation may have large errors, and the UAV can only keep tracking the target by approaching the position provided by the measurement data. As time goes by, the UAV will obtain more measurement information about the target, and the approximate speed of the target can be estimated by using the position difference between the previous and next moments, which is used to update the state and state transfer matrix of the target. Finally, the UKF is used to predict the state of the target, providing effective information for the UAV's action decision at the next moment.

[0136] The UKF prediction step requires the calculation of the δ sampling point matrix:

[0137]

[0138] And given by:

[0139]

[0140] Among them, n x is a constant, usually sampled 2n x +1 δ sampling point, is a scale parameter, is the Cholesky decomposition of the scaled state covariance matrix. n x ×n x dimensional matrix, whose column vector is The δ sampling points are propagated through the state transfer matrix:

[0141]

[0142] Use the unscented weights w n Get the predicted state value:

[0143]

[0144] And calculate the prediction covariance matrix by adding state noise:

[0145]

[0146] Taking into account the influence of state noise, the δ sampling points are redrawn using the transferred state and state covariance estimate:

[0147]

[0148]

[0149] Then propagate through the measurement equation:

[0150]

[0151] The predicted measurement vector and its covariance matrix are:

[0152]

[0153] After receiving the i-th measurement value z k (i) Then, calculate the new information This can be done through the Kalman update process. th Update of the measured value to the state vector and the state covariance matrix prediction value:

[0154]

[0155] in, is the Kalman filter gain, which can be calculated by the following formula:

[0156]

[0157] In the update step, JPDA takes into account all possible measurement-target associations that fall into the association gate and updates the target state estimate by calculating the joint probability. Given by:

[0158]

[0159] Where N k is the number of confirmed measurements from target t at time k, represents the interconnection probability that the ith measurement originates from the target t, i = 0, 1, ..., N k , i = 0 means that no measurement originates from the target t and satisfies the constraint condition:

[0160] The state covariance of the target is updated as:

[0161]

[0162] If the target is about to leave the UAV's field of view, it is necessary to predict the state based on the previous state and motion model, and use the predicted data to control the UAVs or pursue the target.

[0163] For the same target that enters the field of view of multiple UAVs at the same time, it is necessary to perform track association processing in the state center and use the DBSCAN clustering algorithm for track clustering. Assuming the real target state and the local trajectories of UAVs s1, s2 Related, then and The statistical distance between them can be expressed as:

[0164]

[0165] in, and They are and The error covariance matrix of represents the cross covariance between two estimated trajectories caused by common noise.

[0166] If the statistical distance between two trajectories from different UAVs is less than ε, the two trajectories are considered to have the same origin. The trajectories from different UAVs are divided into multiple clusters, each of which corresponds to the same target origin. Assume that there are n c Tracks are fused using the GCI fusion algorithm to obtain the fused state estimate and state covariance estimate:

[0167]

[0168] where ω s Indicates the weight of each UAV data in the fusion process, and the weight satisfies

[0169] Based on the fusion trajectory, a pseudo-measurement equation is established, and multiple drones are centrally trained in the multi-agent proximal policy optimization (MAPPO) algorithm based on the reward function and the existing exploration information and tracking information. After the training, the optimal joint decision of multiple drones is obtained. Among them, the pseudo-measurement equation is the process of using the existing target state information to estimate the target state at the next moment, which is also realized through the UKF-JPDA formula.

[0170] The target may be lost during the UAV tracking process. In this case, the UAV is required to quickly move to the location where the target is lost, predict the state of the target at the next moment based on past information, and pursue it. If the pursuit fails, the UAV continues to perform the exploration mission.

[0171] MAPPO is an algorithm for MARL that extends the single-agent Proximal Policy Optimization (PPO) algorithm. Compared to single-agent PPO, in MAPPO, all agents share a centralized critic. The centralized critic evaluates the behavior of each individual agent in the environment of all agents. To do this, it receives as input the global state and all actions in time step t. CLIP The function calculates the parameter θ, which represents the intensity with which the parameters of the agent's neural network must be adjusted. For the adjustment of the critic's neural network, in addition to the global state, the rewards need to be distributed. Therefore, the agents are relative to their own rewards to maximize the overall reward. The critic in MAPPO itself only needs to be trained. During execution, each agent takes an action based on the available state, which implicitly takes into account the actions of other agents.

[0172] In PPO, L CLIP It means that when the policy is updated, a ratio is calculated, that is, the probability ratio of the new policy and the old policy to choose the same action in the same state. Then, this ratio is clipped and limited to a range to control the update amplitude of the policy parameters, which can be expressed as:

[0173]

[0174] Among them, θ l represents the actor parameters of iteration l, the θ in the brackets represents the parameters to be optimized, and p θ (a k |s k ) represents the probability of action a occurring in state s at time k, represents the probability ratio of the new and old strategies, is the advantage function, clip() means when When it is less than 1-ε, the output is 1-ε; when it is greater than 1+ε, the output is 1+ε; and if it is between 1-ε and 1+ε, it remains unchanged.

[0175] Update the value function parameters by minimizing the mean square error loss function:

[0176]

[0177] Where V θ (s k ) is the value function estimate, is the target return.

[0178] S6: Based on the optimal joint decision, the UAV completes the exploration and tracking of multiple targets to be explored.

[0179] Figure 2 It is the target exploration and tracking process of UAVs based on IACO-MAPPO. As shown in the figure, at the initial moment, the main task of UAV is exploration. When a target enters the field of view of UAV, its main task becomes tracking the target. The joint action decisions of all UAVs at each moment must be trained and output in the MAPPO network. The information obtained by UAVs at each moment is input into a multi-agent parallel training space, and MAPPO is trained using this information. After obtaining the global information at the k moment, a pseudo-measurement equation is established to predict the UAVs action at the next moment. Whether it is UAVs that have detected the target or UAVs in the exploration stage, their next action in the training will try to obtain a higher reward value, and the impact on each other will be considered. In the pseudo-measurement equation, the predicted data is used to plan the next action of UAVs, so that UAVs can better track the target.

[0180] Figure 3 , Figure 4 The following are the exploration and tracking processes of UAVs under k = 50 and k = 100. Since there is a lot of randomness in the path planning of UAVs, a simulation with a relatively common experimental phenomenon is selected for illustration. Figure 3 It can be seen from the figure that when k = 50, since the initial distance of UAV 1 is closer to target b, it finds target b and tracks it during the exploration process. UAV 2 and UAV 3 both find target c during the exploration process and perform collaborative tracking. UAV 4 is still in the exploration stage. Figure 4 As shown in the figure, when k = 100, target a is discovered by UAV 2 because the trajectories of targets a and b are close, and the cooperative tracking of target c ends. UAV 2 and UAV 3 track targets a and c respectively, and UAV 4 is still exploring other areas. It can be seen from the figure that the purpose of UAVs is to explore as many areas as possible and track as many targets as possible.

[0181] Figure 5 , Figure 6 They are the distribution of UAVs exploration and tracking pheromone concentrations under k = 50 and k = 100, corresponding to Figure 3 and Figure 4 The yellow and cyan colors in the figure represent the concentration changes of exploration and tracking pheromones, respectively. The concentration changes from high to low, and the color changes from dark to light. Figure 5 It can be seen that when k = 50, the exploration pheromone covers a wide area while the tracking pheromone only covers a small area. When k = 100, the tracking pheromone covers a larger area than the exploration pheromone, and there are many high-concentration areas. This shows that exploration is the main task of UAVs at the initial moment. As time goes by, the target is gradually detected, and tracking the target becomes the main task of UAVs. Exploration and tracking are carried out simultaneously.

[0182] Figure 7 After 100 independent Monte Carlos, the coverage of the five algorithms was compared as the tasks were performed. It can be seen that there is no obvious difference between IACO-MAPPO and other MARL algorithms at the initial moment. As time goes by, the coverage of this algorithm will gradually be higher than other algorithms. As the exploration task proceeds, IACO-MAPPO will reduce the rate of increase of the coverage rate due to the discovery of the target, so that the curve tends to converge faster. For the independent use of the IACO algorithm to control UAVs, it has obvious advantages over the traditional GA algorithm, but there is still a certain gap compared with IACO combined with MARL. In the early stage of exploration, the size of the coverage rate can illustrate the superiority of the algorithm performance. As the exploration task proceeds, a good algorithm relies on efficient exploration performance to make the coverage rate curve tend to converge faster.

[0183] Figure 8 After 100 independent Monte Carlo tests, the time it takes for different algorithms to find a target was tested. Under the same initial position of the UAVs, it can be seen that there is not much difference in the time it takes for multiple algorithms to explore the first target, but IACO-MAPPO still shows better results in finding the second or third target, which is of course closely related to its higher coverage rate. Figure 8 It is proved once again that the combination of IACO and MARL has a great improvement in the exploration and tracking of UAVs with limited field of view, and MAPPO can be better combined and applied with IACO compared with Multi-agent Deep Q-network (MADQN) and Multi-agent Policy Gradient (MAPG).

[0184] Embodiment 2

[0185] The present invention also provides a multi-UAV target exploration and tracking system based on multi-agent ant colony reinforcement learning, which is used to implement the method, including:

[0186] A grid division module is used to divide the actual environment of the UAV for multi-target tracking into grid units;

[0187] The target initialization module is used to initialize the target state based on the grid unit and obtain the initial observation position of the UAV;

[0188] The transfer probability calculation module is used to calculate the transfer probability of the UAV between each grid cell based on the ant colony algorithm and the initial observation position of the UAV;

[0189] The reward function building module is used to build a reward function to guide the UAV to explore the area and track the target based on the transition probability and grid cells;

[0190] The joint decision module is used to centrally train multiple drones based on the reward function and the existing exploration and tracking information using a multi-agent proximal strategy optimization algorithm to obtain the optimal joint decision; the optimal joint decision includes the joint action decision of exploration and tracking of multiple targets;

[0191] The exploration and tracking module is used to enable the UAV to complete the exploration and tracking of multiple targets to be explored based on the optimal joint decision.

[0192] A further implementation method is that the transition probability calculation module includes an ant colony algorithm improvement unit for improving the existing ant colony algorithm; the ant colony improved algorithm includes:

[0193] The pheromone division subunit is used to divide pheromones into exploration pheromones and tracking pheromones. Among them, the exploration pheromones are used to guide the UAV to explore unfamiliar environments; the tracking pheromones are used to guide the UAV to approach the grid unit of the explored target and to collaborate in exploring and tracking other surrounding targets.

[0194] The exploration pheromone secretion subunit is used to explore the secretion of pheromones in the grid cells that the drone passes along its movement trajectory before the drone detects the target, and volatilizes over time;

[0195] The tracking pheromone secretion subunit is used to stop the secretion of exploration pheromones after the UAV explores the target, and secrete the tracking pheromone in the corresponding grid unit where the UAV explores the target, and diffuse it to the adjacent grid units within the preset range in a decreasing manner; among them, only one type of pheromone is effective in each grid unit at the same time.

[0196] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-UAV target exploration and tracking method based on multi-agent ant colony reinforcement learning, characterized in that: The following steps are involved: Divide the actual environment where the UAV performs multi-target tracking into grid cells; Based on the grid cells, target state initialization is performed to obtain an initial observation position of the UAV; Based on the ant colony algorithm and the initial observation position of the drone, the transfer probability of the drone between each grid cell is calculated; Based on the transition probability and the grid unit, a reward function is established to guide the UAV to perform area exploration and target tracking; Based on the reward function and the existing exploration information and tracking information, a multi-agent proximal strategy optimization algorithm is used to centrally train multiple drones to obtain an optimal joint decision; wherein the optimal joint decision includes a joint action decision for exploration and tracking of multiple targets; Based on the optimal joint decision, the UAV completes the exploration and tracking of multiple targets to be explored.

2. The method according to claim 1, characterized in that: The target state initialization includes initializing the position of the target at the initial moment, the state transfer matrix, the observation noise, the measurement noise and the covariance matrix.

3. The method according to claim 1, characterized in that The ant colony algorithm is an ant colony algorithm for improving the pheromone application method, and the improvement method includes: Pheromones are divided into exploration pheromones and tracking pheromones. Among them, exploration pheromones are used to guide drones to explore unfamiliar environments; tracking pheromones are used to guide drones to approach the grid cells of explored targets and to collaborate in exploring and tracking other surrounding targets. Before the drone detects the target, the exploration pheromone is secreted in the grid cells along the drone's movement trajectory and evaporates over time; After the UAV explores the target, the exploration pheromone stops being secreted, and the tracking pheromone is secreted in the corresponding grid cell where the UAV explores the target, and diffuses in a decreasing manner to the adjacent grid cells within the preset range; among them, only one type of pheromone is effective in each grid cell at the same time.

4. The method according to claim 3, characterized in that The method for calculating the transfer probability of the UAV between each grid cell includes: Presetting the initial value of the exploration pheromone concentration and the initial value of the tracking pheromone concentration; When the UAV is exploring the current grid cell, the local exploration pheromone concentration of the current grid cell is updated based on the initial value of the exploration pheromone concentration and the exploration pheromone increment left by the current grid cell in each exploration; When the drone discovers a target in the grid cell where it is located, the local tracking pheromone concentration of the current grid cell is updated based on the initial value of the tracking pheromone concentration of the grid cell and the tracking pheromone increment left by the drone tracking the target; Calculate path heuristic information based on the Euclidean distance from the current grid cell to the adjacent grid cells; Based on the local exploration pheromone concentration and the local tracking pheromone concentration of the current grid cell, the local exploration pheromone concentration and the local tracking pheromone concentration of the adjacent grid cell, as well as the path heuristic information and the expected heuristic factor, the transfer probability of the UAV from the current grid cell to the adjacent grid cell is obtained.

5. The method according to claim 1, characterized in that: The reward function includes exploration reward, collision reward, tracking reward and pursuit reward.

6. The method according to claim 3, characterized in that The method for obtaining the optimal joint decision includes: When the drone finds the target, it stops secreting exploration pheromones and starts releasing and spreading tracking pheromones. The drone initializes and estimates the target within the field of view; wherein the initialization includes target state initialization, state transfer matrix initialization and covariance initialization; the estimation includes state estimation and state covariance estimation; When the same target enters the field of view of multiple UAVs, the target trajectory is fused based on the initialization and estimation results of the target to obtain a fused trajectory; A pseudo-measurement equation is established based on the fused trajectory, and multiple UAVs are centrally trained in a multi-agent proximal strategy optimization algorithm based on the reward function and existing exploration information and tracking information. After the training is completed, the optimal joint decision of the multiple UAVs is obtained.

7. A multi-UAV target exploration and tracking system based on multi-agent ant colony reinforcement learning, used to implement the method described in any one of claims 1 to 6, characterized in that: include: A grid division module is used to divide the actual environment of the UAV for multi-target tracking into grid units; A target initialization module is used to initialize the target state based on the grid unit to obtain the initial observation position of the UAV; The transfer probability calculation module is used to calculate the transfer probability of the UAV between each grid cell based on the ant colony algorithm and the initial observation position of the UAV; A reward function establishment module, used to establish a reward function for guiding the UAV to perform area exploration and target tracking based on the transition probability and the grid unit; A joint decision module is used to centrally train multiple UAVs using a multi-agent proximal strategy optimization algorithm based on the reward function and existing exploration information and tracking information to obtain an optimal joint decision; wherein the optimal joint decision includes a joint action decision for exploration and tracking of multiple targets; The exploration and tracking module is used to enable the UAV to complete the exploration and tracking of multiple targets to be explored based on the optimal joint decision.

8. The system according to claim 7, characterized in that The transition probability calculation module includes an ant colony algorithm improvement unit, which is used to improve the existing ant colony algorithm; The ant colony improved algorithm comprises: The pheromone division subunit is used to divide pheromones into exploration pheromones and tracking pheromones. Among them, the exploration pheromones are used to guide the UAV to explore unfamiliar environments; the tracking pheromones are used to guide the UAV to approach the grid unit of the explored target and to collaborate in exploring and tracking other surrounding targets. The exploration pheromone secretion subunit is used to explore the secretion of pheromones in the grid cells that the drone passes along its movement trajectory before the drone detects the target, and volatilizes over time; The tracking pheromone secretion subunit is used to stop the secretion of exploration pheromones after the UAV explores the target, and secrete the tracking pheromone in the corresponding grid unit where the UAV explores the target, and diffuse it to the adjacent grid units within the preset range in a decreasing manner; among them, only one type of pheromone is effective in each grid unit at the same time.

Images