Multi-unmanned aerial vehicle collaborative search method imitating bird territory mark

Abstract

The invention discloses a multi-UAV (unmanned aerial vehicle) collaborative search method imitating bird territory marks. The method comprises the following implementation steps: step 1, initializing a search environment; 2, unmanned aerial vehicle action strategy selection and position updating; 3, updating a search map and a territory smell map; 4, calculating a reward function and an observation matrix; 5, updating the reinforcement learning strategy network; and 6, outputting a search result. The method aims at solving the problem that multiple unmanned aerial vehicles search dynamic targets in an unknown environment, dynamic coverage of a target search area can be achieved in combination with a bionic model marked by a bird territory and an MAPPO reinforcement learning algorithm, efficient and intelligent search of multiple unmanned aerial vehicle dynamic targets is achieved, and the cooperation ability between the unmanned aerial vehicles in the search process is improved.

Description

Technical Field

[0001] The invention discloses a multi-UAV collaborative search method imitating bird territory marking, belonging to the field of autonomous control of UAVs. Background Art

[0002] With the development of drone technology, multi-drone systems have shown great potential in various applications, especially in dynamic target search. The collaborative work of multi-drone systems can significantly improve search efficiency and accuracy. A multi-drone system refers to a collaborative working system composed of multiple drones, which share information and assign tasks through a communication network. The advantage of a multi-drone system is that it can cover a larger search area while providing redundancy and reliability in target detection and tracking. Dynamic target search involves identifying and tracking moving targets in a constantly changing environment. The difficulty lies in the unpredictable target path and the target may appear in an area that has been searched, so drones are required to cover the search area multiple times.

[0003] Therefore, how to solve the problem of multiple UAVs searching for dynamic targets in an unknown environment, so as to intelligently partition and dynamically cover the search area and complete a more efficient and intelligent search task for the target area by the multi-UAV system, is an urgent problem to be solved. Summary of the invention

[0004] 1. Purpose of the invention:

[0005] The present invention proposes a multi-UAV collaborative search method that imitates bird territory marking, aiming to solve the problem of multiple UAVs searching for dynamic targets in an unknown environment. By combining the bionic model of bird territory marking and the MAPPO reinforcement learning method, the search area is intelligently partitioned and dynamically covered, so that the multi-UAV system can achieve more efficient and intelligent search for the target area.

[0006] The present invention combines bird territory marking with multi-UAV search to improve the search efficiency of UAVs. Bird territory marking is an animal behavior through which birds can declare and protect their own territory. The three signal forms of bird territory marking - chemical, auditory and visual - can all be used to declare territory. This type of territory marking is narrowly used only for olfactory marking behavior, which means that birds leave chemical traces such as urine, feces or glandular secretions, which continue to be valid after leaving the scene. In a broad sense, it refers to both olfactory marking and auditory and visual marking. The present invention considers imitating the characteristics of birds marking their territory through hormones or body fluids to conduct multi-UAV collaborative search.

[0007] When combining the behavior of bird territory marking for drone cluster search, it is necessary to model the odor diffusion mechanism of birds. The application of odor diffusion models on raster maps usually involves how to simulate and predict the propagation and dissipation of odors in space. Raster maps are a method of dividing geographic space into regular grid cells and assigning specific values ​​to each cell to represent spatial information. In the context of odor diffusion, raster maps can be used to represent odor concentrations in different areas. The following are some key points that describe how odor diffusion models are implemented on raster maps:

[0008] Diffusion direction: Since the present invention only considers the bionic significance of the odor diffusion mechanism in drone search, the diffusion direction is set to diffuse in all directions. During the modeling process, the grid with high "odor" concentration is evenly diffused to the grids in the four directions around it.

[0009] Diffusion rate: The diffusion rate is described using Fick's first law, which was proposed by physiologist Adolf Fick in 1855 to describe the diffusion phenomenon of substances in space due to concentration gradients. Fick's first law describes that under steady-state diffusion conditions, the flow rate (diffusion flux) of the diffusing substance per unit cross-sectional area perpendicular to the diffusion direction per unit time is proportional to the concentration gradient at the cross-section.

[0010] Territory establishment: Each drone or drone group is set to leave a scent on the path during the search process, and the scent continues to spread with the event. Each group (frame) of drones spreads a different scent, so multiple drone scent diffusion maps will be formed, and the territory of each grid belongs to the drone group (individual) corresponding to the strongest scent in the grid area. When the grid determines the ownership, it has an exclusion effect on drones that have not occupied the grid.

[0011] This paper combines the MAPPO (Multi-Agent Proximal Policy Optimization) multi-agent reinforcement learning method for multi-UAV collaborative search tasks. MAPPO optimizes the strategy of each agent through collaborative learning of multiple agents, improving the search efficiency and accuracy of the entire system.

[0012] In the MAPPO framework, each drone acts as an intelligent agent and learns the optimal strategy through interaction with the environment. Intelligent agents work together by sharing information. It adopts the proximal policy optimization (PPO) method to ensure the stability and efficiency of policy updates by limiting the amplitude of policy updates. PPO uses a trust region mechanism to prevent performance degradation caused by excessive policy updates. Through the collaborative learning of multiple intelligent agents, efficient search for dynamic targets is achieved. Intelligent agents improve the overall search effect by sharing experience and strategies.

[0013] The operation process of the MAPPO method in the present invention is as follows Figure 1 As shown in the figure, by establishing a rasterized map as a shared environment, each drone corresponds to an intelligent agent, and the odor map is shared with each other. The drone position is updated through the action strategy, and then the odor map of each drone is updated, so as to allocate the territory, and the observation matrix is ​​obtained based on the territory allocation result and the shared odor map and returned.

[0014] 2. Technical solution:

[0015] The present invention provides a multi-UAV collaborative search method that simulates bird territory marking, and the implementation process is as follows: Figure 2 As shown, the specific implementation steps are as follows:

[0016] Step 1: Initialize the search environment

[0017] S11. Initialize the search environment space and search target information:

[0018] Considering that the present invention uses a reinforcement learning technology based on discrete decision making, the search area needs to be rasterized first. The initialized raster map is an n×n grid. The map part includes k drone odor layers as an observation matrix to record the odor diffusion results of each drone, and an original map layer to record the search results and the covered area. The initial values ​​of all layers are set to 0.

[0019] S12, UAV and target initialization:

[0020] Initially, the position of the i-th UAV is randomly set to (x i ,y i ), and when setting, it should be determined whether there is overlap. If the initial position of the drone overlaps, it should be regenerated. The position of the jth target is randomly set to (x j ,y j ), considering the characteristics of dynamic targets, the moving strategy of dynamic targets is set to random movement, and it moves in a random direction once in each time step.

[0021] Step 2: Drone action strategy selection and position update

[0022] S21. Drone action strategy selection:

[0023] There are four action strategies for drones, divided into up, down, left, and right. Each time an action is executed, the drone's grid map position will be updated. Based on the observations obtained by each drone, the policy network generated by reinforcement learning generates the predicted benefits of executing four actions in the current state, and then calculates the probability distribution of the four actions, and samples an action from the action probability distribution.

[0024] S22, UAV position update:

[0025] Update the position of the drone according to the action selected in step S21, ensure that each action is effectively moved within the boundaries of the map, and determine whether the drone will exceed the boundaries or collide with obstacles after executing the action. If the movement will cause a collision, the movement command is rejected and the collision mark is returned. The drone stays in place and refuses to move, which is deducted when the reward is finally calculated. If there is no collision, the position is updated according to the normal action command, and the position is (x i ,y i ) is updated to (x i ,y i +1), and after performing the right shift, it is updated to (x i +1,y i ).

[0026] Step 3: Search map and territory scent map update

[0027] S31. Search map updates:

[0028] After obtaining the updated list of drone locations, the search map is updated. The search map records the number of times all locations have been traversed by all drones, which is used to intuitively demonstrate the superiority of the method in terms of dynamic coverage of the search area.

[0029] S32, Territory scent map update:

[0030] The territory scent map update is divided into three parts: drone scent map update, drone territory map update and drone observation scent map update.

[0031] First, we need to calculate the odor map at time t+1 based on the position update result of the drone. Assume that the odor map record of the kth drone at time t is This is an n×n matrix. First, we need to calculate the result of the odor map diffusion at time t according to Fick's first law. The mathematical expression of the odor map diffusion is written as:

[0032]

[0033] Where J is the diffusion flux, D is the diffusion coefficient, is the concentration gradient, and the negative sign indicates that the substance always diffuses from the high concentration area to the low concentration area. The specific calculation method is to traverse each grid once, calculate the odor diffusion flux of the grid and its adjacent grids within a step, and then calculate the change in odor concentration of the current grid:

[0034]

[0035] Assuming that the adjacent grid is outside the area, the odor concentration of the adjacent grid is set to 0, so that the odor concentration of the search map maintains a dynamic balance. If the odor diffuses too quickly, the drone will often review the explored area, while if the odor diffuses too slowly, the drone will no longer update the detection data for the explored area, and the efficiency of the drone's territory division will also be slow. Therefore, the diffusion rate should be set reasonably to meet the following formula:

[0036]

[0037] Here, the diffusion coefficient K smell is set to This coefficient ensures that when the average concentration in each grid is p, the odor concentration generated by the drone is the same as the total amount of odor diffused to the outside world from the entire map, achieving a dynamic balance for the final late saturation detection.

[0038] After traversal, the calculated diffusion results are recorded as matrix The odor concentration matrix can be updated:

[0039]

[0040] Here represents the position of the drone at time t+1, and adds 1 to the corresponding odor matrix position. The odor update map is as follows: Figure 3 shown.

[0041] Step 4: Reward function and observation matrix calculation

[0042] S41. Territory map calculation:

[0043] Since the drone odor map has been calculated in step 3, a territory distribution map can be obtained according to the values ​​of each odor map. The calculation method of the territory distribution map is as follows:

[0044]

[0045] According to the territory distribution map, we can know that by comparing the odor concentrations of different drones in the same territory, the territory is assigned to the drone with the strongest odor. For unoccupied territories, their territory ownership is defined as 0. Since the higher the odor concentration of the territory in the reward function, the greater the reward, drones have the greatest willingness to move to unowned areas.

[0046] S42, observation matrix calculation:

[0047] In order to allow the reinforcement learning function to learn as much information as possible and converge as quickly as possible, the observation matrix is ​​divided into three parts. The first part is the position of the current agent, the second part is the position of the obstacle, and the third part is the observation matrix of the current agent. Each part is represented by an n×n matrix. Finally, the three matrices are connected in series to obtain the observation matrix.

[0048] S43, reward function calculation:

[0049] When calculating the reward function, the following points need to be considered:

[0050] 1. The territories corresponding to each drone are of similar size, thus ensuring that the search coverage times of each drone’s corresponding territory are synchronized.

[0051] 2. The search coverage times of each drone in the corresponding territory are similar.

[0052] 3. Determine whether the drone performs invalid operations, such as colliding with boundaries or obstacles.

[0053] 4. Determine whether the total odor map is uniform, which is used to generally describe the uniform coverage of the search area.

[0054]

[0055] Among them, area(k) represents the area of ​​the territory covered by the kth individual, q1, q2, and q3 are the reward coefficients corresponding to each reward function, and μ(i) represents the odor concentration of the i-th area. In the calculation of the reward value of the search individual, the reward value consists of three parts. The first part determines the balance of the coverage of the area by obtaining the variance of the individual's own odor concentration within the territory. This value needs to take a negative sign, because obviously the smaller the variance, the more balanced the coverage; the second part determines the score by judging the proportion of the size of the territory to all the covered areas. In order to ensure that the tasks of each drone are evenly distributed, this coefficient is larger when the territory is closer to the average value; the third part is to judge whether the current action is successful. When the action collides with the boundary or obstacle, the drone will stay in place, and the individual needs to be punished here.

[0056] Step 5: Reinforcement learning policy network update

[0057] The maximum number of steps for each round of training is set to n steps. After executing action a in state s, the reward r and the next state s′ are obtained. The policy network can be updated based on this result.

[0058] We need to calculate the advantage function for the state-action pair, which measures the additional value of each action relative to its expected reward. This can be calculated using the following function:

[0059]

[0060] Among them, Q(s,a) refers to the actual reward value obtained by performing action a in the current state S, and updates the critic network.

[0061] Step 6: Output search results

[0062] Animated display of the searched scent map and territory map.

[0063] The invention discloses a multi-UAV collaborative search method that imitates bird territory marking, which has the advantages and functions of: by combining the bionic model of bird territory marking and the MAPPO reinforcement learning method, dynamic coverage of the target search area can be achieved, efficient and intelligent search of dynamic targets of multiple UAVs can be achieved, and the collaboration ability between UAVs during the search process can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is an update diagram of the MAPPO method proposed in the present invention.

[0065] Figure 2 It is a flowchart of multi-UAV collaborative search that imitates bird territory marking.

[0066] Figure 3 It is a drone smell territory division map.

[0067] Figure 4 It's a drone search for smell maps.

[0068] Figure 5 is the reinforcement learning evolution curve. DETAILED DESCRIPTION

[0069] The feasibility and effectiveness of the multi-UAV collaborative search method for bird territory marking invented in this patent are verified through specific examples. Figure 2 The flowchart of multi-UAV collaborative search simulating bird territory marking is used for example simulation to verify the rationality and effectiveness of the multi-UAV collaborative search method simulating bird territory marking proposed in the present invention. The simulation environment of this example is configured with an Intel i7-12700 processor, 3.40Ghz main frequency, 16G memory, and the dependencies are as follows:

[0070] Operating system: Windows 10

[0071] Development environment: Pycharm 2024.1

[0072] Programming language: Python 3.8

[0073] Dependent libraries: numpy, torch, setproctitle, gym

[0074] Compilation software: pycharm2024.1

[0075] The specific steps of this method are as follows:

[0076] Step 1: Initialize the search environment

[0077] S11. Initialize the search environment space and search target information:

[0078] The initialized grid map is a 10×10 grid. The map part includes three drone odor layers as observation matrices to record the odor diffusion results of each drone, and a 10×10 original map layer to record the search results and covered areas. The initial values ​​of all layers are set to 0.

[0079] S12, UAV and target initialization:

[0080] The position of each drone is randomly set to (2,3), (5,7), (8,6) to ensure that the initial positions of the drones do not overlap. If the initial positions of the drones overlap, they should be regenerated. The target position is randomly set to (10,10), and the target moves once per time step. Considering the characteristics of dynamic targets, the movement strategy of dynamic targets is set to random movement. Each time step moves in a random direction once. If the moving direction collides with the edge of the map, it moves in the opposite direction.

[0081] Step 2: Drone action strategy selection and position update

[0082] S21. Drone action strategy selection:

[0083] The action space of the drone includes four basic actions: moving up, moving down, moving left, and moving right. Based on the observations obtained by each drone, the policy network generated by reinforcement learning generates the predicted benefits of executing the four actions in the current state, and then calculates the probability distribution of the four actions, and samples an action from the action probability distribution generated by the policy network. For example, the probabilities of the up, down, left, and right actions are calculated as 0.2, 0.3, 0.4, and 0.1, respectively, and an action is sampled from them.

[0084] S22, UAV position update:

[0085] Update the position of the drone according to the action selected by the agent, ensuring that each action is a valid move within the boundaries of the map. The updated position needs to be verified to see if it conflicts with the edge of the map. If it does, adjust it to move in the opposite direction. If the move exceeds the boundary or collides with an obstacle, the move is rejected and the collision flag is recorded, and the reward is deducted. For example, a drone at position (3,10) will be rejected if it performs a right move.

[0086] After the action is successfully executed, the drone position is updated. For example, a drone at position (2,3) is updated to (1,3) after moving up, and to (2,4) after moving right.

[0087] Step 3: Search map and territory scent map update

[0088] S31. Search map updates:

[0089] After obtaining the updated list of drone positions, the search map is updated. The search map records the area covered by each drone and updates the covered grid at each step. If the drone at position (2,3) moves to (1,3), the count of position (1,3) is increased by 1, and the number of visits to this grid is recorded.

[0090] S32, Territory scent map update:

[0091] The territory scent map update is divided into three parts: drone scent map update, drone territory map update and drone observation scent map update.

[0092] First, we need to calculate the odor map at time t+1 based on the position update results of the drone. Assume that the odor map record of the first drone at time t is This is a 10×10 matrix. First, we need to calculate the result of the odor map diffusion at time t according to Fick's first law.

[0093] Here, the diffusion coefficient K in the formula smell Set to 0.01, this coefficient ensures that when the average concentration in each grid is 7.25, the concentration of odor generated by the drone is the same as the total amount of odor diffused to the outside world from the entire map, achieving a dynamic balance of the final late saturation detection. For example, if the odor concentration of the current grid is 10, and the concentrations of the four grids around it are all 0, then after calculation, the odor concentration of the current grid is 9.6, and it evenly diffuses 0.1 of odor to the surrounding grids. In particular, when the grid is at the boundary, the odor of the part beyond the boundary defaults to 0.

[0094] After traversal, the calculated diffusion results can be used to update the odor concentration matrix by adding 1 to the corresponding odor matrix position.

[0095] Step 4: Reward function and observation matrix calculation

[0096] S41. Observation matrix calculation:

[0097] Since the drone odor map has been calculated in step 3, the territory map can be calculated based on the values ​​of each odor map to divide the territory to the drone with the strongest odor. For unoccupied territories, their territory ownership is defined as 0. Since the higher the odor concentration of the territory in the reward function, the greater the reward, the drone has the greatest willingness to move to the unowned area.

[0098] S42, reward function calculation:

[0099] The reward value is calculated through observations in three parts. The first part determines the balance of coverage of the area by obtaining the variance of the individual's own odor concentration within the territory. This value needs to have a negative sign, because obviously the smaller the variance, the more balanced the coverage; the second part determines the score by judging the proportion of the size of the territory to all covered areas. In order to ensure that the tasks of each drone are evenly distributed, this coefficient increases when the territory is closer to the average value; the third part determines whether the current action is successful. When the action collides with a boundary or obstacle, the drone will stay where it is, and the individual needs to be punished here.

[0100] Step 5: Reinforcement learning policy network update

[0101] The maximum number of steps for each round of training is set to 50. After executing action a in state s, the reward r and the next state s′ are obtained. The policy network can be updated based on this result.

[0102] Step 6: Output search results

[0103] The searched odor map and territory map are displayed dynamically. The odor territory division map of a drone in a search operation is shown below: Figure 3 As shown, the drone searches for odor maps such as Figure 4 shown.

Claims

1. A multi-UAV collaborative search method that simulates bird territory marking, characterized by: The steps of this method are as follows: Step 1: Initialize the search environment S11. Initialize the search environment space and search target information: The initialized grid map is an n×n grid. The map part includes k drone odor layers as an observation matrix to record the odor diffusion results of each drone, and an original map layer to record the search results and the covered area; S12, UAV and target initialization: Initially, the position of the i-th UAV is randomly set to (x i ,y i ), the position of the jth target is randomly set to (x j ,y j ), considering the characteristics of dynamic targets, the moving strategy of dynamic targets is set to random movement, moving in a random direction once in each time step; Step 2: Drone action strategy selection and position update S21. Drone action strategy selection: There are four action strategies for drones, divided into up, down, left and right. Each time an action is executed, the drone's grid map position will be updated. Based on the observations obtained by each drone, the policy network generated by reinforcement learning generates the predicted benefits of executing four actions in the current state, and then calculates the probability distribution of the four actions, and samples an action from the action probability distribution generated by the policy network. S22, UAV position update: Update the position of the drone according to the action selected in step S21, ensure that each action is effectively moved within the boundary of the map, and determine whether the drone will exceed the boundary or collide with an obstacle after executing the action; If the movement will cause a collision, the movement command is rejected and the collision mark is returned. The drone stays in place and refuses to move, which is deducted when the reward is finally calculated. If there is no collision, the position is updated according to the normal action command, and the position is (x i ,y i ) is updated to (x i ,y i +1), and after performing the right shift, it is updated to (x i +1,y i ); Step 3: Search map and territory scent map update S31. Search map updates: S32, Territory scent map update: First, we need to calculate the odor map at time t+1 based on the position update result of the drone; assuming that the odor map record of the kth drone at time t is This is an n×n matrix. First, we need to calculate the result of the odor map diffusion at time t according to Fick's first law. The mathematical expression of the odor map diffusion is written as: Where J is the diffusion flux, D is the diffusion coefficient, is the concentration gradient, and the negative sign indicates that the substance always diffuses from the high concentration area to the low concentration area. The specific calculation is to traverse each grid once, calculate the odor diffusion flux of the grid and its adjacent grids within a step, and then calculate the change in odor concentration of the current grid: Assuming that the adjacent grid is outside the area, the odor concentration of the adjacent grid is set to 0, so that the odor concentration of the search map maintains a dynamic balance; Set the diffusion rate to meet the following formula: After traversal, the calculated diffusion results are recorded as matrix Update the odor concentration matrix: Here represents the position of the drone at time t+1, and adds 1 to the corresponding odor matrix position; Step 4: Reward function and observation matrix calculation S41. Territory map calculation: According to the values ​​of each odor map, a territory distribution map is obtained. The calculation of the territory distribution map is as follows: According to the territory distribution map, by comparing the odor concentrations of different drones in the same territory, the territory is assigned to the drone with the strongest odor. For unoccupied territories, their territory ownership is defined as 0. Since the higher the odor concentration of the territory in the reward function, the greater the reward, drones have the greatest willingness to move to unowned areas. S42, observation matrix calculation: The observation matrix is ​​divided into three parts. The first part is the position of the current agent, the second part is the position of the obstacle, and the third part is the observation matrix of the current agent. Each part is represented by an n×n matrix. Finally, the three matrices are connected in series to obtain the observation matrix. S43, reward function calculation: Step 5: Reinforcement learning policy network update The maximum number of steps for each round of training is set to n steps. After executing action a in state s, the reward r and the next state s′ are obtained. The policy network is updated based on this result. Step 6: Output search results.

2. The multi-UAV collaborative search method for bird territory marking according to claim 1, characterized in that: In the process of calculating the reward function in step S43, the following points need to be considered: 1) The size of the territory corresponding to each drone is similar, so as to ensure that the search coverage times of each drone’s corresponding territory are synchronized; 2) The search coverage times of each drone in the corresponding territory are similar; 3) Determine whether the drone performs invalid operations, such as colliding with boundaries or obstacles; 4) Determine whether the total odor map is uniform, which is used to generally describe the uniform coverage of the search area; Among them, area(k) represents the area of ​​the territory covered by the kth individual, q1, q2, q3 are the reward coefficients corresponding to each reward function, and μ(i) represents the odor concentration of the i-th area.

3. The multi-UAV collaborative search method for bird territory marking according to claim 2, characterized in that: In the reward function calculation of the search individual, the reward value consists of three parts. The first part determines the balance of coverage of the area by obtaining the variance of the individual's own odor concentration within the territory. This value needs to take a negative sign; the second part determines the score by judging the proportion of the size of the territory to all covered areas. In order to ensure that the tasks of each drone are evenly distributed, this coefficient increases when the territory is closer to the average value; the third part determines whether the current action is successful. When the action collides with a boundary or obstacle, the drone will stay where it is, and the individual needs to be punished here.

4. The multi-UAV collaborative search method for bird territory marking according to claim 1, characterized in that: The fifth step of the reinforcement learning strategy network update requires calculating the advantage function of the state-action pair. The advantage function is used to measure the additional value of each action relative to its expected return. The calculation is performed by the following function: Among them, Q(s,a) refers to the actual reward value obtained by performing action a in the current state S, and updates the critic network.

Images