A multi-UAV collaborative inspection method and system based on reinforcement learning

Abstract

The present invention discloses a multi-UAV collaborative inspection method and system based on reinforcement learning, including S1: collecting map information of the area to be inspected, and rasterizing the map to obtain raster map data; S2: constructing an environmental simulator, simulating the movement of suspicious targets, and obtaining suspicious target movement data; S3: calculating the number of UAVs according to the area of ​​the area to be inspected and the expected inspection time, and dividing the initial inspection area in combination with a clustering algorithm; S4: judging whether to collaboratively track suspicious targets according to the real-time inspection situation, and constructing a regional redivision model to redivision the uninspected area; S5: constructing a collaborative inspection model to control UAVs to collaboratively track suspicious targets. The present invention constructs a suspicious target path simulator and trains a collaborative inspection model, which solves the problem that the traditional method of patrol area division is rigid, cannot be dynamically adjusted according to actual conditions, and is difficult to give full play to the collaborative advantages of multiple UAVs.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a multi-UAV collaborative inspection method and system based on reinforcement learning. Background Art

[0002] Multi-UAV collaborative inspection refers to the use of multiple UAVs to share information, allocate tasks and collaboratively control to jointly complete inspection tasks in specific areas. It aims to improve the efficiency of UAV inspections, expand the scope of inspections, and maintain efficient monitoring capabilities in complex environments.

[0003] Although traditional drone inspection technology has improved inspection efficiency to a certain extent, it still has many shortcomings, especially in terms of multi-drone collaboration. First, traditional drone inspections often use a static area division method, that is, the inspection area is divided into several fixed sub-areas based on maps or expert experience in advance, and then these sub-areas are assigned to each drone for corresponding inspections. This method ignores various dynamic changes that may be encountered during the actual inspection process, such as the movement of suspicious targets, emergencies, etc., resulting in low inspection efficiency; secondly, when a sub-area needs to be closely tracked or inspected, the traditional method usually requires temporary adjustments to the inspection routes and areas of individual drones, and then the remaining uninspected areas are re-planned and assigned to other drones. The adjustment process is complicated and cannot give full play to the advantages of multi-drone collaboration. Summary of the invention

[0004] In view of this, the present invention aims to provide a multi-UAV collaborative inspection method and system based on reinforcement learning, aiming to solve the problem that the traditional method of patrol area division is rigid, cannot be dynamically adjusted according to actual conditions, and is difficult to give full play to the advantages of multi-UAV collaboration.

[0005] A multi-UAV collaborative inspection method based on reinforcement learning includes the following steps:

[0006] S1: Collect map information of the area to be inspected and rasterize the map to obtain raster map data;

[0007] S2: Build an environment simulator to simulate the movement of suspicious targets and obtain the movement data of suspicious targets;

[0008] S3: Calculate the number of drones based on the area to be inspected and the expected inspection time, and divide the initial inspection area using a clustering algorithm;

[0009] S4: Based on the real-time inspection situation, determine whether to conduct collaborative tracking of suspicious targets, and build a regional redivision model to redivision the uninspected areas;

[0010] S5: Build a collaborative patrol model to control drones to collaboratively track suspicious targets;

[0011] Furthermore, the step S1 collects map information of the area to be inspected and rasterizes the map to obtain raster map data, including:

[0012] S11: The map information of the area to be inspected includes the geographical elements, area and boundaries of the map of the area to be inspected;

[0013] S12: preprocessing the map data of the inspection area, including removing noise, smoothing boundaries, and filling missing parts to obtain preprocessed map data;

[0014] S13: Using an eight-directional rasterization algorithm, the preprocessed map data is converted into raster map data; in the raster map data, each grid contains a unique grid coordinate; the area corresponding to each grid in the rasterization is the coverage area of ​​a single frame video when a drone is patrolling.

[0015] Furthermore, the step S2 constructs an environment simulator to simulate the movement of the suspicious target to obtain the movement data of the suspicious target, including:

[0016] S21: Build an environment simulator, the calculation method is:

[0017] {x t}=concat(x 1 ,x 2 ,…,x t )

[0018] V t =VAE({x t})

[0019] h t =LSTM(h t-1 ,x t ,V t-1 )

[0020] Among them, {x t} is the motion trajectory of the suspicious target at time t, concat() is the concatenation operation, x t is the grid coordinate of the suspicious target at time t, V t is the trajectory feature of the suspicious target at time t, VAE() is the variational autoencoder operation, and h t is the suspicious target location feature at time t, LSTM() is the long short-term memory network operation, h t-1 is the suspicious target location feature at time t-1, V t-1 is the trajectory feature of the suspicious target at time t-1, and t is the time index;

[0021] S22: using an environmental simulator to simulate the movement of the suspicious target, and recording the grid coordinates of the suspicious target at each moment, to obtain the movement data of the suspicious target;

[0022] In step S21, an environmental simulator is constructed by combining a variational autoencoder (VAE) and a long short-term memory network (LSTM). The powerful feature extraction and representation learning capabilities of VAE can extract the pattern of trajectory changes from trajectory information. Based on long-term dependency learning and time series prediction capabilities, LSTM simulates the future position of suspicious targets through historical trajectories and current positions, so that the environmental simulator can accurately simulate the movement data of suspicious targets and provide rich training samples for the subsequent collaborative inspection model based on reinforcement learning.

[0023] Compared with traditional methods that only rely on mathematical models and statistics, which are difficult to accurately capture the complex motion patterns of the target, the environmental simulator of the present invention helps the subsequent collaborative patrol model to learn collaborative strategies in more complex environments, thereby improving the collaboration efficiency between drones and further enhancing the overall patrol capability.

[0024] Furthermore, the step S3 calculates the number of drones according to the area of ​​the area to be inspected and the expected inspection time, and divides the initial inspection area in combination with the clustering algorithm, including:

[0025] S31: Calculate the number of drones based on the area to be inspected and the expected inspection time. The calculation method is:

[0026] n=MLP(S,T)

[0027] Where n is the number of drones, MLP() is a fully connected neural network operation, S is the area to be inspected, and T is the expected inspection time;

[0028] S32: Starting from the center point of the area to be inspected, draw n rays with uniform angle distribution, and the angle interval of each ray is Get n sub-regions;

[0029] S33: The center points of each sub-area divided by the ray are used as the initial clustering centers of the clustering algorithm, and the K-means clustering algorithm is used to cluster the grids of the inspection area. The number of clusters is still n, and the clustering center is allowed to move during the clustering process;

[0030] S34: The clustering result is used as the initial patrol area of ​​each UAV, and each grid in the initial patrol area is assigned the area category of the area;

[0031] In step S3, a fully connected neural network is used to calculate the required number of drones based on the area of ​​the area to be inspected and the expected inspection time. Then, the area to be inspected is divided into the initial inspection area for each drone through ray partitioning and K-means clustering algorithm, paving the way for subsequent area redivision. At the same time, the number of drones can also be manually set according to actual conditions, and the system will perform initial inspection area division based on the manually set number of drones.

[0032] Furthermore, the step S4 determines whether to collaboratively track the suspicious target based on the real-time inspection situation, and constructs a regional redivision model to redivision the uninspected area, including:

[0033] S41: According to the suspicious target movement data generated by the environment simulator, when a suspicious target appears in a certain area, the UAV that discovers the suspicious target is controlled to track the suspicious target;

[0034] S42: Calculate the Euclidean distances between all the remaining drones and the suspicious target except the drone that discovered the suspicious target and sort them to obtain a drone distance sequence L, and obtain the grid coordinates of the nearest drone;

[0035] S43: According to the suspicious target category, determine whether to send the nearest UAV to track the suspicious target collaboratively. The calculation method is:

[0036] If the suspicious target is a Class I or Class II target, the nearest UAV will be dispatched to collaboratively track the suspicious target;

[0037] If the target is a Class III target, the nearest UAV will not be sent to conduct collaborative tracking of the suspicious target;

[0038] The Class I targets include: trucks, motorcycles; the Class II targets include: pedestrians, cars, off-road vehicles; the Class III targets include: bicycles, drones;

[0039] S44: If the nearest UAV is dispatched to collaboratively track the suspicious target, a regional redivision model is constructed for the unpatrolled area of ​​the UAV that discovered the suspicious target and the nearest UAV, and the unpatrolled area is redivisioned. The calculation method is:

[0040] C = softmax(MLP(L,s))

[0041] Among them, C is the area category corresponding to the patrol area after regional redivision, softmax() is the softmax function, L is the drone distance sequence, and s is the grid coordinate of the center of the unpatrolled area;

[0042] Compared with traditional methods, traditional methods usually rely on manual or fixed rules to divide areas, and temporarily adjust the inspection plan when suspicious targets appear, which can easily lead to inspection blind spots or repeated inspections, and it is difficult to respond to changes in suspicious targets in real time; the present invention can dynamically adjust and optimize the inspection area of ​​the drone in real time according to the appearance of suspicious targets and the tracking of the drone through area redivision, so that uninspected areas can be taken over by other drones more quickly, ensuring that drones can cover uninspected areas more effectively, reducing inspection blind spots, and enhancing the collaborative operation capabilities between drones.

[0043] Furthermore, the step S5 constructs a collaborative inspection model to control the drone to collaboratively track the suspicious target, including:

[0044] S51: Construct a collaborative patrol model, and define the state space of the collaborative patrol model as the current position and power of the drone that discovers the suspicious target, and the current position and power of the nearest drone; the action space is the grid coordinates of the adjacent positions of the drone; the value function is the Q-learning value function;

[0045] S52: Define the reward function of the collaborative patrol model, which is calculated as follows:

[0046] R=R track +R collaboration +R battery

[0047] Among them, R is the reward function of the collaborative inspection model, R track is the tracking function, R collaboration is the coordination function, R battery is the battery consumption function;

[0048] R track =α×m continuous -β×d grid

[0049] Among them, α is the continuous tracking coefficient, m continuous is the number of grids that the UAV continuously tracks to find suspicious targets, β is the distance penalty coefficient, d grid The grid distance between the UAV that detects the suspicious target and the suspicious target;

[0050]

[0051] Among them, δ is the synergy reward coefficient, d between is the grid distance between the two UAVs, θ is the maximum grid distance between the two UAVs, φ is the cooperative angle between the two UAVs formed with the current position of the suspicious target as the vertex, and φ min is the lower limit of the cooperative angle, φ max is the upper limit of the synergy angle;

[0052]

[0053] Among them, ε is the power coefficient, b is the current power of the drone, and battery is the drone power threshold;

[0054] S53: Based on the defined state space, action space, value function and reward function, the Q table is initialized and the suspicious target movement data is imported. Finally, the collaborative inspection model is trained using the Q-learning method.

[0055] S54: According to the trained collaborative inspection model, the UAV that discovers the suspicious target and the nearest UAV are controlled to track the suspicious target;

[0056] The reward function proposed by this invention is track By giving rewards for continuous tracking and punishing behaviors that are too far away from the target; when the drone is able to continuously track the target, that is, the number of grids continuously tracked increases, it will receive a positive reward, which is measured by the continuous tracking coefficient α; at the same time, in order to punish behaviors that are too far away from the target, the present invention introduces a distance penalty coefficient β, when the grid distance between the drone and the suspicious target increases, it will receive a negative reward, so as to ensure that the drone continuously monitors the target during the tracking process;

[0057] In order to optimize the effect of collaborative tracking, the present invention uses R collaboration Partially encourage the coordinated UAVs to maintain an appropriate distance and coordination angle; when the distance between the two UAVs is d between Less than or equal to the maximum grid distance θ, and the cooperative angle φ between them is within the set lower limit φ min and the upper limit φ max When the distance between the two drones is 0.04, they will receive a collaborative reward. The size of the reward is measured by the collaborative reward coefficient δ, which avoids the drones from being too close or too far away, as well as the inappropriate angle, thereby optimizing the collaborative tracking effect.

[0058] The present invention also passes R battery The power consumption of the drone is partially taken into account; when the current battery b of the drone is lower than the set battery threshold battery, it will receive a negative reward. The negative reward is measured by the battery coefficient ε, which aims to avoid the situation where the drone runs out of power and cannot return during the tracking process.

[0059] Compared with the prior art, the advantages of the present invention are:

[0060] (1) In terms of the overall solution, the present invention introduces a regional redivision mechanism, which effectively solves the problem that the patrol area division in the traditional method is rigid and cannot be flexibly adjusted according to the actual situation. At the same time, the present invention constructs a collaborative patrol model, which can trigger multi-UAV collaborative patrols when suspicious targets are found, thereby improving patrol efficiency and accuracy.

[0061] (2) The present invention combines the generation capability of the variational autoencoder with the temporal modeling capability of the long short-term memory network to construct an environmental simulator that can effectively simulate the motion trajectory of suspicious targets in an actual patrol environment, providing rich training samples for subsequent collaborative patrol models based on reinforcement learning.

[0062] (3) The present invention proposes an initial patrol area and area redivision method based on a grid map, which can dynamically adjust and optimize the patrol area of ​​the drone according to the appearance of suspicious targets and the tracking status of the drone, so that the unpatrolled area can be taken over by other drones more quickly, ensuring that the drone can cover the unpatrolled area more effectively, reducing patrol blind spots, and enhancing the collaborative operation capability between drones.

[0063] (4) The present invention introduces the idea of ​​reinforcement learning to construct a collaborative patrol model and proposes a new reward function. It comprehensively considers the continuous tracking effect of drones, the distance and angle between drones during collaborative patrols, and the power of drones, thereby realizing intelligent collaboration and cooperation among multiple drones. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A schematic diagram of the process of multi-UAV collaborative inspection based on reinforcement learning provided by the present invention;

[0065] Figure 2 A schematic diagram of the algorithm flow of the environment simulator provided by the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0067] Example 1: A multi-UAV collaborative inspection method based on reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0068] S1: Collect map information of the area to be inspected and rasterize the map to obtain raster map data;

[0069] S11: The map information of the area to be inspected includes the geographical elements, area and boundaries of the map of the area to be inspected;

[0070] S12: preprocessing the map data of the inspection area, including removing noise, smoothing boundaries, and filling missing parts to obtain preprocessed map data;

[0071] S13: Using an eight-directional rasterization algorithm, the preprocessed map data is converted into raster map data; in the raster map data, each grid contains a unique grid coordinate; the area corresponding to each grid in the rasterization is the coverage area of ​​a single frame video when a drone is patrolling.

[0072] S2: Build an environment simulator to simulate the movement of suspicious targets and obtain the movement data of suspicious targets, such as Figure 2 As shown;

[0073] S21: Build an environment simulator, the calculation method is:

[0074] {x t}=concat(x 1 ,x 2 ,…,x t )

[0075] V t =VAE({x t})

[0076] h t =LSTM(h t-1 ,x t ,V t-1 )

[0077] Among them, {x t} is the motion trajectory of the suspicious target at time t, concat() is the concatenation operation, x t is the grid coordinate of the suspicious target at time t, V t is the trajectory feature of the suspicious target at time t, VAE() is the variational autoencoder operation, and h t is the suspicious target location feature at time t, LSTM() is the long short-term memory network operation, h t-1 is the suspicious target location feature at time t-1, V t-1 is the trajectory feature of the suspicious target at time t-1, and t is the time index;

[0078] S22: using an environmental simulator to simulate the movement of the suspicious target, and recording the grid coordinates of the suspicious target at each moment, to obtain the movement data of the suspicious target;

[0079] In step S21, an environmental simulator is constructed by combining a variational autoencoder (VAE) and a long short-term memory network (LSTM). The powerful feature extraction and representation learning capabilities of VAE can extract the pattern of trajectory changes from trajectory information. Based on long-term dependency learning and time series prediction capabilities, LSTM simulates the future position of suspicious targets through historical trajectories and current positions, so that the environmental simulator can accurately simulate the movement data of suspicious targets and provide rich training samples for the subsequent collaborative inspection model based on reinforcement learning.

[0080] Compared with traditional methods that only rely on mathematical models and statistics, which are difficult to accurately capture the complex motion patterns of the target, the environmental simulator of the present invention helps the subsequent collaborative patrol model to learn collaborative strategies in more complex environments, thereby improving the collaboration efficiency between drones and further enhancing the overall patrol capability.

[0081] S3: Calculate the number of drones based on the area to be inspected and the expected inspection time, and divide the initial inspection area using a clustering algorithm;

[0082] S31: Calculate the number of drones based on the area to be inspected and the expected inspection time. The calculation method is:

[0083] n=MLP(S,T)

[0084] Where n is the number of drones, MLP() is a fully connected neural network operation, S is the area to be inspected, and T is the expected inspection time;

[0085] S32: Starting from the center point of the area to be inspected, draw n rays with uniform angle distribution, and the angle interval of each ray is Get n sub-regions;

[0086] S33: The center points of each sub-area divided by the ray are used as the initial clustering centers of the clustering algorithm, and the K-means clustering algorithm is used to cluster the grids of the inspection area. The number of clusters is still n, and the clustering center is allowed to move during the clustering process;

[0087] S34: The clustering result is used as the initial patrol area of ​​each UAV, and each grid in the initial patrol area is assigned the area category of the area;

[0088] In step S3, a fully connected neural network is used to calculate the required number of drones based on the area of ​​the area to be inspected and the expected inspection time. Then, the area to be inspected is divided into the initial inspection area for each drone through ray partitioning and K-means clustering algorithm, paving the way for subsequent area redivision. At the same time, the number of drones can also be manually set according to actual conditions, and the system will perform initial inspection area division based on the manually set number of drones.

[0089] S4: Based on the real-time inspection situation, determine whether to conduct collaborative tracking of suspicious targets, and build a regional redivision model to redivision the uninspected areas;

[0090] S41: According to the suspicious target movement data generated by the environment simulator, when a suspicious target appears in a certain area, the UAV that discovers the suspicious target is controlled to track the suspicious target;

[0091] S42: Calculate the Euclidean distances between all the remaining drones and the suspicious target except the drone that discovered the suspicious target and sort them to obtain a drone distance sequence L, and obtain the grid coordinates of the nearest drone;

[0092] S43: According to the suspicious target category, determine whether to send the nearest UAV to track the suspicious target collaboratively. The calculation method is:

[0093] If the suspicious target is a Class I or Class II target, the nearest UAV will be dispatched to collaboratively track the suspicious target;

[0094] If the target is a Class III target, the nearest UAV will not be sent to conduct collaborative tracking of the suspicious target;

[0095] The rights are limited to implementation examples and are not specifically limited. The Class I targets include: trucks and motorcycles; the Class II targets include: pedestrians, cars, and off-road vehicles; the Class III targets include: bicycles and drones;

[0096] Trucks may be used as potential tools for illegal transportation and hiding of dangerous goods, and motorcycles may be used for quick escape, illegal crossing or other high-risk behaviors, so they are classified as Class I targets; pedestrians, cars, and off-road vehicles may be involved in illegal activities, but they are less dangerous than trucks and motorcycles, so they are classified as Class II targets; bicycles travel at a slower speed and have a limited load capacity, and drones may be involved in illegal filming, privacy infringement, etc., and are not likely to cause major dangers, so they are classified as Class III targets;

[0097] S44: If the nearest UAV is dispatched to collaboratively track the suspicious target, a regional redivision model is constructed for the unpatrolled area of ​​the UAV that discovered the suspicious target and the nearest UAV, and the unpatrolled area is redivisioned. The calculation method is:

[0098] C = softmax(MLP(L,s))

[0099] Among them, C is the area category corresponding to the patrol area after regional redivision, softmax() is the softmax function, L is the drone distance sequence, and s is the grid coordinate of the center of the unpatrolled area;

[0100] Compared with traditional methods, traditional methods usually rely on manual or fixed rules to divide areas, and temporarily adjust the inspection plan when suspicious targets appear, which easily leads to inspection blind spots or repeated inspections, and it is difficult to respond to changes in suspicious targets in real time; the present invention can dynamically adjust and optimize the inspection area of ​​the drone in real time according to the appearance of suspicious targets and the tracking of drones through area redivision, so that uninspected areas can be taken over by other drones more quickly, ensuring that drones can cover uninspected areas more effectively, reducing inspection blind spots, and enhancing the collaborative operation capabilities between drones;

[0101] For example, in a certain area to be inspected, four drone inspection areas (A, B, C, D) are initially divided through step S3. The initial cluster center of each area is determined by the K-means clustering algorithm and a drone is assigned. During the inspection, drone A finds a suspicious truck (Class I target) in area A. Drone A immediately tracks the suspicious truck, and then the system calculates the Euclidean distance between other drones (B, C, D) and the current position of the suspicious truck except drone A, and sorts them.

[0102] Assume that the sorting result is L = [B, C, D], which means that drone B is closest to the suspicious truck, drone C is second, and drone D is farthest. Since the suspicious target is a Class I target (truck), according to the preset rules, it is decided to send the closest drone B for collaborative tracking.

[0103] For the unpatrolled areas of drone A and drone B, a regional redivision model is constructed; the central grid coordinates of the unpatrolled part of area A and the unpatrolled part of area B are s 甲 and 乙 ; After the area is redivided, the category of the uninspected part of area A and the uninspected part of area B is D, then the uninspected part of area A and the uninspected part of area B will continue to be inspected by drone D.

[0104] S5: Build a collaborative patrol model to control drones to collaboratively track suspicious targets;

[0105] S51: Construct a collaborative patrol model, and define the state space of the collaborative patrol model as the current position and power of the drone that discovers the suspicious target, and the current position and power of the nearest drone; the action space is the grid coordinates of the adjacent positions of the drone; the value function is the Q-learning value function;

[0106] S52: Define the reward function of the collaborative patrol model, which is calculated as follows:

[0107] R=R track +R collaboration +R battery

[0108] Among them, R is the reward function of the collaborative inspection model, R track is the tracking function, R collaboration is the coordination function, R battery is the battery consumption function;

[0109] R track =α×m continuous -β×d grid

[0110] Among them, α is the continuous tracking coefficient, m continuous is the number of grids that the UAV continuously tracks to find suspicious targets, β is the distance penalty coefficient, d grid The grid distance between the UAV that detects the suspicious target and the suspicious target;

[0111]

[0112] Among them, δ is the synergy reward coefficient, d between is the grid distance between the two UAVs, θ is the maximum grid distance between the two UAVs, φ is the cooperative angle between the two UAVs formed with the current position of the suspicious target as the vertex, and φ min is the lower limit of the cooperative angle, φ ma is the upper limit of the synergy angle;

[0113]

[0114] Among them, ε is the power coefficient, b is the current power of the drone, and battery is the drone power threshold;

[0115] S53: Based on the defined state space, action space, value function and reward function, the Q table is initialized and the suspicious target movement data is imported. Finally, the collaborative inspection model is trained using the Q-learning method.

[0116] S54: According to the trained collaborative inspection model, the UAV that discovers the suspicious target and the nearest UAV are controlled to track the suspicious target;

[0117] The reward function proposed by this invention is track By giving rewards for continuous tracking and punishing behaviors that are too far away from the target; when the drone is able to continuously track the target, that is, the number of grids continuously tracked increases, it will receive a positive reward, which is measured by the continuous tracking coefficient α; at the same time, in order to punish behaviors that are too far away from the target, the present invention introduces a distance penalty coefficient β, when the grid distance between the drone and the suspicious target increases, it will receive a negative reward, so as to ensure that the drone continuously monitors the target during the tracking process;

[0118] In order to optimize the effect of collaborative tracking, the present invention uses R collaboration Partially encourage the coordinated UAVs to maintain an appropriate distance and coordination angle; when the distance between the two UAVs is d between Less than or equal to the maximum grid distance θ, and the cooperative angle φ between them is within the set lower limit φ min and the upper limit φ max When the distance between the two drones is 0.04, they will receive a collaborative reward. The size of the reward is measured by the collaborative reward coefficient δ, which avoids the drones from being too close or too far away, as well as the inappropriate angle, thereby optimizing the collaborative tracking effect.

[0119] The present invention also passes R battery The power consumption of the drone is partially taken into account; when the current power b of the drone is lower than the set power threshold battery, it will receive a negative reward. The negative reward is measured by the power coefficient ε, which aims to avoid the situation where the drone runs out of power and cannot return during the tracking process;

[0120] For example, in a collaborative patrol mission, there are two drones, A and B, which are tracking a suspicious target. Drone A discovers the suspicious target and starts tracking it. At the same time, drone B, as the closest drone, also joins the tracking operation.

[0121] The continuous tracking coefficient α is set to 1, indicating a high degree of emphasis on continuous tracking behavior;

[0122] The distance penalty coefficient β is set to 0.5, which means that the penalty for being too far away from the target is moderate;

[0123] The collaborative reward coefficient δ is set to 5, indicating a high degree of reward for collaborative behavior;

[0124] The power factor ε is set to 3, which means that the penalty for power consumption is high to avoid running out of power;

[0125] The battery threshold is set to 30%, which means that when the battery is lower than 30%, the drone will be penalized for power consumption;

[0126] The drone A continuously tracks the suspicious target for 3 grids. At this time, it receives a continuous tracking reward. At the same time, the grid distance between drone A and the suspicious target is 0, so it does not receive a distance penalty, which is calculated as R track =1×3-0=3;

[0127] The distance between UAV A and UAV B is 4 grids, and the coordination angle between them is 45 degrees; the maximum grid distance θ is set to 10, and the lower limit of the coordination angle φ min Set to 30 degrees, the upper limit of the collaborative angle φ maxis set to 60 degrees; because the coordination condition is met, both drones A and B receive coordination rewards, which are calculated as

[0128] Drone A and Drone B's current battery level is 60%, which is higher than the battery threshold of 30%, so it does not suffer any battery consumption penalty;

[0129] The final reward value R = 3 + 1.5 + 0 = 4.5, indicating that the model gives a large reward to the current behavior of the drone.

[0130] The present invention is particularly suitable for drone inspection scenarios where the inspection range needs to be flexibly adjusted according to suspicious targets; the present invention can dynamically adjust the inspection area and drone coordination strategy, thereby improving the efficiency of multi-drone inspections and achieving efficient tracking and monitoring of suspicious targets.

[0131] Embodiment 2: The present invention also discloses a multi-UAV collaborative inspection system based on reinforcement learning, comprising:

[0132] Map data collection and rasterization module: collects map information of the area to be inspected, and rasterizes the map to obtain raster map data;

[0133] Environmental simulator construction module: construct an environmental simulator, simulate the movement of suspicious targets, and obtain suspicious target movement data;

[0134] Initial patrol area division module: Calculate the number of drones based on the area to be patrolled and the expected patrol time, and divide the initial patrol area in combination with the clustering algorithm;

[0135] Area redivision module: Based on the real-time inspection situation, determine whether to conduct collaborative tracking of suspicious targets, and build an area redivision model to redivision uninspected areas;

[0136] Collaborative patrol module: Build a collaborative patrol model to control drones to collaboratively track suspicious targets.

[0137] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0139] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-UAV collaborative inspection method based on reinforcement learning, characterized in that: The following steps are involved: S1: Collect map information of the area to be inspected and rasterize the map to obtain raster map data; S2: Build an environment simulator to simulate the movement of suspicious targets and obtain the movement data of suspicious targets; S3: Calculate the number of drones based on the area to be inspected and the expected inspection time, and divide the initial inspection area using a clustering algorithm; S4: Based on the real-time inspection situation, determine whether to conduct collaborative tracking of suspicious targets, and build a regional redivision model to redivision the uninspected areas; S5: Build a collaborative patrol model to control drones to collaboratively track suspicious targets, including: S51: Construct a collaborative patrol model, and define the state space of the collaborative patrol model as the current position and power of the drone that discovers the suspicious target, and the current position and power of the nearest drone; the action space is the grid coordinates of the adjacent positions of the drone; the value function is the Q-learning value function; S52: Define the reward function of the collaborative patrol model, which is calculated as follows: R=R track +R collaboration +R battery Among them, R is the reward function of the collaborative inspection model, R track is the tracking function, R collaboration is the coordination function, R battery is the battery consumption function; R track =α×m continuous -β×d grid Among them, α is the continuous tracking coefficient, m continuous is the number of grids that the UAV continuously tracks to find suspicious targets, β is the distance penalty coefficient, d grid The grid distance between the UAV that detects the suspicious target and the suspicious target; Among them, δ is the synergy reward coefficient, d between is the grid distance between the two UAVs, θ is the maximum grid distance between the two UAVs, φ is the cooperative angle between the two UAVs formed with the current position of the suspicious target as the vertex, and φ min is the lower limit of the cooperative angle, φ max is the upper limit of the synergy angle; Among them, ε is the power coefficient, b is the current power of the drone, and battery is the drone power threshold; S53: Based on the defined state space, action space, value function and reward function, the Q table is initialized and the suspicious target movement data is imported. Finally, the collaborative inspection model is trained using the Q-learning method. S54: Based on the trained collaborative patrol model, the UAV that discovers the suspicious target and the nearest UAV are controlled to track the suspicious target.

2. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 1 is characterized in that: The S1 step includes: S11: The map information of the area to be inspected includes the geographical elements, area and boundaries of the map of the area to be inspected; S12: preprocessing the map information of the inspection area, including removing noise, smoothing boundaries, and filling missing parts to obtain preprocessed map data; S13: Using an eight-directional rasterization algorithm, the preprocessed map data is converted into raster map data; in the raster map data, each grid contains a unique grid coordinate; the area corresponding to each grid in the rasterization is the coverage area of ​​a single frame video when a drone is patrolling.

3. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 1 is characterized in that: The S2 step includes: The variational autoencoder and the long short-term memory network are integrated to extract the trajectory features of the suspicious target through the variational autoencoder. The trajectory features of the suspicious target are combined with the current position, and the movement of the suspicious target is simulated through the long short-term memory network. Finally, the grid coordinates of the suspicious target at each moment are recorded to obtain the movement data of the suspicious target.

4. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 3 is characterized in that: The S2 step includes: S21: Build an environment simulator, the calculation method is: {x t }=concat(x1,x2,…,x t ) V t =VAE({x t }) h t =LSTM(h t-1 ,x t ,V t-1 ) Among them, {x t } is the motion trajectory of the suspicious target at time t, concat() is the concatenation operation, x t is the grid coordinate of the suspicious target at time t, V t is the trajectory feature of the suspicious target at time t, VAE() is the variational autoencoder operation, and h t is the suspicious target location feature at time t, LSTM() is the long short-term memory network operation, h t-1 is the suspicious target location feature at time t-1, V t-1 is the trajectory feature of the suspicious target at time t-1, and t is the time index; S22: Using an environmental simulator, simulate the movement of the suspicious target, and record the grid coordinates of the suspicious target at each moment to obtain the movement data of the suspicious target.

5. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 1 is characterized in that: The S3 step includes: The number of drones is calculated according to the area of ​​the area to be inspected and the expected inspection time; the inspection sub-areas are divided according to the number of drones; the K-means clustering algorithm is used to cluster the grids of the inspection area, and the clustering results are used as the initial inspection area of ​​each drone.

6. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 5 is characterized in that: The S3 step includes: S31: Calculate the number of drones based on the area to be inspected and the expected inspection time. The calculation method is: n=MLP(S,T) Where n is the number of drones, MLP() is a fully connected neural network operation, S is the area to be inspected, and T is the expected inspection time; S32: Starting from the center point of the area to be inspected, draw n rays with uniform angle distribution, and the angle interval of each ray is Get n sub-regions; S33: The center points of each sub-area divided by the ray are used as the initial clustering centers of the clustering algorithm, and the K-means clustering algorithm is used to cluster the grids of the inspection area. The number of clusters is still n, and the clustering center is allowed to move during the clustering process; S34: The clustering results are used as the initial patrol area of ​​each UAV, and each grid in the initial patrol area is assigned a regional category of the area.

7. The multi-UAV collaborative inspection method based on reinforcement learning according to claim 6 is characterized in that: The S4 step includes: S41: According to the suspicious target movement data generated by the environment simulator, when a suspicious target appears in a certain area, the UAV that discovers the suspicious target is controlled to track the suspicious target; S42: Calculate the Euclidean distances between all the remaining drones and the suspicious target except the drone that discovered the suspicious target and sort them to obtain a drone distance sequence L, and obtain the grid coordinates of the nearest drone; S43: According to the suspicious target category, determine whether to send the nearest UAV to track the suspicious target collaboratively. The calculation method is: Mark suspicious targets. If the suspicious targets are Class I or Class II targets, send the nearest UAV to track the suspicious targets collaboratively. If the target is a Class III target, the nearest UAV will not be sent to conduct collaborative tracking of the suspicious target; S44: If the nearest UAV is dispatched to collaboratively track the suspicious target, a regional redivision model is constructed for the unpatrolled area of ​​the UAV that discovered the suspicious target and the nearest UAV, and the unpatrolled area is redivisioned. The calculation method is: C = softmax(MLP(L, s)) Among them, C is the area category corresponding to the patrol area after regional redivision, softmax() is the softmax function, L is the drone distance sequence, and s is the grid coordinate of the center of the unpatrolled area.

8. A multi-UAV collaborative inspection system based on reinforcement learning, characterized in that: include: Map data collection and rasterization module: collects map information of the area to be inspected, and rasterizes the map to obtain raster map data; Environmental simulator construction module: construct an environmental simulator, simulate the movement of suspicious targets, and obtain suspicious target movement data; Initial patrol area division module: Calculate the number of drones based on the area to be patrolled and the expected patrol time, and divide the initial patrol area in combination with the clustering algorithm; Area redivision module: Based on the real-time inspection situation, determine whether to conduct collaborative tracking of suspicious targets, and build an area redivision model to redivision uninspected areas; Collaborative inspection module: build a collaborative inspection model to control drones to collaboratively track suspicious targets; To realize a multi-UAV collaborative inspection method based on reinforcement learning as described in any one of claims 1-7.

Images