Heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision

Abstract

The present application relates to a kind of heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision, belong to heterogeneous multi-agent coordination movement and obstacle avoidance control technical field, solve the low success rate of existing technology in driving, cluster out of control and the problem of low task completion quality.The second decision network of pre-training makes that control mode conversion condition is no longer fixed, can avoid the phenomenon that cluster appears out of control, improve the stability of control method, also can automatically adjust the direction of unmanned aerial vehicle around cluster movement;Through the contour of cluster generation unmanned aerial vehicle arc motion trajectory, so that unmanned aerial vehicle appears in the perception range of cluster internal node sufficiently, to maximize the control effect of unmanned aerial vehicle to cluster node, and the switching of control mode is also more flexible, more suitable for obstacle environment, the success rate of driving is high;Through the first decision network of pre-training, timely adjust the control parameter of arc control, optimize the driving process, further improve the success rate of driving.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous multi-agent coordinated motion and obstacle avoidance control technology, specifically to a heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making. Background Technology

[0002] Much research on multi-agent cooperation has been conducted after the introduction of behavior-based control paradigms. Shepherding, as a typical heterogeneous multi-agent behavioral paradigm, has attracted much attention from researchers. Currently, Shepherding control methods include GCM+V control (Global Centre of Mass+V Control, a "V"-shaped control method based on the center of mass) and arc control methods.

[0003] The essence of both arc-shaped control and GCM+V control methods lies in the fact that when the swarm radius exceeds a given threshold, the drone will move around the swarm in an arc or V-shaped trajectory, driving individuals on the periphery towards the swarm center. Simultaneously, the drone adjusts its trajectory based on the target's orientation to ensure the swarm moves towards the target's position. However, this method of promoting swarm convergence through circling movement can potentially fail.

[0004] First, when obstacles are added to the environment, the stagnation of a cluster within the obstacle's recesses can lead to individual members becoming isolated. For example... Figure 1 and Figure 2 As shown, as the swarm radius increases, the drone's movement amplitude will increase, leading to a loss of swarm control. Even if the swarm remains behind an obstacle groove without any individual drones leaving the group, the drones cannot traverse the obstacle to drive the swarm around it. Furthermore, the driving path guided by a few simple rules may be blocked by the obstacle, making it difficult to drive the swarm to the target area.

[0005] Secondly, although the GCM+V control method enables the conversion between collect and drive actions, the conditions for this conversion during task execution are fixed. This can lead to the UAV performing unnecessary actions under certain conditions as the task progresses. For example, when the swarm is divided into two parts by an obstacle, such as... Figure 3 As shown, the drive action should be stopped at this point, and the collect action should be taken to bring the cluster together. However, since the cluster has only been divided into two parts, the cluster radius is still less than the threshold. Therefore, the drone will continue to take the drive action, which will cause the cluster to continue to go out of control.

[0006] Finally, both the GCM+V control method and the arc control method have their corresponding control parameters. For example, the arc control method mainly includes threshold parameters to determine whether the UAV meets the conditions for direction change and parameters to determine the distance of the UAV from the swarm centroid. These parameters are fixed throughout the entire process of the UAV's mission, but in reality, there is a suitable range of values ​​for each situation. If the parameters are not within their suitable range, it will lead to low mission completion quality and swarm control failure.

[0007] In summary, existing technologies suffer from several problems, including low success rates in driving away groups due to individuals remaining in the cluster under obstacle conditions, low task completion quality due to fixed control mode switching conditions and the inability to adjust control parameters in real time, and cluster loss of control. Summary of the Invention

[0008] In view of the above problems, the present invention provides a heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making, which solves the problems of low success rate of driving away individuals due to stagnation of clusters under obstacle conditions, low task completion quality due to fixed control mode switching conditions and inability to adjust control parameters in real time, and cluster loss of control in the prior art.

[0009] It should be noted that "low mission completion quality" refers to a low success rate for drones, or even if the mission is successfully completed, the energy consumption and time consumption are high, and the overall state of the swarm is poor. "Fixed control mode switching conditions" and "inability to adjust control parameters in real time" can both lead to swarm loss of control, but the invention addresses these two shortcomings through a first decision network and a second decision network, respectively, thereby reducing the probability of swarm loss of control.

[0010] This invention provides a heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making. The heterogeneous multi-agent system includes unmanned aerial vehicles (UAVs) and a non-cooperative agent swarm. The non-cooperative agent swarm possesses autonomous consciousness but cannot cooperate with the UAV. The method includes the following steps:

[0011] S1. Obtain environmental feature information, process it based on PRM, and plan the cluster path; wherein, the cluster path is composed of the starting point, cluster path nodes and the center point of the target area connected in sequence;

[0012] S2. The UAV interacts with the environment to obtain the feature information of the non-cooperative intelligent agent cluster and the feature information of obstacles, and determines whether the non-cooperative intelligent agent cluster has reached the target area; if so, the mission ends; otherwise, the UAV is controlled to move along the UAV path node, driving the non-cooperative intelligent agent cluster to move to the next cluster path node, and obtaining the information of the next cluster path node.

[0013] S3. Merge the feature information of the non-cooperative intelligent agent cluster in S2, the feature information of the obstacles, and the information of the next cluster path node into state feature information, and input it into the second decision network;

[0014] S4. The second decision network selects either the collect control mode or the drive control mode for the UAV based on the state feature information; in the collect control mode, the UAV path nodes are updated using the GCM-targeting trajectory generator; in the drive control mode, the state feature information is input into the first decision network, and the control parameters of the arc trajectory generator are output; the UAV path nodes are updated using the arc trajectory generator based on the control parameters and the state feature information.

[0015] S5. Repeat S2 to S4 until the non-cooperative agent cluster reaches the target area.

[0016] Furthermore, S1, based on PRM planning, obtains the following cluster paths:

[0017] Multiple sampling points are randomly generated within a two-dimensional planar region including the starting point and the target area; if an obstacle exists within the two-dimensional planar region and the sampling point coincides with the obstacle, the sampling point is removed.

[0018] The A* algorithm is used to find the shortest path through the sampling points to the target area, which is called the cluster path. The cluster path does not pass through obstacles, and the sampling points on the cluster path are cluster path nodes.

[0019] Furthermore, in S4, updating the UAV path nodes using the GCM-targeting trajectory generator includes:

[0020] The contour radius of the non-cooperative agent cluster is calculated;

[0021] Compare the outline radius of the non-cooperative agent cluster with a preset outline radius threshold; if the outline radius of the non-cooperative agent cluster is greater than the preset outline radius threshold, generate a drone path node to drive the non-cooperative agent furthest from the center of the non-cooperative agent cluster toward the center; otherwise, generate a drone path node to make the drone follow and drive the non-cooperative agent cluster along the cluster path.

[0022] Furthermore, S4, the control parameters of the arc trajectory generator include a threshold θ2 for changing the rotation direction of the UAV and the radius of curvature d of the UAV trajectory. over .

[0023] Furthermore, the first decision network is obtained through pre-training of a neural network; the pre-training of the first decision network includes a first reward and punishment mechanism, which includes target point reward, cluster state penalty, and drone movement penalty; among which,

[0024] Target point reward refers to the reward given when a drone moves along a drone path node and drives a non-cooperative intelligent agent swarm to reach the swarm path node that serves as the target point.

[0025] Drone movement penalty refers to the penalty imposed on drone movement, used to limit energy consumption caused by drone movement;

[0026] Cluster state penalties include discrete state penalties and velocity vector state penalties. Discrete state penalties are imposed when the contour radius of a non-cooperative agent cluster exceeds a preset contour radius threshold, or when an outlier appears. Velocity vector state penalties are imposed when the velocity vector offset angle exceeds a preset velocity vector offset angle threshold. The velocity vector offset angle is the angle difference between the cluster center velocity vector and the ideal velocity vector, where the ideal velocity vector is the vector pointing from the cluster center to the center of the target region.

[0027] Furthermore, the second decision network pre-trains the neural network and sets up a forced learning reward and punishment mechanism; among which,

[0028] The forced learning reward and punishment mechanism refers to the punishment imposed on a non-cooperative agent cluster if it fails to take correct measures when the cluster is in extreme states, including abnormal cluster radius and abnormal stagnation. Specifically, this includes:

[0029] When the outline radius of the non-cooperative agent cluster is greater than the preset outline radius threshold, and the second decision network still does not use the collect method to gather the non-cooperative agent cluster, resulting in the outline radius of the non-cooperative agent cluster becoming larger and larger, the second decision network is penalized.

[0030] When the contour radius of the non-cooperative agent cluster is less than the preset contour radius threshold, and the second decision network continues to use the collect method to gather the non-cooperative agent cluster, causing the non-cooperative agent cluster to stagnate within the preset time window threshold, the second decision network is penalized.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects:

[0032] (1) The heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making of the present invention, compared with the arc control method, makes the control mode switching conditions no longer fixed by setting a pre-trained second decision network, which can avoid the phenomenon of the cluster going out of control, improve the stability of the control method, and can also automatically adjust the direction of the UAV's movement around the cluster.

[0033] (2) The heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making of the present invention generates the arc motion trajectory of the UAV by conforming to the cluster outline, so that the UAV fully appears within the perception range of the nodes inside the cluster, thereby maximizing the control effect of the UAV on the cluster nodes. Moreover, the switching of control mode is more flexible, more suitable for obstacle environment, and has a high success rate of driving away obstacles.

[0034] (3) The heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making of the present invention can adjust the control parameters of arc control in a timely manner by setting a pre-trained first decision network, thereby reducing the probability of cluster loss of control and optimizing the driving process to further improve the driving success rate. Attached Figure Description

[0035] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0036] Figure 1 A schematic diagram illustrating the failure of the existing GCM+V control method under obstructed conditions;

[0037] Figure 2 This is a schematic diagram illustrating the failure of existing arc control methods under obstructed conditions.

[0038] Figure 3 A schematic diagram showing a cluster separated by obstacles;

[0039] Figure 4 This diagram illustrates the technical problem to be solved by the present invention and the corresponding technical means.

[0040] Figure 5 This is a schematic diagram of the cluster path proposed in this invention;

[0041] Figure 6 This is a schematic diagram of the GCM-targeting control method proposed in this invention;

[0042] Figure 7 This is a schematic diagram of the arc control method proposed in this invention;

[0043] Figure 8 This is a framework diagram of the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making proposed in this invention.

[0044] Figure 9 The flowchart shows the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making proposed in this invention.

[0045] Figure 10 This is a schematic diagram of the reward and punishment mechanism for the pre-training process of the first decision network proposed in this invention;

[0046] Figure 11 This is a schematic diagram of static obstacles in a simulated environment.

[0047] Figure 12 This is a schematic diagram of the round reward curve during the pre-training process of the first decision network proposed in this invention.

[0048] Figure 13 This is a schematic diagram of the round reward curve during the pre-training process of the second decision network proposed in this invention.

[0049] Figure 14 A bar chart comparing the total number of time steps for the three control methods;

[0050] Figure 15 A bar chart comparing the total distance traveled by UAVs using three different control methods;

[0051] Figure 16 A bar chart comparing the average cluster radius of the three control methods;

[0052] Figure 17 A bar chart comparing the outlier rates of the three control methods;

[0053] Figure 18 A bar chart comparing the success rates of the three control methods;

[0054] Figure 19 A schematic diagram of the UAV trajectory using the arc-shaped control method;

[0055] Figure 20 A schematic diagram of the UAV trajectory using the GCM+V control method;

[0056] Figure 21 This is a schematic diagram of the UAV trajectory for the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making proposed in this invention. Detailed Implementation

[0057] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0058] like Figure 4 As shown, to address the issue of individual members leaving or becoming stranded in environments with obstacles, this paper employs a Probabilistic Path Map (PRM) algorithm to plan the cluster path. To address the problem of fixed control mode switching conditions, a pre-trained neural network is used to decide on the switching between collect and drive control modes. Furthermore, to address the issue of control parameters not being adjustable in real time, a pre-trained neural network selects control parameters based on the current situation.

[0059] This invention discloses a heterogeneous multi-agent motion control method based on adaptive hierarchical decision-making, specifically a heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making. The heterogeneous multi-agent includes a drone and a non-cooperative agent swarm. The non-cooperative agent swarm has autonomous consciousness but cannot cooperate with the drone. The method includes the following steps:

[0060] S1. Obtain environmental feature information, process it based on PRM, and plan the cluster path; wherein, the cluster path is composed of the starting point, cluster path nodes and the center point of the target area connected in sequence.

[0061] like Figure 5 As shown, the cluster paths obtained based on PRM planning include:

[0062] Multiple sampling points are randomly generated within a two-dimensional planar region including the starting point and the target area; if an obstacle exists within the two-dimensional planar region and the sampling point coincides with the obstacle, the sampling point is removed.

[0063] The A* algorithm is used to find the shortest path through the sampling points to the target area, which is called the cluster path. The cluster path does not pass through obstacles, and the sampling points on the cluster path are cluster path nodes.

[0064] S2. The UAV interacts with the environment to obtain feature information of the non-cooperative intelligent agent cluster and obstacle feature information, and determines whether the non-cooperative intelligent agent cluster has reached the target area; if so, the mission ends; otherwise, the UAV is controlled to move along the UAV path node, driving the non-cooperative intelligent agent cluster to move to the next cluster path node, and obtaining the information of the next cluster path node.

[0065] S3. Merge the feature information of the non-cooperative intelligent agent cluster in S2, the feature information of the obstacles, and the information of the next cluster path node into state feature information, and input it into the second decision network.

[0066] S4. The second decision network selects either the collect control mode or the drive control mode for the UAV based on the state feature information; in the collect control mode, the UAV path nodes are updated using the GCM-targeting trajectory generator; in the drive control mode, the state feature information is input into the first decision network, and the control parameters of the arc trajectory generator are output; the UAV path nodes are updated using the arc trajectory generator based on the control parameters and the state feature information.

[0067] The control parameters of the arc trajectory generator include the threshold θ2 used to change the rotation direction of the UAV and the radius of curvature d of the UAV trajectory. over .

[0068] Specifically, such as Figure 6 As shown, updating the UAV path nodes using the GCM-targeting trajectory generator includes:

[0069] The contour radius of the non-cooperative agent cluster is calculated;

[0070] Compare the outline radius of the non-cooperative agent cluster with a preset outline radius threshold; if the outline radius of the non-cooperative agent cluster is greater than the preset outline radius threshold, generate a drone path node to drive the non-cooperative agent furthest from the center of the non-cooperative agent cluster toward the center; otherwise, generate a drone path node to make the drone follow and drive the non-cooperative agent cluster along the cluster path.

[0071] like Figure 7 As shown, updating the UAV path nodes using the arc trajectory generator based on the control parameters and state feature information includes:

[0072] The line segment DG is obtained by connecting the position D of the UAV with the position G of the target area.

[0073] From the perspective of the drone, if the center C of the non-cooperative intelligent agent cluster is located to the left of the straight line segment DG, a drone path node is generated to make the drone rotate clockwise around the non-cooperative intelligent agent cluster; otherwise, a drone path node is generated to make the drone rotate counterclockwise around the non-cooperative intelligent agent cluster.

[0074] Connect the position D of the UAV with the center C of the non-cooperative intelligent agent cluster using a straight line segment to obtain the straight line segment DC.

[0075] When the drone rotates clockwise around the cluster of non-cooperative intelligent agents, the non-cooperative intelligent agent that is furthest from the straight line DC on the left side of the drone is selected, and the drone's path nodes are distributed on an arc pointing to that non-cooperative intelligent agent.

[0076] When the drone rotates counterclockwise around the cluster of non-cooperative intelligent agents, the non-cooperative intelligent agent that is furthest from the straight line DC on the right side of the drone is selected, and the drone's path nodes are distributed on an arc pointing to that non-cooperative intelligent agent.

[0077] Set the threshold θ2 for changing the rotation direction.

[0078] A reversal criterion is established based on a threshold θ2 for changing the rotation direction to prevent the drone from continuously rotating clockwise or counterclockwise around the non-cooperative agent swarm. When the reversal criterion is met, the drone switches the direction of rotation around the non-cooperative agent swarm. The reversal criterion is specifically expressed as follows:

[0079]

[0080] in, Let be the vector pointing from the center C of the non-cooperative intelligent agent cluster to the position D of the UAV; Let be the vector pointing from the center C of the non-cooperative agent cluster to the position G of the target region.

[0081] It is easy to understand that the drone will move alternately clockwise or counterclockwise around the non-cooperative intelligent agent swarm, and the distance between it and the center of the non-cooperative intelligent agent swarm will be d. furthest +d over ; where d furthest It is the distance between the non-cooperative agent that is farthest from the center of the non-cooperative agent cluster and the center.

[0082] S5. Repeat S2 to S4 until the non-cooperative agent cluster reaches the target area.

[0083] The first and second decision networks form a hierarchical decision network model to realize the adaptive hierarchical decision-making of this method; the framework of the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making is as follows: Figure 8 As shown, the process is as follows: Figure 9 As shown.

[0084] It should be noted that the first decision network was obtained through pre-training of the neural network.

[0085] The first decision network is used to adjust and output the control parameters of the arc trajectory generator to the arc trajectory generator. The control parameters of the arc trajectory generator include the threshold θ2 for changing the rotation direction of the UAV and the radius of curvature d of the UAV trajectory. overHere, we skip assigning a value to θ2 and directly set the action set of the first decision network to the UAV rotation direction, as shown in Table 1:

[0086] Table 1. Set of Actions in the First Decision Network

[0087] Serial Number 0 1 action Rotate clockwise Rotate counterclockwise

[0088] The pre-training of the first decision network includes a first reward and punishment mechanism, which comprises target point rewards, cluster state penalties, and drone movement penalties, such as... Figure 10 As shown.

[0089] Target point reward refers to the reward given when a drone moves along a drone path node and drives a non-cooperative intelligent agent swarm to reach the swarm path node that serves as the target point.

[0090] Drone movement penalties refer to the penalties imposed on drones for moving. Because drones traveling too far will result in significant energy consumption, penalties are necessary to constrain their movement.

[0091] Cluster state penalties include discrete state penalties and velocity vector state penalties. Discrete state penalties are imposed when the contour radius of a non-cooperative agent cluster exceeds a preset contour radius threshold, or when an outlier appears. Velocity vector state penalties are imposed when the velocity vector offset angle exceeds a preset velocity vector offset angle threshold. The velocity vector offset angle is the angle difference between the cluster center velocity vector and the ideal velocity vector, which is the vector pointing from the cluster center to the center of the target region. It should be noted that the direction of this vector is used here, so its magnitude is not important.

[0092] The second decision network is obtained by pre-training the neural network and adding a forced learning reward and punishment mechanism on the basis of the first decision network.

[0093] The second decision network is used to decide whether the UAV should adopt the collect control mode or the drive control mode under the current environmental conditions. The collect control mode corresponds to the GCM-targeting aggregation method of the GCM-targeting trajectory generator; the drive control mode corresponds to the arc control method of the arc trajectory generator. Therefore, the action set of the second decision network is shown in Table 2.

[0094] Table 2 Q2 Decision Network Action Set

[0095]

[0096] The forced learning reward and punishment mechanism refers to the punishment imposed on a non-cooperative agent cluster if it fails to take correct measures when the cluster is in extreme states, including abnormal cluster radius and abnormal stagnation. Specifically, this includes:

[0097] When the outline radius of the non-cooperative agent cluster exceeds a preset outline radius threshold, and the second decision network still does not use the collect method to gather the non-cooperative agent cluster together, resulting in the outline radius of the non-cooperative agent cluster becoming larger and larger, the second decision network is penalized.

[0098] When the contour radius of the non-cooperative agent cluster is less than the preset contour radius threshold, and the second decision network continues to use the collect method to gather the non-cooperative agent cluster, causing the non-cooperative agent cluster to stagnate within the preset time window threshold, the second decision network is penalized.

[0099] The heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making of the present invention, compared with the arc control method, makes the control mode switching conditions no longer fixed by setting a pre-trained second decision network, which can avoid the loss of control of the swarm and improve the stability of the control method. Moreover, it can automatically adjust the direction of the UAV's movement around the swarm. Compared with the GCM+V control method, by generating the UAV's arc motion trajectory by conforming to the swarm contour, the UAV is fully within the perception range of the nodes inside the swarm, thereby maximizing the UAV's control over the swarm nodes. Moreover, the switching of control modes is more flexible, more suitable for obstacle environments, and has a high success rate of driving away obstacles. By setting a pre-trained first decision network, the control parameters of the arc control can be adjusted in a timely manner, reducing the probability of the swarm losing control, while optimizing the driving away process and further improving the driving away success rate.

[0100] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment:

[0101] A kinematic model was established by selecting drones and robot swarms.

[0102] The agents in the cluster are selected as robots, and their kinematic model is chosen as a typical point mass model, that is:

[0103]

[0104] Where x and y represent the robot's coordinates, v represents the robot's velocity magnitude, φ represents the robot's direction of motion, the control variables a and ω represent the rate of change of its velocity magnitude and the rate of change of its velocity direction angle, respectively, i represents the robot's index, and N represents the number of robots. It is worth noting that the drone model is the same as the robot; i = 0 represents the drone.

[0105] The modeling of interactions within a robot swarm is based on the classic Boid model, represented as:

[0106] F = ρ sep ·F sep ++ρ align ·F align +ρ coh ·F coh +ρ avoid ·F avoid ;

[0107] in, These represent separation, aggregation, alignment, and repulsion forces, respectively. ρ sep ρ align ρ coh and ρ avoid These represent the coefficients of separation, aggregation, alignment, and repulsion forces, respectively.

[0108] Simulation conditions and initial parameter design.

[0109] Set the simulation environment size to 800×600m 2 There are 20 robots. Initially, the robots are randomly distributed in a rectangular area centered at (120m, 510m), and the size of this rectangular area is 80×60m. 2 The perception range of individuals in the cluster is set to 65m.

[0110] Table 3 shows the weight coefficients of each force within the cluster. The drone is set to start from (50m, 550m) to drive the robot cluster. The target area has a radius of 80m and its center point coordinates are (240m, 420m). The drone completes its mission when all individuals in the robot cluster reach the target area.

[0111] Table 3 shows the weighting coefficients for each force in the Boid model.

[0112] Weighting coefficient <![CDATA[ρ sep ]]> <![CDATA[ρ coh ]]> <![CDATA[ρ align ]]> <![CDATA[ρ avoid ]]> numerical values 0.5 0.5 0.5 0.75

[0113] Six static obstacles of fixed size are placed in the environment, with dimensions as follows: Figure 11 As shown.

[0114] Table 4 shows a summary of the serial numbers and location information of static obstacles in the environment.

[0115] Table 4 Obstacle Types and Location Information

[0116] Serial Number type Initial position of center coordinates (m) 1 static (175m, 200m) 2 static (400m, 200m) 3 static (625m, 200m) 4 static (175m, 400m) 5 static (400m, 400m) 6 static (625m, 400m)

[0117] Training of hierarchical decision network model.

[0118] The input information for the first decision network is shown in Table 5, where... The difference between the coordinate vectors of non-cooperative agents towards the cluster center in two adjacent time steps. This represents the difference in coordinate vectors between cluster path nodes and non-cooperative agents towards the cluster center.

[0119] Table 5. Set of state observations for the Q1 decision network

[0120]

[0121] Here, DQN is used, and the first decision network is trained according to the action set and reward / punishment mechanism in the previous technical solution. Its hyperparameters are shown in Table 6:

[0122] Table 6. Hyperparameters during the training process of the first decision network.

[0123] project Number of Affine layers Playback buffer data size Batch size Learning rate numerical values 3 500000 128 <![CDATA[2.5*10 -4 ]]> project ε initial maximum value ε decay minimum value Discount factor γ numerical values 1 0.05 0.99

[0124] The training results of the first decision network are as follows: Figure 12 As shown. By Figure 12 It can be seen that the curve shows a clear upward trend from the 70th to the 150th round during the training process, which indicates that the Q1 network is learning in the right direction.

[0125] The state observation set and hyperparameters of the second decision network are the same as those of the first decision network. The network is then trained according to the action set and reward mechanism described in the previous technical solution, resulting in the training results of the second decision network as follows: Figure 13 As shown. By Figure 13 It can be seen that the curve shows a clear upward trend from the 100th to the 150th round during the training process, which indicates that the second decision network is learning in the correct direction.

[0126] Simulations were performed using Python programming. Fifteen independent repeated experiments were conducted for each of the three control methods. The data for each indicator are recorded in Tables 7 to 9. To avoid the influence of the environment itself, the average radius of the cluster and the individual outlier rate of the drones after their withdrawal are recorded in Table 10.

[0127] Table 7 Simulation experimental data of the arc control method

[0128] Evaluation indicators 1 2 3 4 5 6 7 8 Total time steps 4348 3762 2715 4800 4800 4800 4800 4800 Total distance traveled by the drone (m) 7010.1 6228.1 4215.6 8169.5 7939.7 7770.1 7696.8 8020.3 Average radius (m) 33.16 42.56 30.04 77.37 41.93 106.03 150.92 122.00 outlier 0.423 0.525 0.182 0.692 0.731 0.821 0.705 0.667

[0129] (Note: The serial numbers in the item column of the simulation test data table represent the serial numbers corresponding to 15 independent repeated experiments.)

[0130] Continued from Table 7

[0131] Evaluation indicators 9 10 11 12 13 14 15 average value Total time steps 2501 4800 4800 2777 4159 2493 4800 3251 Total distance traveled by the drone (m) 3982.8 8105.4 7438.6 4212.8 6692.4 4261.1 7585.2 5229.0 Average radius (m) 28.65 308.61 235.42 29.02 79.41 35.56 105.08 39.77 outlier 0.100 0.872 0.833 0.156 0.588 0.775 0.872 0.393

[0132] Table 8 Simulation data of GCM+V control method

[0133] Evaluation indicators 1 2 3 4 5 6 7 8 Total time steps 4800 3397 4800 2611 4800 3473 3304 1963 Total distance traveled by the drone (m) 6058.0 3778.6 3549.9 2600.3 7890.6 4605.4 4024.9 2383.2 Average radius (m) 33.93 31.20 28.06 30.22 259.98 34.27 31.19 30.62 outlier 0.679 0.545 0.385 0.048 0.833 0.554 0.704 0.406

[0134] Continued from Table 8

[0135] Evaluation indicators 9 10 11 12 13 14 15 average value Total time steps 4800 4800 4800 2032 4068 4800 4800 2978 Total distance traveled by the drone (m) 3320.8 2451.7 3264.5 2706.3 3353.8 3746.4 4261.9 3350.4 Average radius (m) 28.27 26.17 28.63 31.62 29.33 27.62 30.46 31.21 outlier 0.205 0.013 0.269 0.273 0.288 0.372 0.385 0.402

[0136] Table 9 Simulation Experiment Data of Hierarchical Decision Control Method

[0137] Evaluation indicators 1 2 3 4 5 6 7 8 Total time steps 2186 2344 2114 2563 2223 4800 2214 2303 Total distance traveled by the drone (m) 3759 4058 3594 4413 3764 7997.1 3769 3971 Average radius (m) 30.82 29.6 30.83 32.37 31.59 272.26 30.51 31.35 outlier 0.114 0.342 0.412 0.463 0.167 0.859 0.114 0.297

[0138] Continued from Table 9

[0139] Evaluation indicators 9 10 11 12 13 14 15 average value Total time steps 2602 3521 2369 2197 2614 2081 3255 2470.2 Total distance traveled by the drone (m) 4421 5593 4022 3701 4484 3575.5 5434 4183 Average radius (m) 33.51 31.36 31.68 32.25 34.01 29.851 34.09 31.7 outlier 0.762 0.491 0.132 0.143 0.214 0.303 0.453 0.315

[0140] Table 10 Natural State Simulation Experiment Data

[0141] Evaluation indicators 1 2 3 4 5 6 7 8 Average radius (m) 35.36 28.75 34.74 34.13 31.81 43.23 32.82 29.61 outlier 0.063 0.219 0.188 0.781 0.594 0.875 0.406 0.313

[0142] Continued from Table 10

[0143] Evaluation indicators 9 10 11 12 13 14 15 average value Average radius (m) 30.92 29.41 30.91 28.77 31.35 31.75 35.38 32.60 outlier 0.375 0.250 0.000 0.125 0.438 0.281 0.219 0.342

[0144] First, based on the total time steps index in Tables 4 to 6, the success rate can be calculated. The success rate of the arc control method is 46.7%, the success rate of the GCM+V control method is 46.7%, and the success rate of the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision proposed in this invention is 93.3%.

[0145] The average values ​​of the evaluation index data in Tables 4 to 6 are plotted below as follows: Figures 14-18 As shown in the set of bar charts, it can be seen that the average time, individual out-of-group rate, and success rate of the UAV under the proposed adaptive hierarchical decision-based heterogeneous multi-agent navigation and obstacle avoidance control method are all better than the other two control methods. Although the total distance of the UAV does not reach the optimal level, it is better than the GCM+V control method, while the average cluster radius is basically consistent with the arc control method.

[0146] The three methods for plotting the UAV trajectory during the simulation process are as follows: Figures 19-21 As shown, when drones or swarms are blocked or interfered with by obstacles, only the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision proposed in this invention can effectively bypass the obstacles and drive the swarm to the target area.

[0147] The simulation results above verify that the heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making proposed in this invention is superior to the GCM+V control method and the arc control method in terms of average time, individual outlier rate and success rate, and is more suitable for complex environments with static obstacles.

[0148] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making, characterized in that, Heterogeneous multi-agent systems include drones and non-cooperative agent clusters. The non-cooperative agent clusters possess autonomous consciousness but cannot cooperate with drones, and include the following steps: S1. Obtain environmental feature information, process it based on PRM, and plan the cluster path; wherein, the cluster path is composed of the starting point, cluster path nodes and the center point of the target area connected in sequence; S2. The UAV interacts with the environment to obtain the feature information of the non-cooperative intelligent agent cluster and the feature information of obstacles, and determines whether the non-cooperative intelligent agent cluster has reached the target area; if so, the mission ends; otherwise, the UAV is controlled to move along the UAV path node, driving the non-cooperative intelligent agent cluster to move to the next cluster path node, and obtaining the information of the next cluster path node. S3. Combine the feature information of the non-cooperative intelligent agent cluster in S2, the feature information of the obstacles, and the information of the next cluster path node into state feature information, and input it into the second decision network; S4. The second decision network selects either the collect control mode or the drive control mode for the UAV based on the state feature information; in the collect control mode, the UAV path nodes are updated using the GCM-targeting trajectory generator; in the drive control mode, the state feature information is input into the first decision network, and the control parameters of the arc trajectory generator are output; the UAV path nodes are updated using the arc trajectory generator based on the control parameters and the state feature information. S5. Repeat S2~S4 until the non-cooperative agent cluster reaches the target area; In S4, updating the UAV path nodes using the GCM-targeting trajectory generator includes: The contour radius of the non-cooperative agent cluster is calculated; Compare the outline radius of the non-cooperative agent cluster with a preset outline radius threshold; if the outline radius of the non-cooperative agent cluster is greater than the preset outline radius threshold, generate a drone path node to drive the non-cooperative agent furthest from the center of the non-cooperative agent cluster toward the center; otherwise, generate a drone path node to make the drone follow and drive the non-cooperative agent cluster along the cluster path. Updating the UAV path nodes using an arc trajectory generator based on the control parameters and state characteristic information includes: Connect the drone's position with a straight line segment. Location relative to the target area Obtain the line segment From the perspective of drones, if not the center of a cooperative intelligent agent cluster Located on a straight segment If the left side is selected, a drone path node is generated to make the drone rotate clockwise around the non-cooperative agent cluster; otherwise, a drone path node is generated to make the drone rotate counterclockwise around the non-cooperative agent cluster.

2. The adaptive hierarchical decision based heterogeneous multi-agent navigation and obstacle avoidance control method according to claim 1, characterized in that, S1, based on PRM planning, the cluster paths include: Multiple sampling points are randomly generated within a two-dimensional planar region including the starting point and the target area; if an obstacle exists within the two-dimensional planar region and the sampling point coincides with the obstacle, the sampling point is removed. use The algorithm finds the shortest path through the sampling points to the target area as the cluster path; the cluster path does not pass through obstacles, and the sampling points on the cluster path are cluster path nodes.

3. The adaptive hierarchical decision based heterogeneous multi-agent navigation and obstacle avoidance control method according to claim 2, characterized in that, S4, the control parameters of the arc trajectory generator include a threshold value for causing the UAV to change the direction of rotation and the radius of curvature of the UAV trajectory .

4. The heterogeneous multi-agent navigation and obstacle avoidance control method based on adaptive hierarchical decision-making according to claim 3, characterized in that, The first decision network is obtained through pre-training of a neural network; the pre-training of the first decision network includes a first reward and punishment mechanism, which includes target point reward, cluster state penalty, and drone movement penalty; among which, Target point reward refers to the reward given when a drone moves along a drone path node and drives a non-cooperative intelligent agent swarm to reach the swarm path node that serves as the target point. Drone movement penalty refers to the penalty imposed on drone movement, used to limit energy consumption caused by drone movement; Cluster state penalties include discrete state penalties and velocity vector state penalties. Discrete state penalties are imposed when the contour radius of a non-cooperative agent cluster exceeds a preset contour radius threshold, or when an outlier appears. Velocity vector state penalties are imposed when the velocity vector offset angle exceeds a preset velocity vector offset angle threshold. The velocity vector offset angle is the angle difference between the cluster center velocity vector and the ideal velocity vector, where the ideal velocity vector is the vector pointing from the cluster center to the center of the target region.

5. The adaptive hierarchical decision based heterogeneous multi-agent navigation and obstacle avoidance control method according to claim 4, characterized in that, The second decision network pre-trains the neural network and sets up a forced learning reward and punishment mechanism; among which, The forced learning reward and punishment mechanism refers to the punishment imposed on a non-cooperative agent cluster if the second decision network fails to take correct measures when the cluster is in extreme states, including abnormal cluster radius and abnormal stagnation. Specifically, this includes: When the outline radius of the non-cooperative agent cluster is greater than the preset outline radius threshold, and the second decision network still does not use the collect method to gather the non-cooperative agent cluster, resulting in the outline radius of the non-cooperative agent cluster becoming larger and larger, the second decision network is penalized. When the contour radius of the non-cooperative agent cluster is less than the preset contour radius threshold, and the second decision network continues to use the collect method to gather the non-cooperative agent cluster, causing the non-cooperative agent cluster to stagnate within the preset time window threshold, the second decision network is penalized.

Images