Unmanned aerial vehicle-unmanned vehicle hybrid clustering reinforcement learning optimization method and system

Abstract

The invention discloses an unmanned aerial vehicle-unmanned vehicle hybrid clustering reinforcement learning optimization method and system, and relates to the technical field of path planning, and the method comprises the following steps: obtaining a target point set, distributing the target point set to a region, and carrying out the optimization of the target point density of the target point set based on a modified mean shift clustering algorithm. Dividing the area into circular areas; deploying the UAV and the UGV into a circular area, and after the UAV reaches the area, deploying the UAV by the UGV to move until the UAV reaches a specified target point; wherein the UAV and the UGV are trained on the basis of a multi-agent depth deterministic strategy gradient algorithm, so that the path of the UAV and the path of the UGV are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning, and in particular to a method and system for optimizing hybrid clustering reinforcement learning of unmanned aerial vehicles and unmanned vehicles. Background Art

[0002] Drones can offer the advantages of high altitude and wide field of view, while UGVs have longer battery duration and can perform precision operations on ground objects. However, UGVs are limited by limited vertical reach and UAVs are limited by limited power supply, resulting in limited operational range and duration. However, in a collaborative system, UAVs can reach destinations that UGVs cannot, UGVs can transport UAVs to save battery life, and are better able to handle ground objects with greater payload capacity. Collaborative UAV-UGV systems show promise in intelligence surveillance, reconnaissance, target positioning, and critical path planning in obstructed environments.

[0003] In the existing technology, research has explored the deployment of multiple drones or unmanned vehicles to perform specific complex tasks. Some scholars have designed a UAV-UGV collaborative system for planetary exploration, whose goal is to assign a drone to an unmanned vehicle in an environment without any obstacles to reach a set of target points while minimizing the travel distance. Drones play an important role in planning paths and guiding collaborative systems, using imitation-enhanced deep reinforcement learning (IADRL) to solve the huge complexity of drone-unmanned vehicle collaboration. Although the work discussed above has helped to improve and overcome many challenges in drone-unmanned vehicle combination hybrid systems, none of them consider the collaboration of multiple drones and unmanned vehicles in obstacle environments, which enhances the variability of vehicles and the robustness of the system. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a UAV-UAV hybrid clustering reinforcement learning optimization method and system.

[0005] In the first aspect, the purpose of the present invention can be achieved by the following technical solution: a hybrid clustering reinforcement learning optimization method for unmanned aerial vehicles and unmanned vehicles, the method comprising the following steps:

[0006] Obtain a set of target points, assign the set of target points to regions, and divide the regions into circular regions based on the modified mean shift clustering algorithm and the target point density of the set of target points;

[0007] The UAV and UGV are deployed in a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm, thereby optimizing the paths of the UAV and UGV.

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the modified mean shift clustering algorithm is as follows:

[0009] Set the radius R of the UAV flight area, the set P of target points in the environment, Q is a copy of the target point set, the preset maximum number of iterations is k, and the moving threshold is T;

[0010] Create an empty set Z;

[0011] When there are still unassigned target points in the environment, a point is randomly selected from the set P as the initial center point of the current clustering window;

[0012] Create a clustering window centered on the initial point and calculate the mean of all points in the window;

[0013] Through the iterative process, the center point of the clustering window is continuously moved to the area with higher density. The process is realized by calculating the mean of all points in the window. When the moving distance of the center of the clustering window is less than the preset threshold or the maximum number of iterations is reached, the iteration is stopped.

[0014] Add the final stable clustering window center point to the regional center point set Z, remove all points within the clustering window from the set P, and mark them as assigned;

[0015] For the remaining unassigned target points in the environment, repeat the steps to assign them until all target points are assigned to the corresponding areas.

[0016] At the end of the algorithm, the center point set Z of all circular areas is returned.

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of deploying the UAV and the UGV to the circular area:

[0018] Deploy UAVs and UGVs to various regions and form multiple alliances;

[0019] The number of alliances K sent to a region;

[0020] The trained drone and unmanned vehicle models cover the target, where the alliance is considered available when all deployed drones land on available unmanned vehicles after clearing the area;

[0021] When a region is cleared, the previously allocated K coalitions are available and then allocated to the next target region. ;

[0022] Repeat until all areas are cleared and all goals are reached.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: forming multiple alliances is an alliance for deploying multi-agent deep reinforcement learning training in each area;

[0024] Constraints of Multi-agent Deep Reinforcement Learning Training, F a and L g UAV Self-driving cars The path length is shortened by the following formula while keeping the UAV and the unmanned vehicle safe and free from collision, expressed as:

[0025]

[0026] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the multi-agent deep deterministic policy gradient algorithm includes: state action representation, training process and reward calculation process.

[0027] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the action of the state is represented as follows:

[0028] Drone The action space is defined as ξ a =(u a , v a ), driverless cars The action space is defined as ξ g (y g , v g );

[0029] Drone The state space is defined as O a , driverless car The action space is defined as O g O g ;

[0030] The state space O of a specific autonomous vehicle g g It is composed of the position relative to all targets in the ground environment, the position relative to all obstacles in the environment, the speed of the unmanned vehicle g itself, its own speed, and the position of other unmanned vehicles relative to the unmanned vehicle g;

[0031] The state space O of a specific drone a aIt is composed of the positions of all targets and obstacles, the positions of other drones relative to drone a, the speed of drone a and its own speed, the relative position of the unmanned vehicle closest to drone a, and a binary variable indicating whether drone a has reached the specified target. When the drone has not reached the specified target, the value is 0, and when it has reached the specified target, the variable is 1; if this notifies the drone that it has reached the specified target, it will proceed to land on the unmanned vehicle.

[0032] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the training process is as follows:

[0033] Suppose there are N agents in each training scenario, each with its own deterministic strategy μ i , these strategies are parameterized as θ = {θ1, θ2, ..., θ N}, for agent i, the deterministic policy gradient μ i Written as:

[0034]

[0035] Where x represents the state space {o1, o2, ..., o N}, represents the value function Q, a i is the action of agent i, o i is the observation of each agent i, is the experience record of all agents (x, x′, a1, a2, ..., a N , r1, r2, ... r N ); when all agents have completed their actions, x′ represents the new state of the environment, r i is the reward obtained by agent i; the loss function It is expressed as:

[0036]

[0037] in,

[0038]

[0039] Initialize the number of training cycles to E, determine the number of time steps in each cycle to T, initialize the number of agents to N, and initialize the experience replay buffer

[0040] For each agent, initialize a random process For action exploration, the initial state is received as x′. For each time step, each agent selects an action based on the current strategy and exploration Execute all actions a=(a1,a2,...,a N ) and observe the corresponding reward r and state x′ and store (x, a, r, x′) in the experience replay buffer

[0041] From the Experience Replay Buffer Randomly sample small batch data samples S(x, a, r, x′) in

[0042]

[0043] Update the critic network by minimizing the loss function, Equation (10);

[0044] Use the data S sampled from the experience replay buffer to update the actor network through equation (9);

[0045] Repeat the above process until all training cycles are completed;

[0046] Update the target network parameters of agent i

[0047] θ′ i ←τθ i +(1-τ)θ′ i (13)

[0048] Value Function Used to update the Critic network And the feedback goes back to help train the model.

[0049] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the reward calculation process is as follows:

[0050] Unmanned vehicle reward components,The detailed calculation of different reward components in the unmanned vehicle reward structure is determined as follows:

[0051] Reward based on target distance: The reward component r1 is distributed according to the distance between all targets and agents. The distance w from all unmanned vehicle agents to each ground target is calculated and stored in a set d w Among all the values, the minimum distance min(d w ) is the distance of the nearest unmanned vehicle to each ground target w; after calculation, the minimum values ​​are summed and this negative reward component is assigned to each agent:

[0052]

[0053] Rewards based on agent-agent collisions: A reward component r2 is derived, and agents are penalized for being within collision distance of each other;

[0054] Consider a region around each agent, named the danger zone, with width σ V With δ V , represents the maximum distance at which collisions occur between agents; whenever the danger zones of two agents overlap or collide, that is, the distance d between the agents V <(δ V +σ V ), it is a false collision. According to the size of the overlap of the dangerous areas of the two agents, a negative reward is given to the false collision agent. The more overlapping areas, the greater the magnitude of the negative reward. If the agents collide, the agents receive the maximum negative reward value and record the collision information for evaluation. The reward value is estimated as follows:

[0055] r2=r2-T2(δ V +σ V -d V )

[0056] Reward based on vehicle-obstacle collision: Consider another reward component r3 when the agent collides with an obstacle; there is a danger zone σ near the obstacle O , when a collision occurs, the maximum distance between the agent and the obstacle is δ O ; When the danger zone overlaps, that is, the distance between the agent and the obstacle, d o <(δ o +σ o ), add negative rewards to the colliding agent; r3 is determined as follows,

[0057] r3=r3-T3(δ O +σ O -d O )

[0058] Unmanned vehicle R G The reward function of is estimated as follows:

[0059] r G =r1+r2+r3

[0060] This reward is assigned to each self-driving car using the MADDPG framework.

[0061] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: a drone reward component:

[0062] The distance-to-target reward objective encourages a reward component r4, which adds the distance from each target (p∈P) to the nearest agent. The distance value is converted into a negative reward to form a reward component; when the drone reaches its designated target, r4 becomes zero.

[0063]

[0064] Reward based on agent-agent collision: There is a danger zone σ near each agent V , when two dangerous areas overlap or cause a false collision, a negative reward component r5 is added to the agent reward; the reward part r5 is determined as follows:

[0065] r5=r5-T2(δ V +σ V -d V )

[0066] Reward mechanism based on agent-obstacle collision: using the danger zone σ O The reward component r6 of the unmanned vehicle environment is calculated by the following formula: r6 = r6-T2(δ O +σ O -d O )

[0067] Returns the reward for the nearest unmanned vehicle: Another reward component r7 is added for drones; if a drone does not reach the specified target, it will receive a negative reward r T ; After reaching the target, the reward is based on the distance d between the drone and its nearest unmanned vehicle h is replaced by the reward value;

[0068] r7=r7-T1(d h )

[0069] The final reward of the drone is calculated by adding together each reward component. The reward function R for each agent is A Determine as follows:

[0070] r A =r4+r5+r6+r7

[0071] After calculation, use this reward to train the drone.

[0072] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses a UAV-UAV hybrid clustering reinforcement learning optimization system, comprising:

[0073] A region division module is used to obtain a target point set, assign the target point set to a region, and divide the region into circular regions based on a modified mean shift clustering algorithm combined with the target point density of the target point set;

[0074] The path optimization module is used to deploy the UAV and UGV into a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm to optimize the paths of the UAV and UGV.

[0075] Beneficial effects of the present invention:

[0076] The present invention divides the target into multiple areas and deploys an alliance of unmanned vehicles and drones through one-to-many collaboration between unmanned vehicles and drones. This method of allocating targets to various circular areas based on density and distance significantly reduces the time required to reach these targets. At the same time, the application of heuristic algorithms in MEANCRFT simplifies the complexity of the regional segmentation and path planning tasks of the target location. We use the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to train the UAV-UGV alliance to minimize the time step and avoid collisions while reaching all targets in an obstructed environment. The present invention improves the efficiency and success rate of target search, reduces energy waste, and achieves effective path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a schematic flow chart of the method of the present invention;

[0079] Figure 2 It is a two-dimensional visualization diagram of the relationship between the key elements of the UAV-UAV network of the present invention;

[0080] Figure 3 is a flow chart of the present invention for deploying a UAV-UAV alliance;

[0081] Figure 4 It is a schematic diagram of collision between intelligent agents of the present invention;

[0082] Figure 5 is a schematic diagram of a collision between an intelligent agent of the present invention and an obstacle;

[0083] Figure 6 It is a schematic diagram of the system structure of the present invention;

[0084] Figure 7 Schematic diagram of the effect of changing the number of targets in the environment of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Embodiment 1:

[0087] like Figure 1 As shown, a hybrid clustering reinforcement learning optimization method for drones and unmanned vehicles includes the following steps:

[0088] S101: Acquire a target point set, assign the target point set to a region, and divide the region into circular regions based on a modified mean shift clustering algorithm and the target point density of the target point set;

[0089] In order to divide the area, the improved mean shift clustering algorithm is used to set the radius R of the UAV flight area, the set P of target points in the environment, Q is a copy of the target point set, the maximum number of iterations is preset to k, and the moving threshold is T.

[0090] Create an empty set Z to store the center point of each circular area.

[0091] When there are still unassigned target points in the environment, randomly select a point from the set P as the initial center point of the current clustering window. Create a clustering window centered on the initial point and calculate the mean of all points in the window.

[0092] Through the iterative process, the center point of the clustering window is continuously moved to the area with higher density. This process is achieved by calculating the mean of all points in the window. When the moving distance of the center of the clustering window is less than the preset threshold or the maximum number of iterations is reached, the iteration stops.

[0093] The final stable clustering window center point is added to the regional center point set Z, and all points within the clustering window are removed from the set P and marked as assigned. For the remaining unassigned target points in the environment, the initial point is selected and the above iterative process is performed until all target points are assigned to the corresponding regions.

[0094] At the end of the algorithm, the center point set Z of all circular areas is returned.

[0095] like Figure 2As shown in Figure 1, in a typical scenario, when there are a large number of targets available, the coalition flying from one target to another becomes much less efficient when we consider the number of times the drones have to leave the vehicle, reach the target, and then return to the vehicle. Since the number of aerial targets can vary, selecting the nearest aerial target and continuing to move to the next one requires a lot of time and battery. Our modified mean shift clustering method saves a lot of time and battery by dividing the available targets into regions based on density, which is a big advantage of our proposed solution. Each region has a radius R on the flight range V of the drones. This allows the coalition to arrive at a specific area and deploy drones where they can move freely to reach available targets without having to return to the drone. This increases the number of targets reached while reducing the time required to complete the mission.

[0096] like Figure 3 As shown, after grouping all targets into circular regions, we deploy a coalition trained with multi-agent deep reinforcement learning in each region to reach the target while avoiding collisions.

[0097] First, the number K of coalitions is sent to a region.

[0098] Next, the trained drone and vehicle models cover the target. The coalition is considered available only when all deployed drones land on available vehicles after clearing the area.

[0099] When training, the unmanned vehicle is trained first, because the only function of the unmanned vehicle is to reach the specified target. For actions such as drone landing, it only needs to suspend its function and does not need training. On the other hand, the drone needs to reach the specified target and return to the nearest unmanned vehicle. This makes the task more complicated and also requires information about the nearest unmanned vehicle in its observation space. We first train the unmanned vehicle and use this training model to obtain some unmanned vehicle coordinates in the environment. Next, in the drone training scenario, we use these demonstration coordinates as input to the drone's observation space. In this way, the first half of the drone training is to reach the specified target. The next half is to "land" on the nearest unmanned vehicle, whose position is provided by our demonstration data. In this way, we divide the training into two parts, using our unmanned vehicle training model to train our drone. Finally, these two independent training models are combined in a full simulation environment.

[0100] There, the available targets are first divided into zones. Then, only the trained AV models move the coalition to the designated zone. After reaching the zone, the AV deploys the drones and together they move through the environment to reach all targets in the zone. The AVs will navigate the environment to reach the designated targets, avoiding obstacles and other AVs, while the drones will also move through the environment, avoiding obstacles and other drones to reach the designated aerial targets. When the drones reach their respective targets, they move to the nearest AV location. The selected landing AV pauses its model until the drone lands. Afterwards, it resumes its model to reach the ground targets.

[0101] Constraints of Multi-agent Deep Reinforcement Learning Training, F a and L g UAV Self-driving cars The goal of this step is to shorten the distance as much as possible while keeping the UAV and the unmanned vehicle safe and free from collision, which can be expressed as:

[0102]

[0103] While achieving the only goal, we need to follow certain constraints. The following are the constraints that need to be met:

[0104]

[0105] The goal of this step is to minimize the path length F of all UAVs a and the path length L of all unmanned vehicles g The sum of is shown in Eq. (1). At a given time, each UAV and UGV is assigned an aerial target and a ground target respectively. The constraint in Eq. (2) specifies the position of each aerial target (x p ,y p ) are all on the path F of the drone a Coverage, similarly, Eq. (3) specifies that each ground target (x p ,y p ) are all on the path L of the unmanned vehicle g In addition, the paths of the UAV and the unmanned vehicle are represented as F a and L g , under the constraints of equations (4) and (5), it does not collide with any obstacle (x p ,y p ). The constraints of Eq. (6) and Eq. (7) represent the Do not interact with another drone Collision, at the same time, each unmanned vehicle Do not interact with another autonomous vehicle Collision. The constraint in Eq. (8) states that the drone must return to the nearest unmanned vehicle at the same position as the unmanned vehicle at step n, which is the last step of an episode.

[0106] Then, when a region is cleared, the previously allocated K coalitions are available and then allocated to the next target region. Finally, the above steps are repeated until all regions are cleared and all targets are reached.

[0107] S102: Deploy the UAV and UGV into a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm, thereby optimizing the paths of the UAV and UGV.

[0108] The Multi-Agent Deep Deterministic Policy Gradient algorithm (MADDPG) is an Actor-Critic, model-free algorithm based on deterministic policy gradients. It can operate on a continuous action space, can cope with environmental complexity, and has a certain degree of stability and data efficiency.

[0109] Drone The action space is defined as ξ a =(u a , a ), driverless cars The action space is defined as ξ g (y g , g )

[0110] Drone The state space is defined as O a , driverless car The state space is defined as O g

[0111] The state space O of a specific unmanned vehicle gg g It is composed of the position relative to all targets in the ground environment, the position relative to all obstacles in the environment, the speed of the unmanned vehicle gg itself and its own speed, and the position of other unmanned vehicles relative to the unmanned vehicle g;

[0112] The state space O of a specific drone a aIt consists of the positions of all targets and obstacles, the positions of other drones relative to drone a, drone a’s own speed and its own speed, the relative position of the unmanned vehicle closest to drone a, and a binary variable indicating whether drone a has reached the specified target. The value is 0 when the drone has not reached the specified target, and 1 when it has reached the specified target. This notifies the drone that it has reached the specified target and it can now begin the process of landing on the unmanned vehicle. Both of these are complementary to the state space of the drone agent. These inputs are used to train the model in different environments. After the training phase is completed, the trained model will be deployed in a combined environment of UGV and drone collaboration to reach the target in an obstacle environment;

[0113] Assume that there are N agents in each training scenario, each with its own deterministic strategy μ i , these strategies can be parameterized as θ = {θ1, θ2, ..., θ N}, for agent i, the deterministic policy gradient μ i Can be written as:

[0114]

[0115] Where x represents the state space {o1, o2, ..., o N}, represents the value function Q, a i is the action of agent i, o i is the observation of each agent i, is the experience record of all agents (x, x′, a1, a2, ..., a N , r1, r2, ... r N ); when all agents have completed their actions, x′ represents the new state of the environment, r i is the reward obtained by agent i. Loss function It can be expressed as:

[0116]

[0117] in,

[0118]

[0119] a. Initialize the number of training cycles to E, determine the number of time steps in each cycle to T, initialize the number of agents to N, and initialize the experience replay buffer

[0120] b. For each agent, initialize a random process For action exploration, which helps the agent to explore during training, receiving the initial state x′, for each time step, each agent selects an action based on the current strategy and exploration Execute all actions a=(a1,a2,...,a N ) and observe the corresponding reward r and state x′ and store (x, a, r, x′) in the experience replay buffer

[0121] c. Replaying from the Experience Buffer Randomly sample a small batch of data samples S(x, a, r, x′) from

[0122]

[0123] Update the critic network by minimizing the loss function, Eq. (10),

[0124] Use the data S sampled from the experience replay buffer to update the actor network through equation (9);

[0125] d. Repeat the above process until all training cycles are completed;

[0126] e .Update the target network parameters of agent i

[0127] θ′ i ←τθ i +(1-τ)θ′ i (13)

[0128] Value Function Used to update the Critic network And the feedback goes back to help train the model.

[0129] Agents are trained in their respective environments using multi-agent reinforcement learning. In such a setting, an appropriate reward structure is necessary to obtain acceptable behavior from the agents (i.e., drones and unmanned vehicles in the respective environments). In the developed drone and unmanned vehicle environments, multiple homogeneous agents are trained to reach the available goals in a satisfactory manner. In order to obtain such an output, we have to come up with a reward structure that encourages multiple agents to reach their goals efficiently. At the same time, the agents also need to avoid any obstacle collisions on the way to the goal. For this purpose, the reward value assigned to each agent is divided into some components, which are affected by some parameters of the environment and are different for each environment type. In order to encourage the agent to reach the goal, the distance between the agent and the goal is taken into account. In order to avoid collisions, the distance between the intelligent sports agents and the distance between the agent and the obstacles are also considered. It should be noted that for the various reward components, a function T is used. I , which takes a distance as input and returns a reward value

[0130] Unmanned vehicle reward components,The detailed calculation of different reward components in the unmanned vehicle reward structure can be determined as follows:

[0131] a. Reward based on target distance: In each type of environment, there may be multiple targets and multiple agents, where the agents need to work together to find a way to effectively reach all available targets. To this end, the reward component r1 is distributed according to the distance between all targets and agents. Specifically, the minimum distance between available targets and agents is considered. In order to distribute such a reward, first, the distance from all unmanned vehicle agents to each ground target is calculated and stored in a set d w Among all these values, the minimum distance min(d w ) is the distance of the nearest unmanned vehicle to each ground target. After calculation, we sum the minimum values ​​and assign this negative reward component to each agent.

[0132]

[0133] b. Rewards based on agent-agent collisions: Agents must avoid collisions with other agents when navigating the environment. To this end, we derive a reward component r2 where agents are penalized for being within collision distance of each other.

[0134] Here, we consider a region around each agent and name it the danger zone, with a width of σ V With δ V , which represents the maximum distance at which collisions between agents will occur. Figure 3An example scenario of the danger zones of agents is given. Whenever the danger zones of two agents overlap or collide, that is, the distance d between the agents V <(δ V +σ V ), we call it a false collision. Depending on the size of the overlapping area, a negative reward is given to the false collision agent. The more overlapping areas there are, the larger the magnitude of the negative reward. If the agent does collide, the agent gets the maximum negative reward value and the collision information is recorded for evaluation. The reward value can be estimated as follows:

[0135] r2=r2-T2(δ V +σ V -d V ) (15)

[0136] c. Reward based on vehicle-obstacle collision: Consider another reward component r3 for the collision between the agent and the obstacle. It works similarly to the agent-agent collision component, where, like the agent, there is also a danger zone σ near the obstacle O ,like Figure 4 When a collision occurs, the maximum distance between the agent and the obstacle is δ O When the danger zone overlaps, that is, the distance between the agent and the obstacle, d o <(δ o +σ o ), adding a negative reward to the agent that collided. This value also increases with the length of the overlapping area. This reward value. r3 can be determined as follows,

[0137] r3=r3-T3(δ O +σ O -d O ) (16)

[0138] After calculating each reward component for each unmanned vehicle agent, their sum is the final guidance reward for guiding the agent through obstacles to reach the goal. G The reward function of is estimated as follows:

[0139] r G =r1+r2+r3 (17)

[0140] This reward is assigned to each self-driving car using the MADDPG framework.

[0141] Drone reward components. The training environment is divided based on the type of vehicle being trained. However, since the training environments for drones and unmanned vehicles are similar, the reward structure for drones is almost similar to that for unmanned vehicles. The drone agent reward is divided into several components. The description and calculation methods of these components are given below.

[0142] a. Reward based on target distance: Similar to the self-driving car environment, we have a target encouragement reward component r4, which sums the distance of each target (p∈P) to the nearest agent. This distance value is converted into a negative reward to form the reward component. It should be noted that for the drone scenario, when the drone reaches its designated target, r4 becomes zero, and the rest of the reward components remain active.

[0143]

[0144] b. Rewards based on agent-agent collisions: Agent-agent collisions are more severe for aerial vehicles because they are more sensitive to collisions than ground vehicles. To this end, the same danger zone concept is adopted from unmanned vehicles. There is a danger zone σ near each agent. V , when two danger zones overlap or cause a false collision, a negative reward component r5 is added to the agent reward. This component helps the drones learn to avoid collisions with each other while navigating. The reward component r5 can be determined as follows,

[0145] r5=r5-T2(δ V +σ V -d V ) (19)

[0146] c. Reward mechanism based on agent-obstacle collision: In an obstacle environment, the drone also needs to pass through obstacles to reach the target safely. To this end, we use a reward mechanism similar to the danger zone σ O The reward component r6 of the unmanned vehicle environment provides a negative reward based on the overlapping area between the agent and the obstacle. The calculation formula is as follows:

[0147] r6=r6-T2(δ O +σ O -d O ) (20)

[0148] d. Reward for returning to the nearest unmanned vehicle: Although the two environments have very similar functionality, the key difference in the drone environment is that the drone needs to return to the nearest unmanned vehicle after reaching the target. To encourage this behavior, another reward component r7 is added for the drone. Here, if a drone does not reach the designated target, it receives a negative reward R T After reaching the target, the reward is calculated based on the distance d between the drone and its nearest unmanned vehicle. h The reward value is replaced.

[0149] r7=r7-T1(d h ) (twenty one)

[0150] The final reward of the drone is calculated by adding together each reward component. The reward function R for each agent is A It can be determined as follows:

[0151] r A =r4+r5+r6+r7 (22)

[0152] After calculation, we use this reward to train the drone.

[0153] Embodiment 2: In the second aspect, as Figure 6 As shown, in order to achieve the above-mentioned purpose, the present invention discloses a UAV-UAV hybrid clustering reinforcement learning optimization system, comprising:

[0154] A region division module 11 is used to obtain a target point set, assign the target point set to a region, and divide the region into circular regions based on a modified mean shift clustering algorithm combined with the target point density of the target point set;

[0155] The path optimization module 12 is used to deploy the UAV and UGV into a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm, thereby optimizing the paths of the UAV and UGV.

[0156] Specifically, we created a simulated training environment based on OpenAI's Gym platform. It was created to simulate an environment with a fixed number of obstacles, in which a collaborative alliance of UAVs and UGVs will be deployed to plan a reasonably safe and fast path to some goal. We assumed that the area of ​​the developed custom environment was 2000×2000 m 2 . At the beginning of each episode, the positions of the agent, the goal, and the obstacles are randomly distributed according to a uniform random distribution to ensure robustness in working on different environments. For the region radius, we used 25m, as it is a convenient distance for the size of the vehicle and the obstacles we chose. Essentially, in each episode, a new map is given for training, which ensures the randomization of the training process. Unmanned vehicles navigate on the ground to reach their destination. They only need movements along 2 axes to move on the surface they are on. For this reason, we chose a 2D environment to train the UGV agent. Here, we use 0.01 as the learning rate for the actor and critic networks, and consider using a batch size of 1024 to sample the memory buffer. The discount factor γ is set to 0.99, and 0.01 is used as the value of data τ. The following table shows the training parameter values ​​of the MEANCRFT-MADDPG system.

[0157] Table 2. Training parameters used in MEANCRFT-MADPG

[0158]

[0159] Figure 7 The effect of changing the number of targets in the environment is shown. In this experiment, the number of drones and unmanned vehicles is fixed to 8 and 2 respectively, and the radius of each area is set to 25m. The error bars are contained in Figure 7 In all the graphs, we use the lowest and highest values ​​of all simulation results as the lowest and highest points of the error bars of a metric, respectively. The results show that the values ​​in each direction do not exceed 5% of the mean, which verifies the robustness of our results.

[0160] Figure 7 (a) shows the number of steps required to reach the goal when the number of goals in the environment changes. Here, we see that MEANCRFT-MADDPG outperforms COPO, re-marl, and MAPPO. In addition, the CRFT graph without partitioning seems to be similar to MEANCRFT-MADDPG. Figure 1 This is because, in order to calculate the number of steps per episode, the agent needs to complete the episode by reaching all goals. CRFT without partitioning prefers to reach all goals in one episode only when the randomly generated goals are very close to each other and the number of goals is minimal. In this case, the battery limitation problem of the drone does not harm its performance. In all other cases, the agent cannot reach all available goals due to the limited number of steps, which represents the limited battery of the drone.

[0161] Figure 7 (b) shows that the number of collisions per 1000 episodes increases as the number of targets in the environment increases. This is because when the number of targets is the least, most drones will not be deployed in action due to the number of aerial targets being less than the number of available drones. While the number of drones and unmanned vehicles in the environment is constant, the increase in the number of targets enables the unmanned vehicles to deploy a larger number of drones to reduce the time of the entire event. This results in an increase in the number of collisions as the number of agents increases. MADDPG also easily beats other methods in terms of collisions.

[0162] Figure 7(c) Mission completion rate as a percentage, where the mission is considered complete only when all targets are reached and all drones return to the stationary vehicle. Here, the partition-free approach fails because as the number of targets increases, the approach cannot determine how to get the deployed drones back to their respective vehicles, which can be very far away, causing the drones to lose battery and fail to complete the mission. MEANCRFT-MADDPG works very well, with an almost perfect success rate for lower numbers of targets, but still maintains impressive values ​​when the number of targets increases. DRE-MARL and COPO maintain decent performance and perform almost similarly in terms of mission completion. MEANCRFT-MAPPO essentially fails as the number of targets increases. In our experiments, we realized that the policy approach of MEANCRFT-MAPPO works well when the environment is simple. It starts to perform worse as the number of agents and the complexity of the environment increase. In the developed system, we tried to fine-tune the environment as well as other parameters; however, the difference was not significant compared to MEANCRFT-MADDPG.

[0163] Figure 7 (d) depicts the goal attainment accuracy, which is very similar to the previous metrics. The difference is that here we consider reaching all the individual goals in an episode instead of completing the task as in the previous metrics. We calculate the ratio of the number of goals reached by a method to the total number of goals in an episode. MEANCRFT-MADDPG maintains a consistently high accuracy, which is more pronounced as the goal count increases. Its performance does drop slightly as the number of goals increases, but it still maintains an accuracy of over 95%. This shows that the developed MEANCRFT-MADDPG method is robust to changes in the goal count compared to traditional methods. MEANCRFT-MAPPO performs poorly due to the higher complexity of the environment and its accuracy also decreases as the goal count increases. CRFT without regions starts out with an accuracy of 80%, but due to its limited reachability, its performance drops drastically as it cannot reach most of the goals. The other two baseline algorithms, COPO and DRE-MARL, perform almost similarly but still fall short compared to MADDPG.

[0164] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0165] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0167] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A hybrid clustering reinforcement learning optimization method for drones and unmanned vehicles, characterized in that: The method comprises the following steps: Obtain a set of target points, assign the set of target points to regions, and divide the regions into circular regions based on the modified mean shift clustering algorithm and the target point density of the set of target points; The UAV and UGV are deployed in a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm, thereby optimizing the paths of the UAV and UGV.

2. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 1, characterized in that: The modified mean shift clustering algorithm is as follows: Set the radius R of the UAV flight area, the set P of target points in the environment, Q is a copy of the target point set, the preset maximum number of iterations is k, and the moving threshold is T; Create an empty set Z; When there are still unassigned target points in the environment, a point is randomly selected from the set P as the initial center point of the current clustering window; Create a clustering window centered on the initial point and calculate the mean of all points in the window; Through the iterative process, the center point of the clustering window is continuously moved to the area with higher density. The process is realized by calculating the mean of all points in the window. When the moving distance of the center of the clustering window is less than the preset threshold or the maximum number of iterations is reached, the iteration is stopped. Add the final stable clustering window center point to the regional center point set Z, remove all points within the clustering window from the set P, and mark them as assigned; For the remaining unassigned target points in the environment, repeat the steps to assign them until all target points are assigned to the corresponding areas; At the end of the algorithm, the center point set Z of all circular areas is returned.

3. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 1, characterized in that: The process of deploying UAVs and UGVs to a circular area: Deploy UAVs and UGVs to various regions and form multiple alliances; The number of alliances K sent to a region; The trained drone and unmanned vehicle models cover the target, where the alliance is considered available when all deployed drones land on available unmanned vehicles after clearing the area; When a region is cleared, the previously allocated K coalitions are available and then allocated to the next target region; Repeat until all areas are cleared and all goals are reached.

4. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 3 is characterized in that: The forming of multiple alliances is an alliance for deploying multi-agent deep reinforcement learning training in each region; Constraints of Multi-agent Deep Reinforcement Learning Training, F a and L g UAV Self-driving cars The path length is shortened by the following formula while keeping the UAV and the unmanned vehicle safe and free of collision, expressed as:

5. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 1, characterized in that: The multi-agent based deep deterministic policy gradient algorithm includes: state action representation, training process and reward calculation process.

6. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 5, characterized in that: The actions of the state are expressed as follows: Drone The action space is defined as ξ a =(u a , v a ), driverless cars The action space is defined as Drone The state space is defined as O a , driverless car The action space is defined as O g ; The state space O of a specific autonomous vehicle g g It is composed of the position relative to all targets in the ground environment, the position relative to all obstacles in the environment, the speed of the unmanned vehicle g itself, its own speed, and the position of other unmanned vehicles relative to the unmanned vehicle g; The state space O of a specific drone a a It is composed of the positions of all targets and obstacles, the positions of other drones relative to drone a, the speed of drone a and its own speed, the relative position of the unmanned vehicle closest to drone a, and a binary variable indicating whether drone a has reached the specified target. When the drone has not reached the specified target, the value is 0, and when it has reached the specified target, the variable is 1; if this notifies the drone that it has reached the specified target, it will proceed to land on the unmanned vehicle.

7. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 6, characterized in that: The training process is as follows: Suppose there are N agents in each training scenario, each with its own deterministic strategy μ i , these strategies are parameterized as θ = {θ1, θ2, ..., θ N }, for agent i, the deterministic policy gradient μ i Written as: Where x represents the state space {o1, o2, ..., o N }, represents the value function Q, a i is the action of agent i, o i is the observation of each agent i, is the experience record of all agents (x, x′, a1, a2, ..., a N , r1, r2, ... r N ) replay; when all agents have completed their actions, x′ represents the new state of the environment, r i is the reward obtained by agent i; the loss function It is expressed as: in, Initialize the number of training cycles to E, determine the number of time steps in each cycle to T, initialize the number of agents to N, and initialize the experience replay buffer For each agent, initialize a random process For action exploration, the initial state is received as x′. For each time step, each agent selects an action based on the current strategy and exploration Execute all actions a=(a1,a2,...,a N ) and observe the corresponding reward r and state x′ and store (x, a, r, x′) in the experience replay buffer From the Experience Replay Buffer Randomly sample small batch data samples S(x, a, r, x′) in Update the critic network by minimizing the loss function, Equation (10); Use the data S sampled from the experience replay buffer to update the actor network through equation (9); Repeat the above process until all training cycles are completed; Update the target network parameters of agent i θ′ i ←tth i +(1-τ)θ′ i (13) Value Function Used to update the Critic network And the feedback goes back to help train the model.

8. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 7, characterized in that: The reward calculation process is as follows: Unmanned vehicle reward components,The detailed calculation of different reward components in the unmanned vehicle reward structure is determined as follows: Reward based on target distance: The reward component r1 is distributed according to the distance between all targets and agents. The distance w from all unmanned vehicle agents to each ground target is calculated and stored in a set d w Among all the values, the minimum distance min(d w ) is the distance of the nearest unmanned vehicle to each ground target w; after calculation, the minimum values ​​are summed and this negative reward component is assigned to each agent: Rewards based on agent-agent collisions: A reward component r2 is derived, and agents are penalized for being within collision distance of each other; Consider a region around each agent, named the danger zone, with width σ V With δ V , represents the maximum distance at which collisions occur between agents; whenever the danger zones of two agents overlap or collide, that is, the distance d between the agents V <(δ V +σ V ), it is a false collision; according to the size of the overlap area of ​​the dangerous areas of the two agents, a negative reward is given to the false collision agent. The more overlapping areas, the greater the magnitude of the negative reward; if the agents collide, the agents receive the maximum negative reward value and record the collision information for evaluation; the reward value is estimated as follows: r2=r2-T2(δ V +s V -d V ) Reward based on vehicle-obstacle collision: Consider another reward component r3 when the agent collides with an obstacle; There is a dangerous area near the obstacle σ O , when a collision occurs, the maximum distance between the agent and the obstacle is δ o ; When the danger zone overlaps, that is, the distance between the agent and the obstacle, d o <(δ o +σ o ), add negative rewards to the colliding agent; r3 is determined as follows, r3=r3-T3(δ O +σ O -d O ) Unmanned vehicle R G The reward function of is estimated as follows: r G ＝r1+r2+r3 This reward is assigned to each self-driving car using the MADDPG framework.

9. The method for optimizing hybrid clustering of unmanned aerial vehicles and unmanned vehicles according to claim 8, characterized in that: Drone Bonus Components: The distance-to-target reward objective encourages a reward component r4, which adds the distance from each target (p∈P) to the nearest agent. The distance value is converted into a negative reward to form a reward component; when the drone reaches its designated target, r4 becomes zero. Reward based on agent-agent collision: There is a danger zone σ near each agent V , when two dangerous areas overlap or cause a false collision, a negative reward component r5 is added to the agent reward; the reward part r5 is determined as follows: r5=r5-T2(δ V +s V -d V ) Reward mechanism based on agent-obstacle collision: using the danger zone σ O The reward component r6 of the unmanned vehicle environment provides a negative reward based on the overlapping area between the agent and the obstacle. The calculation formula is as follows: r6=r6-T2(δ O +σ O -d O ) Returns the reward for the nearest unmanned vehicle: Another reward component r7 is added for drones; if a drone does not reach the specified target, it will receive a negative reward r T ; After reaching the target, the reward is based on the distance d between the drone and its nearest unmanned vehicle h is replaced by the reward value; <h2 style=";text-align:left;direction:ltr">r7 = r7 - T1(d<h2 style=";text-align:left;direction:ltr"> h <h2 style=";text-align:left;direction:ltr"> ) The final reward of the drone is calculated by adding together each reward component. The reward function R of each agent is A Determine as follows: r A ＝r4+r5+r6+r7 After calculation, use this reward to train the drone.

10. A UAV-UAV hybrid clustering reinforcement learning optimization system, characterized in that: include: A region division module is used to obtain a target point set, assign the target point set to a region, and divide the region into circular regions based on a modified mean shift clustering algorithm combined with the target point density of the target point set; The path optimization module is used to deploy the UAV and UGV into a circular area. After arriving at the area, the UGV deploys the UAV to move until it reaches the designated target point; wherein the UAV and UGV are trained based on a multi-agent deep deterministic policy gradient algorithm to optimize the paths of the UAV and UGV.

Images