Unmanned aerial vehicle cluster formation cooperative multi-target distribution method and system based on multi-agent layered reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster formation cooperative multi-target distribution method and system based on multi-agent layered reinforcement learning. Based on the idea of hierarchical reinforcement learning and multi-agent collaborative decision-making, a hierarchical reinforcement learning architecture of top-layer decision-making and bottom-layer execution is provided, a top-layer decision-making model is a single-agent system, the position, target location and transportation target information of an unmanned aerial vehicle cluster are analyzed, and task targets of all unmanned aerial vehicles are decided; the bottom layer execution model is a multi-agent system, and each agent decides a moving path according to the landform state observed by the agent. According to the method, the multi-target allocation task is clearly divided into two sub-tasks, namely task allocation and autonomous execution, so that the training time is shortened and the execution capability of the unmanned aerial vehicle cluster is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative planning, and particularly to a method and system for dynamic collaborative multi-objective allocation of unmanned aerial vehicle (UAV) swarm formation based on multi-agent reinforcement learning. Background Art

[0002] With the rapid development of intelligent agent devices such as UAVs, unmanned vehicles, and unmanned ships, heterogeneous intelligent agents have gradually acquired the ability to perform specific tasks in various environments. Through the participation of these intelligent agents, the labor cost can be significantly reduced, the work efficiency can be improved, and the personnel safety can be ensured. Currently, the control technology of a single intelligent agent (such as a UAV) has been relatively mature. However, there is still a great deal of room for research and development in the collaborative control technology of multi-intelligent agents (such as a UAV swarm).

[0003] The UAV formation control technology can efficiently complete complex tasks such as battlefield reconnaissance, forest fire rescue, and personnel search after a disaster. However, problems such as insufficient collaborative ability between intelligent agents and difficulty in coping with changes in dynamic and complex environments of control strategies limit the wide application of multi-agent technology.

[0004] To achieve the collaborative work of multi-intelligent agents, an important task is to solve the problems of data sharing and cooperation between intelligent agents. The current single-layer reinforcement learning technology has deficiencies in the collaborative planning of multi-intelligent agents. It is urgent to use hierarchical reinforcement learning to coordinate multiple intelligent agents so that they can flexibly cope with complex scenarios and propose solutions.

[0005] Hierarchical reinforcement learning for multi-agent tasks mainly makes full use of the idea of divide and conquer, divides the main task into multiple different sub-tasks, solves them within a smaller scale, trains them separately, and thus realizes the dimensionality reduction of the state space. The application of hierarchical reinforcement learning ideas in intelligent agents is mainly divided into two categories. The first category does not specify definite sub-tasks, and learns and generates corresponding sub-tasks by itself by referring to the hierarchical reinforcement learning mechanism based on Option, and completes the main task goal in stages. The second category is to determine the definite sub-task functions. For example, the multi-objective allocation task is divided into two sub-tasks: dividing the target and finding the target. After being fully trained by setting the reward function respectively, the top-layer output result is used as the input of the bottom layer and the bottom layer is trained again to obtain the final model. Summary of the Invention

[0006] Aiming at the problem that the UAV swarm is difficult to converge or requires too much computing resources when completing complex target allocation tasks in the prior art, a method for collaborative multi-objective allocation of UAV swarm formation based on multi-agent hierarchical reinforcement learning is provided. The top-layer decision-making model is trained by a single-agent algorithm and the bottom-layer collaborative control model is trained by a multi-agent algorithm. The multi-objective allocation task is clearly divided into two sub-tasks: task allocation and autonomous execution, reducing the training time and improving the execution ability of the UAV swarm.

[0007] According to one aspect of the specification of the present invention, a method for multi-agent hierarchical reinforcement learning-based collaborative multi-objective allocation of an unmanned aerial vehicle (UAV) swarm formation is provided. A hierarchical reinforcement learning architecture with top-level decision-making and bottom-level execution is constructed. Among them, the top-level decision-making model is a single-agent system, which analyzes the position of the UAV swarm, the target location, and the transportation target information, and decides the task objectives of each UAV; the bottom-level execution model is a multi-agent system, and each agent decides the movement path according to the terrain state observed by itself.

[0008] As a further technical solution, the method further includes:

[0009] Simulate and design a collaborative multi-objective allocation scenario for the UAV swarm. In the scenario, it is set that initially, the UAVs and the required transportation targets have no fixed deployment positions and are randomly generated in the scenario, while the target locations are fixedly generated.

[0010] As a further technical solution, the method further includes:

[0011] The top-level decision-making model divides tasks according to the global state information, allocates transportation targets to each UAV in the swarm, so as to generate a collaborative allocation scheme that meets the requirements;

[0012] The bottom-level execution model plans the real-time movement route for the UAVs according to the tasks divided by the top level for each UAV, the real-time perception situation of each of our devices, and the real-time position information of each UAV, so as to plan and generate the optimal transportation route for the UAVs;

[0013] The top-level decision-making model transmits the results of the task allocation of each UAV to the bottom level. Each UAV at the bottom level realizes transportation based on the top-level tasks and gives feedback to the top level, and returns the task completion results and the environmental state to the top level.

[0014] As a further technical solution, the method further includes:

[0015] The top-level decision-making model allocates tasks to each UAV according to the global state information and based on the set reward function, and uses a single-agent algorithm for learning.

[0016] As a further technical solution, the method further includes:

[0017] The bottom-level execution model executes the allocated tasks according to the local state information observed by the UAVs and based on the set reward function, and uses a multi-agent algorithm for learning.

[0018] As a further technical solution, the method further includes:

[0019] Train the top-level decision-making model using a single-agent algorithm, train the bottom-level execution model using a multi-agent algorithm, and construct multiple environments for parallel training; through the interaction between the agent and the environment, an experience pool is formed to support the training of the agent.

[0020] As a further technical solution, the training further includes:

[0021] For the top-level decision-making model, adopt a parallel training method, and at the same time collect the data pool of the experience pool by interacting with multiple parallel environments. Use the single-agent algorithm to extract the experience pool formed by different scenarios and use it to train the top-level decision-making model;

[0022] For the bottom-level execution model, adopt the same parallel training structure as the top-level decision-making model, and first set the drone to randomly select the transportation target for training. Use the multi-agent algorithm to extract the experience pool formed by different scenarios and use it to train the bottom-level execution model;

[0023] For overall training, after training the corresponding models separately at the top level and the bottom level, connect the top level and the bottom level and train them in the same simulation environment to obtain the final model.

[0024] According to one aspect of the specification of the present invention, there is provided a multi-agent hierarchical reinforcement learning unmanned aerial vehicle cluster formation collaborative multi-objective allocation system, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the multi-agent hierarchical reinforcement learning unmanned aerial vehicle cluster formation collaborative multi-objective allocation method described above.

[0025] According to one aspect of the specification of the present invention, there is provided a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the multi-agent hierarchical reinforcement learning unmanned aerial vehicle cluster formation collaborative multi-objective allocation method described above is implemented.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1) Aiming at the problem of random actions of the policy network of the multi-agent system in the initial stage, a hierarchical reinforcement learning method is proposed. Use the top-level decision-making single-agent model to allocate tasks for each unmanned aerial vehicle, reducing the problem of random actions of the policy network of the bottom-level multi-agent model in the initial stage;

[0028] 2) The method proposed by the present invention improves the ability of the unmanned aerial vehicle group to complete coupled tasks, enhances the target allocation ability and coordination ability of the unmanned aerial vehicle for dynamically changing environments, and improves the convergence and robustness of the multi-unmanned aerial vehicle system. Description of the Drawings

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0030] Figure 1 It is a flowchart of multi - target allocation for collaborative formation of an unmanned aerial vehicle (UAV) cluster using multi - agent hierarchical reinforcement learning according to an embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of the UAV simulation physical model designed in the embodiment of the present invention.

[0032] Figure 3 It is a structure diagram of the top - layer decision - making model in the embodiment of the present invention.

[0033] Figure 4 It is a structure diagram of the bottom - layer execution model in the embodiment of the present invention.

[0034] Figure 5 It is a diagram of the overall network framework in the embodiment of the present invention. Detailed implementation manners

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. In addition, the technical features in each embodiment or individual embodiment provided by the present invention can be combined with each other arbitrarily to form a new technical solution. This combination is not restricted by the order of steps and / or the mode of structural composition, but must be based on what can be achieved by those of ordinary skill in the art. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0036] The embodiment of the present invention provides a method for multi - target allocation of collaborative formation of an unmanned aerial vehicle cluster using multi - agent hierarchical reinforcement learning, which constructs a hierarchical reinforcement learning architecture of top - layer decision - making and bottom - layer execution. The overall framework is as Figure 5 shown. Among them, the top - layer decision - making model is a single - agent system, which analyzes the position of the UAV cluster, the target location, and the transportation target information, and decides the task objectives of each UAV; the bottom - layer execution model is a multi - agent system, and each agent decides the movement path according to the terrain state observed by itself.

[0037] Embodiment 1

[0038] like Figure 1 As shown, an embodiment of the present invention provides a multi-agent reinforcement learning method for drone cluster collaborative multi-target allocation, comprising the following steps:

[0039] Step 1: Simulate and design a drone cluster collaborative multi-target allocation scenario for round-based training of the hierarchical algorithm.

[0040] The main task of the present invention is to enable the drone to start from the starting point, collect the transportation targets scattered all over the map to the same target point, and multiple drones plan the optimal path to collect the transportation targets according to the top-level task allocation results, and avoid collision with obstacles during the following. Figure 2 The simulated environment map used in the present invention processes the map and obtains map data, such as obtaining obstacle information and passable road information.

[0041] In this example, there are three drones, and their physical properties are as follows:

[0042]

[0043] Where UAV is the state space of the UAV; is a set of drones, N1 is the number of drones, and in this embodiment, N1 is 3; is the attribute of the N1th drone. In this embodiment, the attributes of each drone in the drone group are the same; is the coordinate value of the N1th UAV; v s is the speed of the drone, in this embodiment, v s ∈[0,0.5]; a s is the acceleration of the drone, in this embodiment a s ∈[0,1];ta s Indicates the transport capacity of the drone. In this embodiment, s ∈[0,4];θ s is the deflection angle of the drone.

[0044] The speed and acceleration of the drone are set as the components on the coordinate axis at different times t. The component values ​​are determined by the components of the vector values ​​of the current speed v(t) and acceleration a(t) on the heading angle θ. The motion model in this example is:

[0045]

[0046] In the formula, Denote the velocity components of the UAV in the x and y directions at time t, which are determined by the components of the vector values of the velocity v(t) and acceleration a(t) of the UAV at time t on the heading angle θ; x(t) and y(t) are the coordinate components of the UAV or the movable target in the two-dimensional plane rectangular coordinate system.

[0047] Meanwhile, in this embodiment, 6 transportation targets and 1 target location are set, and their physical attributes are respectively represented as follows:

[0048]

[0049] In the formula, Obs is the state space of the transportation target; is the set of transportation targets, N2 is the number of transportation targets, and in this embodiment, N2 takes the value of 6; is the attribute of the N2th transportation target; is the coordinate value of the N2th transportation target, w N2 is the weight of each transportation target. In this embodiment, w s ∈ [0, 4].

[0050]

[0051] In the formula, Tar is the state space of the target location; is the set of target locations, N3 is the number of target locations, and in this embodiment, N3 takes the value of 1; is the attribute of the N3th target location, and its attributes include coordinates

[0052] Step 2: The top-level decision-making layer divides tasks according to the global situation information such as the target location and transportation target information, and combines the information such as the position distribution and transportation capacity of our UAVs to allocate transportation targets to each UAV in the cluster, so as to generate a collaborative allocation scheme that meets the requirements.

[0053] For the top-level decision-making model of this embodiment, it is necessary to use the UAV information, transportation target information, and obstacle information as inputs, select the target information that each UAV should transport, and use it as the output. Figure 3 is the structure diagram of the top-level decision-making model of this embodiment. The top-level decision-making model allocates tasks to each UAV according to the global state information and based on the set reward function, and uses the single-agent algorithm for learning. The reward function of the top-level decision-making model is designed as follows:

[0054]

[0055] In the formula, r1 is the total reward that the top-level decision-making model can obtain, r ta is the reward that can be obtained when each UAV is assigned a transportation task, is the number of the selected UAV, is the number of the selected transportation target. When the transportation capacity of the drone is greater than the weight of the transportation target, a positive reward can be obtained; otherwise, a negative reward is obtained; r d is the reward added considering the movement path of the drone. The farther the distance between the drone and the corresponding transportation target, the greater the negative reward obtained. β is the weight of the obtained negative reward, d ij is the distance between the selected drone and the corresponding transportation target.

[0056] Step 3: Based on the information such as the tasks assigned to each drone by the top layer, the real-time perception of the situation by our equipment, and the real-time positions of each drone, the underlying execution model plans the real-time movement route for the drone to generate the optimal transportation route of the drone.

[0057] For the underlying execution model of this embodiment, it is necessary to use the drone information, the coordinate information of the transportation target assigned by the top layer, and the obstacle information observed by the drone as inputs, and each drone outputs its movement direction and movement acceleration. Figure 4 is the structure diagram of the underlying execution model of this embodiment. The underlying execution model executes the assigned tasks according to the local state information observed by the drone and based on the set reward function, and uses the multi-agent algorithm for learning. The reward function of the underlying execution model is designed as follows:

[0058]

[0059] In the formula, r2 is the total reward of the underlying decision-making model, r t is the negative reward for the transportation time of the drone. The longer the transportation time, the more severe the punishment; r ds is the distance between the drone and the corresponding transportation target. The closer the distance, the greater the positive reward obtained; r v is the task completion reward. When the drone successfully transports the transportation target to the target location, a very large positive reward is obtained; otherwise, a very large negative reward is obtained if the task is not completed within the specified time.

[0060] Step 4: The top layer transmits the results of the task assignment of each drone to the lower layer. Each drone in the lower layer realizes transportation based on the top-layer task and gives feedback to the top layer, and returns the task completion result and the environmental state to the top layer.

[0061] For the top layer of this embodiment, the transmitted information includes the drone transportation task target, the task time limit, and the completion reward. Each aspect is represented as follows:

[0062] 1) The drone transportation target is represented as follows:

[0063]

[0064] Among them, represents the drone executing the task, Indicates the transportation target assigned to the UAV. Indicates the location information of the transportation target. Indicates the weight information of the transportation target. In this embodiment, all UAVs will be assigned transportation targets, and each UAV is assigned at most one transportation target. After any UAV completes the transportation, the top layer reassigns tasks to this UAV until all transportation targets have been transported to the target location.

[0065] 2) The time limit and completion reward are expressed as follows:

[0066] {"deadline":step,"r v ":reward}

[0067] Among them, "deadline":step represents the time limit from the top layer to the bottom layer. The bottom layer strategy needs to complete the task within the specified time step, otherwise the task fails. "r v ":reward represents the reward after the bottom layer completes the transportation task, and adjusts its behavior with this reward so that the bottom layer can better learn and complete the task.

[0068] For the feedback information generated after the bottom layer transmits information as the task ends in this embodiment, it includes task information and information of each UAV. The bottom layer feedback information is expressed as follows:

[0069] {"success":true,"steps_taken":steps,"UAV_position":(x UAV ,y UAV )}

[0070] Among them, "success":true represents the task completion result. "true" represents that the task is completed, that is, the UAV completes the transportation task within the time step specified by the top layer. Otherwise, the task fails and is marked as "false". "steps_taken":steps represents the specific time taken for the UAV to complete the transportation task, which is used for the task evaluation of the top layer. "UAV_position":(x UAV ,y UAV represents the specific position of the UAV that completed the task to assist the top layer in the next task assignment.

[0071] Step 5, construct multiple environments, use the single-agent algorithm to parallel train the top-layer decision-making model, and use the multi-agent algorithm to train the bottom-layer execution model at the same time.

[0072] For the model training process of this embodiment, it can be divided into top-layer training, bottom-layer training, and top-bottom comprehensive training. The specific training process is as follows:

[0073] 1) For the top - level decision - making model, a parallel training method is adopted. Meanwhile, by interacting with multiple parallel environments, the collection of the experience pool data pool is realized. The experience pool formed by different scenarios is extracted using a single - agent algorithm, and it is used to train the top - level decision - making model.

[0074] 2) For the bottom - level execution model, the same parallel training structure as the top - level decision - making model is adopted. First, it is set that the drones randomly select transportation targets for training. The experience pool formed by different scenarios is extracted using a multi - agent algorithm, and it is used to train the bottom - level execution model.

[0075] 3) Overall training: After training the corresponding models separately at the top - level and bottom - level, the top - level and bottom - level are connected and trained in the same simulation environment to obtain the final model that can adapt to more complex environments.

[0076] Embodiment 2

[0077] Based on the same inventive concept, the present invention also provides a multi - agent reinforcement learning unmanned aerial vehicle (UAV) cluster formation collaborative multi - target allocation system, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a multi - agent reinforcement learning UAV cluster collaborative multi - target allocation method as described above.

[0078] Embodiment 3

[0079] Based on the same inventive concept, the present invention also provides a multi - agent reinforcement learning UAV cluster formation collaborative multi - target allocation system, including a readable storage medium. A computer program is stored on the readable storage medium, and when the computer program is executed, it implements a multi - agent reinforcement learning UAV cluster formation collaborative multi - target allocation method as described above.

[0080] Specifically in implementation, the method proposed by the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. The system devices for implementing the method, such as a computer - readable storage medium storing the corresponding computer program of the technical solution of the present invention and a computer device including running the corresponding computer program, should also be within the protection scope of the present invention.

[0081] In summary of the above embodiments, based on the hierarchical reinforcement learning and multi - agent collaborative decision - making ideas, the present invention proposes a hierarchical reinforcement learning architecture with top - level decision - making and bottom - level execution. The top - level decision - making model is a single - agent system, which analyzes the position of the UAV cluster, the target location, and the transportation target information, and decides the task objectives of each UAV. The bottom - level execution model is a multi - agent system, and each agent decides the movement path according to the terrain state observed by itself. The present invention clearly divides the multi - target allocation task into two subtasks: task allocation and autonomous execution, reducing the training time and enhancing the execution ability of the UAV cluster.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target allocation method for drone swarm formation coordination based on multi-agent hierarchical reinforcement learning, characterized in that: A hierarchical reinforcement learning architecture with top-level decision-making and bottom-level execution was constructed. The top-level decision-making model is a single-agent system, which analyzes the location, target location and transportation target information of the drone cluster and decides the mission objectives of each drone. The bottom-level execution model is a multi-agent system, and each agent decides the movement path based on the terrain conditions observed by itself.

2. According to claim 1, the multi-agent hierarchical reinforcement learning method for drone swarm formation collaborative multi-target allocation is characterized in that: The method further comprises: The simulation designs a scenario of coordinated multi-target allocation of drone swarms. At the beginning of the scenario setting, the drones and the required transportation targets have no fixed deployment locations, they are randomly generated in the scenario, and the target locations are fixedly generated.

3. The multi-target allocation method for drone swarm formation based on multi-agent hierarchical reinforcement learning according to claim 2 is characterized in that: The method further comprises: The top-level decision model divides tasks according to the global state information and assigns transportation targets to each drone in the cluster to generate a collaborative allocation plan that meets the requirements; The bottom-level execution model divides each drone's tasks according to the top-level, the real-time situation perception of each device and the real-time location information of each drone, and plans the movement route for the drone in real time to generate the optimal transportation route for the drone; The top-level decision model transmits the results of each drone's task allocation to the bottom level. The bottom-level drones realize transportation based on the top-level tasks and give feedback to the top level, returning the task completion results and environmental status to the top level.

4. The multi-target allocation method for drone swarm formation coordination based on multi-agent hierarchical reinforcement learning according to claim 3 is characterized in that: The method further comprises: The top-level decision model assigns tasks to each drone based on global state information and a set reward function, and uses a single-agent algorithm for learning.

5. The multi-target allocation method for drone swarm formation coordination based on multi-agent hierarchical reinforcement learning according to claim 3 is characterized in that: The method further comprises: The underlying execution model assigns tasks based on the local state information observed by the drone and the set reward function, and uses a multi-agent algorithm for learning.

6. The multi-target allocation method for drone swarm formation based on multi-agent hierarchical reinforcement learning according to claim 1 is characterized in that: The method further comprises: The top-level decision model is trained using a single-agent algorithm, the bottom-level execution model is trained using a multi-agent algorithm, and multiple environments are constructed for parallel training. Through the interaction between the agent and the environment, an experience pool is formed to support the training of the agent.

7. The multi-target allocation method for drone swarm formation coordination based on multi-agent hierarchical reinforcement learning according to claim 6 is characterized in that: The training also includes: For the top-level decision model, a parallel training method is adopted. At the same time, the experience pool data pool is collected by interacting with multiple parallel environments. The experience pool formed by different scenarios is extracted using a single-agent algorithm, and used to implement the training of the top-level decision model. For the bottom-level execution model, the same parallel training structure as the top-level decision model is adopted. The drone is first set to randomly select transportation targets for training. The multi-agent algorithm is used to extract the experience pool formed by different scenarios and use it to implement the training of the bottom-level execution model. Overall training, after the top and bottom layers are trained separately to obtain the corresponding models, the top and bottom layers are connected and trained in the same simulation environment to obtain the final model.

8. A multi-agent hierarchical reinforcement learning drone swarm formation collaborative multi-target allocation system, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a multi-agent hierarchical reinforcement learning method for drone cluster formation collaborative multi-target allocation as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed, it implements the drone cluster formation collaborative multi-target allocation method of multi-agent hierarchical reinforcement learning as described in any one of claims 1-7.

Images