Unmanned aerial vehicle group dynamic cooperative control method and system based on agent reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle group dynamic cooperative control method and system based on agent reinforcement learning, and the method comprises the steps: decomposing a cooperative task which needs to be carried out by an unmanned aerial vehicle into a plurality of sub-tasks, and carrying out the modeling of the sub-tasks as a Markov decision process; and carrying out dynamic cooperative control on the unmanned aerial vehicle group by using a multi-commentator structure depth deterministic strategy gradient MCDDPG algorithm to complete a cooperative task of the unmanned aerial vehicle group. Aiming at the problem that an action strategy network of a multi-agent system randomly acts at an initial stage, an MCDDPG algorithm of a multi-commentator architecture is provided and is used for jointly optimizing and solving an optimal action strategy network. For the weight distribution optimization problem of a plurality of reviewer modules in the training process, a dynamic difficulty priority algorithm is adopted, and dynamic distribution of weight values of all the modules is achieved. According to the method provided by the invention, the capability of completing the coupling task by the unmanned aerial vehicle group is improved, the cooperative capability of the unmanned aerial vehicles for a dynamic change environment is enhanced, and the convergence and robustness of a multi-unmanned aerial vehicle system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative planning, and in particular to a method and system for dynamic collaborative control of a swarm of unmanned aerial vehicles based on agent reinforcement learning. Background Art

[0002] With the continuous development of intelligent equipment (such as drones, unmanned vehicles, unmanned ships, etc.), various heterogeneous intelligent agents have the ability to carry out designated tasks in different environments. With the assistance of intelligent agents, manpower consumption can be greatly reduced and personal safety can be improved. Intelligent agents have strong information acquisition capabilities. At present, the development of single drone control technology is relatively complete, but the research on collaborative control technology of multi-drone cluster formations still has a lot of room for development. UAV formation control technology can quickly and efficiently complete complex coupling tasks such as battlefield environment exploration, forest fire rescue, and post-disaster personnel search, but it has poor collaborative ability and control strategies that are difficult to adapt to dynamically changing complex environments, which makes multi-agent control technology difficult to popularize in practical applications. Therefore, how to enable intelligent agents to share data with each other and work together to complete work tasks is an urgent problem to be solved in the field of multi-agent collaborative planning. Summary of the invention

[0003] Aiming at the problem that the existing UAV swarm is difficult to converge or requires too much computing resources to converge when completing complex real-world tasks, a UAV swarm dynamic collaborative control method based on agent reinforcement learning is provided, which includes the following steps:

[0004] Step 1: Set the physical properties of the drone, obstacles, and targets according to the drone's flight mission;

[0005] Step 2: Set the kinematic equations of the drone and the target, establish the correlation between velocity, acceleration and direction of motion, and set constraints to comply with the rules of action in the physical world;

[0006] Step 3: Decompose the collaborative tasks that the drones need to carry out into multiple subtasks, and model the subtasks as Markov decision processes. Use the multi-critic structured deep deterministic policy gradient MCDDPG algorithm to dynamically coordinate the drone swarm and complete the collaborative tasks of the drone swarm.

[0007] Step 3.1, input the environmental status information obtained by the drone into the action strategy network, and the action strategy network outputs the action that each drone should take in the current environment;

[0008] Step 3.2, use the critic module to evaluate the current action, and use the dynamic difficulty priority algorithm to optimally allocate the weights of different critic modules;

[0009] Step 3.3, based on the evaluation value of the critic module, the action policy network is optimized using the deep deterministic policy gradient algorithm;

[0010] In step 3.4, the trained action strategy network is deployed on each UAV to guide the UAV to complete the collaborative task.

[0011] Furthermore, the flight mission in step 1 is that the drone tracks the dynamic target from the starting point, and multiple drones form a coordinated encirclement formation around the target, and avoid collision between the drones and obstacles during the tracking process, as well as collision between the drones;

[0012] The physical properties of the drone are represented as follows:

[0013]

[0014] Where UAV is the state space of the UAV; is the set of drones, N1 is the number of drones; is the attribute of the N1th UAV; is the coordinate value of the N1th UAV; r s is the safe flight radius of the UAV; v s is the speed of the drone; a s is the acceleration of the drone; c s Indicates whether drones can communicate with each other; θ s is the heading angle of the UAV.

[0015] The physical properties of obstacles are expressed as follows:

[0016]

[0017] Where Obs is the state space of the obstacle; is the obstacle set, N2 is the number of obstacles; is the attribute of the N2th obstacle; is the coordinate value of the N2th obstacle; r c is the radius of the obstacle.

[0018] The physical properties of the moving target are expressed as follows:

[0019]

[0020] Where Tar is the state space of the moving target; is the set of mobile targets, N3 is the number of mobile targets; The properties of the N3th moving target, including the target coordinates Target collision radius r t 、Target speed v t 、Target acceleration at 、Target deflection angle θ t .

[0021] Furthermore, the kinematic equations of the UAV and the mobile target in step 2 are:

[0022]

[0023] In the formula, It represents the velocity components of the UAV or mobile target in the x and y directions at time t, which is determined by the components of the vector values ​​of the velocity v(t) and acceleration a(t) of the UAV or mobile target at the heading angle θ at time t; x(t) and y(t) are the coordinate components of the UAV or mobile target in the two-dimensional rectangular coordinate system.

[0024] The constraints defining the properties of drones and mobile targets are:

[0025]

[0026] In the formula, v represents the speed of the drone or moving target, v min Indicates the minimum speed of the drone or moving target, v max represents the maximum speed of the drone or mobile target, a represents the acceleration of the drone or mobile target, and a min Indicates the minimum acceleration of the drone or moving target, a max represents the maximum acceleration of the drone or mobile target, (x,y) represents the position of the drone or mobile target, (x min ,x max ) represents the moving range of the drone or mobile target on the x-axis, (y min ,y max ) represents the moving range of the UAV or moving target on the y-axis.

[0027] Furthermore, in step 3.1, the drone inputs the acquired environmental information, target information and self-state information into the action strategy network, and the action strategy network outputs the action that each drone should take in the current environment, including speed v, acceleration a and direction angle θ. Environmental information includes the position and radius of obstacles, as well as the position, speed, acceleration and direction angle of other drones. Target information includes the position, collision radius, speed, acceleration and deflection angle of the target. The drone's own state information includes the drone's position, speed, acceleration and direction angle. The action strategy network uses a multi-layer perceptron neural network.

[0028] Furthermore, there are two critic modules in step 3.2. The first critic module is composed of a deep neural network architecture based on a multilayer perceptron, including three fully connected layers. The state information and action information of the drone are converted into a one-dimensional vector and input into the multilayer perceptron deep neural network. Under the network structure composed of the fully connected layer and the activation function, the evaluation value of the current action is output; wherein the state information is composed of the drone's own state information, environmental information and target information. The drone's own state information includes the drone's position, speed, acceleration and direction angle. The environmental information includes the position and radius of obstacles, as well as the position, speed, acceleration and direction angle of other drones. The target information includes the target's position, collision radius, speed, acceleration and deflection angle. The action information is the speed, acceleration and heading angle of the drone output by the action strategy network.

[0029] The second critic module is the gravitational field model, which is a mathematical model expressed as: Q f (s′,a′)=-f(s′)[1-cos(χ)], where Q f (s′, a′) is the evaluation value of the gravitational field model for the action decision a′ taken by the current UAV under the acquired state information s′, f(s′) is the resultant force formed by the attraction of the target to the UAV and the repulsion of the obstacle to the UAV under the current state information s′, χ is the angle between the acceleration direction and the resultant force f; when the acceleration direction of the UAV is consistent with the direction of the resultant force, χ = 0, and the evaluation value Q is f (s′, a′) = 0, the action decision in the current state will obtain a high reward value, otherwise, χ≠0, the corresponding evaluation value Q f (s′,a′)<0, then the action decision will obtain a low reward value.

[0030] The weight of the critic module is dynamically adjusted according to the ratio of the difference between the evaluation value of the critic module and the reward value obtained by the drone. The weight β1 and β2 of each critic module are calculated as follows:

[0031]

[0032]

[0033] y i =R i +γ1Q′(s i ' +1 ,μ′(s i ' +1 |θ′)|w′) (10)

[0034] Where N is the number of drones, L1 represents the error between the evaluation value and reward value of the drone group by the first critic module composed of a deep neural network, and y i represents the reward value obtained by the i-th drone under the current action, Q1(s i ′,a i ′|w) represents the first critic module's response to the state information s obtained by the i-th UAV i 'The action taken is a i ′, w is the parameter weight in the neural network of the first critic module, L2 represents the evaluation value of the second critic module composed of the gravitational field model on the matching degree between the acceleration direction of the current action of the drone group and the direction of the resultant force, Q2(s i ′,a i ′) is the second critic module’s response to the state information s obtained by the i-th UAV i 'The action a taken i ′, γ1 represents the distance discount factor, Q′(s i ' +1 ,μ′(s i ' +1 |θ′)|w′) represents the evaluation result of the two critic modules on the reward obtained by the current action at the next moment, s i ' +1 represents the state information at the next moment, μ′(s i ' +1 |θ′) represents the action strategy network at the next moment, θ′ represents the network parameters in the action strategy network at the next moment, w′ represents the network parameters of the first critic module at the next moment, R i is the actual reward value obtained by the i-th drone under the current action. The specific calculation formula is as follows:

[0035] R i =r 1i +r 2i +r 3i +r 4i (11)

[0036] r 2i = -0.1 × steps (12)

[0037] r 3i =0.1×d i (13)

[0038] In the formula, r 1i The reward value obtained by the i-th UAV for completing the coordinated encirclement task of the target; r 2iis the penalty value obtained by the i-th UAV for failing to complete the coordinated encirclement task of the target, steps refers to the total number of actions used to execute the task; r 3i is the reward value obtained by the i-th UAV when it moves towards the target, d i represents the distance that the i-th UAV moves to the target point; if the i-th UAV collides with an obstacle, collides with other UAVs, or collides with the boundary of the environment during driving, the reward value r 4i = -1, otherwise r 4i =0.

[0039] Furthermore, in step 3.3, the action selection of the action policy network is jointly evaluated by the multi-critic module, and the action policy network is optimized using the deep deterministic policy gradient algorithm according to the evaluation function. The evaluation function is calculated as follows:

[0040] Q=β1Q1+β2Q2 (14)

[0041] β1+β2=1 (15)

[0042] Where Q represents the evaluation value of the current UAV action, Q1 and Q2 are the evaluation values ​​of the current UAV action by the first and second critic modules, respectively, and β1 and β2 are the weights of the first and second critic modules, respectively.

[0043] The deep deterministic policy gradient descent function is:

[0044]

[0045] In the formula, is the objective function J(μ θ )’s gradient value; β1 and β2 are the weights of the first and second critic modules respectively; is the objective function of the first critic module defined according to the evaluation function; When the current environment information, target information and the drone's own state information are input into the action strategy network, the action strategy network outputs the action gradient of the action; The error gradient change between the action evaluation value and the reward value obtained by inputting the current action and the environment information, target information, and the drone's own state information into the first critic module composed of a multi-layer perceptron is used to guide the current action; is the objective function of the second critic module defined according to the evaluation function; It is the error gradient between the current action acceleration direction and the resultant force direction.

[0046] Assume that the current action leads to the gradient If the value of the action increases, it means that the action is not desirable in the current environment; otherwise, it is desirable. The ultimate goal is to make the objective function J(μ θ )The gradient during the update process ▽ θ J(μ θ ) is 0, that is, the action strategy is improved by optimizing the objective function through policy gradient.

[0047] The present invention also provides a dynamic collaborative control system for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning, which is used to implement a dynamic collaborative control method for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning as described above.

[0048] Moreover, it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the above-mentioned method of dynamic collaborative control of a drone swarm based on intelligent agent reinforcement learning.

[0049] Alternatively, it includes a readable storage medium, on which a computer program is stored, and when the computer program is executed, the above-mentioned method for dynamic collaborative control of a drone swarm based on intelligent agent reinforcement learning is implemented.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] 1) Aiming at the problem of random actions in the initial stage of the action policy network of the multi-agent system, a deep deterministic policy gradient algorithm with a multi-critic architecture is proposed to jointly optimize and solve the optimal action policy network;

[0052] 2) For the weight allocation optimization problem of multiple critic modules during the training process, a dynamic difficulty priority algorithm is used to achieve dynamic allocation of weight values ​​for each module;

[0053] 3) The method proposed in the present invention improves the ability of drone groups to complete coupling tasks, enhances the coordination ability of drones in dynamically changing environments, and improves the convergence and robustness of multi-drone systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is a flow chart of dynamic collaborative control of drone swarms based on agent reinforcement learning in an embodiment of the present invention.

[0056] Figure 2 This is a structural diagram of the MCDDPG algorithm proposed in the present invention.

[0057] Figure 3 This is a structural diagram of the multi-critic module in the MCDDPG algorithm proposed in this invention.

[0058] Figure 4 This is a graph showing the convergence results of the action strategy network learning of the drone swarm under the MCDDPG algorithm.

[0059] Figure 5(a)-Figure 5(c) Figure 5(a) is a visualization of the coupled task execution process of the UAV swarm dynamically following the target from multiple angles, where Figure 5(b) is the target tracking subtask, Figure 5(c) is the obstacle avoidance subtask, and Figure 5(c) is the collaborative encirclement subtask. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the drawings and embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] Example 1

[0062] like Figure 1 As shown, an embodiment of the present invention provides a method for dynamic cooperative control of a drone swarm based on agent reinforcement learning, comprising the following steps:

[0063] Step 1: Set the physical properties of the drone, obstacles, and targets according to the drone's flight mission.

[0064] The main task of the present invention is to enable the drone to track the dynamic target from the starting point. After catching up with the dynamic target, multiple drones will cooperate to surround the dynamic target to obtain image information from multiple angles. After that, the drone group will keep the surrounding formation and move with the dynamic target, and complete the path planning of the drone group during the tracking process to avoid collision with obstacles and collision between drones. Assuming that the drones, obstacles and targets interact with each other at the same height, in this embodiment, there are 3 drones, 1 obstacle and 1 target. The physical properties of the three are respectively expressed as follows:

[0065]

[0066] Where UAV is the state space of the UAV; is a set of drones, N1 is the number of drones, and in this embodiment, N1 is 3; is the attribute of the N1th drone. In this embodiment, the attributes of each drone in the drone group are the same; is the coordinate value of the N1th UAV; r s is the safe flight radius of the UAV, which is 0.035m in this embodiment; v s is the speed of the drone, in this embodiment, v s ∈[0,0.5m / s]; a s is the acceleration of the drone, in this embodiment a s ∈[0,1m / s 2 ];c s Indicates whether drones can communicate with each other; θ s is the heading angle of the UAV, in this embodiment, θ s ∈[0,360°].

[0067]

[0068] Where Obs is the state space of the obstacle; is the obstacle set, N2 is the number of obstacles, and in this embodiment, N2 is 1; is the attribute of the N2th obstacle; is the coordinate value of the N2th obstacle; r c is the radius of the obstacle.

[0069]

[0070] Where Tar is the state space of the target; is the target set, N3 is the number of targets, and in this embodiment, N3 is 1; The attributes of the N3th target. When the target moves, its attributes include coordinates Target collision radius r t 、Target speed v t 、Target acceleration a t 、Target deflection angle θ t In this embodiment, r t The value is 0.35m, v t ∈[0,0.5m / s], a t ∈[0,1m / s 2 ],θ t ∈[0,360°].

[0071] Step 2, set the kinematic equations of the drone and the target, establish the correlation between velocity, acceleration and direction of motion, and set constraints to conform to the action rules of the physical world.

[0072] The kinematic equations of the UAV and the moving target are:

[0073]

[0074] In the formula, It represents the velocity components of the UAV or mobile target in the x and y directions in the two-dimensional rectangular coordinate system at time t, which is determined by the components of the vector values ​​of the velocity v(t) and acceleration a(t) of the UAV or mobile target at the heading angle θ at time t; x(t) and y(t) are the coordinate components of the UAV or mobile target in the two-dimensional rectangular coordinate system.

[0075] The constraints defining the properties of drones and mobile targets are:

[0076]

[0077] In the formula, v represents the speed of the drone or moving target, v min Indicates the minimum speed of the drone or moving target, v max represents the maximum speed of the drone or mobile target, a represents the acceleration of the drone or mobile target, and a min Indicates the minimum acceleration of the drone or moving target, a max represents the maximum acceleration of the drone or mobile target, (x,y) represents the position of the drone or mobile target, (x min ,x max ) represents the moving range of the drone or mobile target on the x-axis, (y min ,y max ) represents the moving range of the drone or mobile target on the y-axis. In this embodiment, the speed and acceleration limit ranges are consistent with the attributes in step 2, that is, the speed does not exceed 0.5 and the acceleration does not exceed 1. The task environment in the embodiment is a 1000×1000 range space, and the upper and lower limits of the coordinate axis are -500 to 500.

[0078] Step 3: Decompose the collaborative tasks that the drones need to carry out into multiple subtasks, and model the subtasks as Markov decision processes. Use the Multi-Critic Deep Deterministic Policy Gradient (MCDDPG) algorithm to dynamically coordinate and control the drone swarm to complete the collaborative tasks of the drone swarm.

[0079] In step 3.1, various information obtained by the drone is input into the action strategy network, and the action strategy network outputs the action that each drone should take in the current environment.

[0080] Figure 2The diagram is a structural diagram of the MCDDPG algorithm. The drone obtains environmental information (the position and radius of obstacles, the position, speed, acceleration and direction angle of other drones) and target information (the position, collision radius, speed, acceleration and deflection angle of the target) from the mission scene, and inputs it together with its own state information (the position, speed, acceleration and direction angle of the drone itself) into the action strategy network. The action strategy network outputs the action that each drone should take in the current environment. This action will further affect the environment to change accordingly. The two critic modules do not participate in the interaction of the environment, but only output an evaluation of the current action. The evaluation value is used to reflect the quality of the action under the current environmental information, and to guide the action strategy network to update the weight parameters and bias parameters, so that the action strategy network can output actions with higher reward values. The action strategy network uses a multi-layer perceptron neural network, and the output actions include speed v, acceleration a and heading angle θ.

[0081] In step 3.2, the critic module is used to evaluate the current action, and the dynamic difficulty priority algorithm is used to optimally allocate the weights of different critic modules.

[0082] The specific structures of the two critic modules are as follows: Figure 3 As shown in the figure, the first critic module is composed of a deep neural network architecture based on a multi-layer perceptron, which contains three fully connected layers and is used to evaluate the value of the actions taken by the drone in the current state. By converting the state information and action information into a one-dimensional vector and inputting it into the multi-layer perceptron deep neural network, the evaluation value of the current action is output under the network structure composed of fully connected layers and activation functions, so as to obtain the deep relationship between the drone action and the environment and other drones. The state information is composed of the drone's own state information (the drone's own position, speed, acceleration and direction angle), environmental information (the position and radius of obstacles, the position, speed, acceleration and direction angle of other drones) and target information (the target's position, collision radius, speed, acceleration and deflection angle), and the action information is the speed, acceleration and heading angle of the drone output by the action strategy network. This critic module is mainly used to evaluate the action decision of the drone group to complete the subtask of forming an encirclement formation and tracking dynamic targets together.

[0083] The second critic module is the gravitational field model, which is a mathematical model expressed as: Q f (s′,a′)=-f(s′)[1-cos(χ)], where Q f (s′, a′) is the evaluation value of the gravitational field model for the action decision a′ taken by the current UAV under the acquired state information s′, f(s′) is the resultant force formed by the attraction of the target on the UAV and the repulsion of the obstacle on the UAV under the current state information s′, and χ is the angle between the acceleration direction and the resultant force f. Figure 3 The blue dot represents the target, and the pink dot represents the drone. When the acceleration direction of the drone is consistent with the direction of the resultant force, χ = 0, and the evaluation value Q f (s′, a′) = 0, the action decision in the current state will obtain a high reward value, otherwise, χ≠0, the corresponding evaluation value Q f (s′,a′)<0, then the action decision will receive a low reward value. The critic module mainly evaluates the action decisions of the drone to complete the obstacle avoidance and tracking subtasks.

[0084] When the action behavior is more consistent with the output of the layer perceptron deep neural network, the reward value under the first critic module is higher; when the action behavior is more consistent with the gravitational field model, the reward value under the second critic module is higher. In order to learn more appropriate actions in appropriate scenarios, different weights are assigned to each critic module. In order to make the weight value distribution reasonable, the dynamic difficulty priority algorithm is used to solve the optimal distribution of weight values ​​and improve the algorithm capabilities.

[0085] The weight of the critic module is dynamically adjusted according to the ratio of the difference between the evaluation value of the critic module and the reward value obtained by the drone. The weight β1 and β2 of each critic module are calculated as follows:

[0086]

[0087] y i =R i +γ1Q′(s i ' +1 ,μ′(s i ' +1 |θ′)|w′) (10)

[0088] Where N is the number of drones, L1 represents the error between the evaluation value and reward value of the drone group by the first critic module composed of a deep neural network, and y i represents the reward value obtained by the i-th drone under the current action, Q1(s i ′,a i ′|w) represents the first critic module's response to the state information s obtained by the i-th UAV i 'The action taken is a i ′, w is the parameter weight in the neural network of the first critic module, L2 represents the evaluation value of the second critic module composed of the gravitational field model on the matching degree between the acceleration direction of the current action of the drone group and the direction of the resultant force. Its value is a fixed value and does not need to be trained and updated; Q2(s i ′,a i ′) is the second critic module’s response to the state information s obtained by the i-th UAV i'The action taken is a i ′, γ1 represents the distance discount factor, which is used to weaken the impact of the current action on the evaluation of subsequent actions, Q′(s i ' +1 ,μ′(s i ' +1 |θ′)|w′) represents the evaluation result of the two critic modules on the reward obtained by the current action at the next moment, s i ' +1 represents the state information at the next moment, μ′(s i ' +1 |θ′) represents the action strategy network at the next moment, θ′ represents the network parameters in the action strategy network at the next moment, w′ represents the network parameters of the first critic module at the next moment, R i is the actual reward value obtained by the i-th drone under the current action. The specific calculation formula is as follows:

[0089] R i =r 1i +r 2i +r 3i +r 4i (11)

[0090] r 2i = -0.1 × steps (12)

[0091] r 3i =0.1×d i (13)

[0092] In the formula, r 1i The reward value obtained by the i-th UAV for completing the coordinated encirclement task of the target is set manually according to the actual situation. In this embodiment, it is set to 10; r 2i is the penalty value obtained by the i-th UAV for failing to complete the coordinated encirclement task of the target, steps refers to the total number of actions used to execute the task; r 3i is the reward value obtained by the i-th UAV when it moves towards the target, d i represents the distance that the i-th UAV moves to the target point; if the i-th UAV collides with an obstacle, collides with other UAVs, or collides with the boundary of the environment during driving, the reward value r 4i = -1, otherwise r 4i =0.

[0093] In step 3.3, based on the evaluation value of the critic module, the action policy network is optimized using the deep deterministic policy gradient algorithm to output actions with higher reward values.

[0094] The action selection of the action policy network is jointly evaluated by the multi-critic module, and the action policy network is optimized using the deep deterministic policy gradient algorithm based on the evaluation function. The evaluation function is calculated as:

[0095] Q=β1Q1+β2Q2 (14)

[0096] β1+β2=1 (15)

[0097] Where Q represents the evaluation value of the current UAV action, Q1 and Q2 are the evaluation values ​​of the current UAV action by the first and second critic modules, respectively, and β1 and β2 are the weights of the first and second critic modules, respectively.

[0098] The deep deterministic policy gradient descent function is:

[0099]

[0100] In the formula, is the objective function J(μ θ )’s gradient value; β1 and β2 are the weights of the first and second critic modules respectively; is the objective function of the first critic module defined according to the evaluation function; When the current environment information, target information and the drone's own state information are input into the action strategy network, the action strategy network outputs the action gradient of the action; The error gradient change between the action evaluation value and the reward value obtained by inputting the current action and the environment information, target information, and the drone's own state information into the first critic module composed of a multi-layer perceptron is used to guide the current action; is the objective function of the second critic module defined according to the evaluation function; It is the error gradient between the current action acceleration direction and the resultant force direction.

[0101] Assume that the current action leads to the gradient If the value is high, it means that the action is not advisable in the current environment, otherwise it is advisable. The ultimate goal is to make the objective function J(μ θ ) The gradient during the update process is 0, that is, the action strategy is improved by optimizing the objective function through policy gradient.

[0102] In step 3.4, the trained action strategy network is deployed on each UAV to guide the UAV to complete the collaborative task.

[0103] The example task proposed in this example can be achieved through the MCDDPG algorithm, so that the drone swarm can accumulate experience by interacting with the environment, and learn from the experience through the multi-critic module, continuously optimizing the drone's action strategy, so that it can follow dynamic targets and avoid collisions between drones and obstacles on the way. According to environmental changes, the drone swarm can automatically adjust and coordinate with each other to form a multi-angle formation to surround and observe the target.

[0104] The openai-gym simulation platform is used to define and train the mission scenarios of this embodiment. The MCDDPG algorithm strategy is used to train based on the interaction data between the drone and the environment, and the weight parameters and bias parameters in the action strategy network are iteratively updated. When L1 is as small as possible, the evaluation value and reward value of the drone group are closer, and the MCDDPG algorithm tends to converge. The convergence process of the training of three drones through the MCDDPG algorithm is shown in the figure. Figure 4 After training, the action strategy network obtains the actions (speed v, acceleration a and heading angle θ) of each drone at different times and states. The trained action strategy network is deployed on each drone to achieve tracking, obstacle avoidance and collaborative encirclement of dynamic targets. The visualization effect is shown in Figure 5(a)-5(c) shown.

[0105] Example 2

[0106] Based on the same inventive concept, the present invention also provides a dynamic collaborative control system for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning, including a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a dynamic collaborative control method for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning as described above.

[0107] Example 3

[0108] Based on the same inventive concept, the present invention also provides a dynamic collaborative control system for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning, including a readable storage medium, on which a computer program is stored. When the computer program is executed, a dynamic collaborative control method for a swarm of unmanned aerial vehicles based on intelligent agent reinforcement learning as described above is implemented.

[0109] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0110] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific implementation cases described, or replace them with similar methods, without departing from the spirit of the present invention or exceeding the scope defined by the appended claims.

Claims

1. A method for dynamic cooperative control of drone swarms based on agent reinforcement learning, characterized in that: The following steps are involved: Step 1: Set the physical properties of the drone, obstacles, and targets according to the drone's flight mission; Step 2: Set the kinematic equations of the drone and the target, establish the correlation between velocity, acceleration and direction of motion, and set constraints to comply with the rules of action in the physical world; Step 3: Decompose the collaborative tasks that the drones need to carry out into multiple subtasks, and model the subtasks as Markov decision processes. Use the multi-critic structured deep deterministic policy gradient MCDDPG algorithm to dynamically coordinate the drone swarm and complete the collaborative tasks of the drone swarm. Step 3.1, input the environmental status information obtained by the drone into the action strategy network, and the action strategy network outputs the action that each drone should take in the current environment; Step 3.2, use the critic module to evaluate the current action, and use the dynamic difficulty priority algorithm to optimally allocate the weights of different critic modules; Step 3.3, based on the evaluation value of the critic module, the action policy network is optimized using the deep deterministic policy gradient algorithm; In step 3.4, the trained action strategy network is deployed on each UAV to guide the UAV to complete the collaborative task.

2. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 1, characterized in that: The flight mission in step 1 is that the drone tracks the dynamic target from the starting point. Multiple drones form a coordinated encirclement formation around the target and avoid collisions with obstacles and collisions between drones during the tracking process. The physical properties of the drone are expressed as follows: Where UAV is the state space of the UAV; is the set of drones, N1 is the number of drones; is the attribute of the N1th UAV; is the coordinate value of the N1th UAV; r s is the safe flight radius of the UAV; v s is the speed of the drone; a s is the acceleration of the drone; c s Indicates whether drones can communicate with each other; θ s is the heading angle of the UAV; The physical properties of obstacles are expressed as follows: Where Obs is the state space of the obstacle; is the obstacle set, N2 is the number of obstacles; is the attribute of the N2th obstacle; is the coordinate value of the N2th obstacle; r c is the radius of the obstacle; The physical properties of the moving target are expressed as follows: Where Tar is the state space of the moving target; is the set of mobile targets, N3 is the number of mobile targets; The properties of the N3th moving target, including the target coordinates Target collision radius r t 、Target speed v t 、Target acceleration a t 、Target deflection angle θ t .

3. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 1, characterized in that: The kinematic equations of the UAV and the mobile target in step 2 are: In the formula, It represents the velocity component of the UAV or mobile target in the x and y directions at time t, which is determined by the vector value of the velocity v(t) and acceleration a(t) of the UAV or mobile target at time t at the heading angle θ; x(t) and y(t) are the coordinate components of the UAV or mobile target in the two-dimensional rectangular coordinate system; The constraints defining the properties of drones and mobile targets are: In the formula, v represents the speed of the drone or moving target, v min Indicates the minimum speed of the drone or moving target, v max represents the maximum speed of the drone or mobile target, a represents the acceleration of the drone or mobile target, and a min Indicates the minimum acceleration of the drone or moving target, a max represents the maximum acceleration of the drone or mobile target, (x,y) represents the position of the drone or mobile target, (x min ,x max ) represents the moving range of the drone or mobile target on the x-axis, (y min ,y max ) represents the moving range of the UAV or moving target on the y-axis.

4. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 1, characterized in that: In step 3.1, the drone inputs the acquired environmental information, target information and its own state information into the action strategy network. The action strategy network outputs the action that each drone should take in the current environment, including speed v, acceleration a and direction angle θ; environmental information includes the position and radius of obstacles, as well as the position, speed, acceleration and direction angle of other drones; target information includes the position, collision radius, speed, acceleration and deflection angle of the target; drone’s own state information includes the drone’s position, speed, acceleration and direction angle; The action policy network uses a multi-layer perceptron neural network.

5. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 1, characterized in that: There are two critic modules in step 3.

2. The first critic module is composed of a deep neural network architecture based on a multi-layer perceptron, which includes three fully connected layers. It converts the state information and action information of the drone into a one-dimensional vector and inputs it into the multi-layer perceptron deep neural network. Under the network structure composed of the fully connected layer and the activation function, the evaluation value of the current action is output; among them, the state information is composed of the drone's own state information, environmental information and target information. The drone's own state information includes the drone's position, speed, acceleration and direction angle. The environmental information includes the position and radius of obstacles, as well as the position, speed, acceleration and direction angle of other drones. The target information includes the target's position, collision radius, speed, acceleration and deflection angle. The action information is the speed, acceleration and heading angle of the drone output by the action strategy network.

6. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 5, characterized in that: The second critic module in step 3.2 is the gravitational field model, which is a mathematical model expressed as: Q f (s′,a′)=-f(s′)[1-cos(χ)], where Q f (s′, a′) is the evaluation value of the gravitational field model for the action decision a′ taken by the current UAV under the acquired state information s′, f(s′) is the resultant force formed by the attraction of the target to the UAV and the repulsion of the obstacle to the UAV under the current state information s′, χ is the angle between the acceleration direction and the resultant force f; when the acceleration direction of the UAV is consistent with the direction of the resultant force, χ = 0, and the evaluation value Q is f (s′, a′) = 0, the action decision in the current state will obtain a high reward value, otherwise, χ≠0, the corresponding evaluation value Q f (s′,a′)<0, then the action decision will obtain a low reward value.

7. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 6, characterized in that: In step 3.2, the weight of the critic module is dynamically adjusted according to the ratio of the difference between the evaluation value of the critic module and the reward value obtained by the drone. The weight β1 and β2 of each critic module are calculated as follows: y i =R i +γ1Q′(s i ′ +1 ,μ′(s i ′ +1 |θ′)|w′) (10) Where N is the number of drones, L1 represents the error between the evaluation value and reward value of the drone group by the first critic module composed of a deep neural network, and y i represents the reward value obtained by the i-th drone under the current action, Q1(s i ′,a i ′|w) represents the first critic module's response to the state information s obtained by the i-th UAV i 'The action taken is a i ′, w is the parameter weight in the neural network of the first critic module, L2 represents the evaluation value of the second critic module composed of the gravitational field model on the matching degree between the acceleration direction of the current action of the drone group and the direction of the resultant force, Q2(s i ′,a i ′) is the second critic module’s response to the state information s obtained by the i-th UAV i 'The action taken is a i ′, γ1 represents the distance discount factor, Q′(s i ' +1 ,μ′(s i ' +1 |θ′)|w′) represents the evaluation result of the two critic modules on the reward obtained by the current action at the next moment, s i ' +1 represents the state information at the next moment, μ′(s i ' +1 |θ′) represents the action strategy network at the next moment, θ′ represents the network parameters in the action strategy network at the next moment, w′ represents the network parameters of the first critic module at the next moment, R i is the actual reward value obtained by the i-th drone under the current action. The specific calculation formula is as follows: R i ＝r 1i +r 2i +r 3i +r 4i (11) r 2i =-0.1×steps (12) r 3i ＝0.1×d i (13) In the formula, r 1i The reward value obtained by the i-th UAV for completing the coordinated encirclement task of the target; r 2i is the penalty value obtained by the i-th UAV for failing to complete the coordinated encirclement task of the target, steps refers to the total number of actions used to execute the task; r 3i is the reward value obtained by the i-th UAV when it moves towards the target, d i represents the distance that the i-th UAV moves to the target point; if the i-th UAV collides with an obstacle, collides with other UAVs, or collides with the boundary of the environment during driving, the reward value r 4i = -1, otherwise r 4i =0.

8. The method for dynamic cooperative control of a drone swarm based on agent reinforcement learning according to claim 1, characterized in that: In step 3.3, the action selection of the action policy network is jointly evaluated by the multi-critic module, and the action policy network is optimized using the deep deterministic policy gradient algorithm according to the evaluation function. The evaluation function is calculated as follows: Q=β1Q1+β2Q2 (14) β1+β2＝1 (15) In the formula, Q represents the evaluation value of the current drone action, Q1 and Q2 are the evaluation values ​​of the first and second critic modules on the current drone action, and β1 and β2 are the weights of the first and second critic modules respectively; The deep deterministic policy gradient descent function is: In the formula, is the objective function J(μ θ )’s gradient value; β1 and β2 are the weights of the first and second critic modules respectively; is the objective function of the first critic module defined according to the evaluation function; When the current environment information, target information and the drone's own state information are input into the action strategy network, the action strategy network outputs the action gradient of the action; The error gradient change between the action evaluation value and the reward value obtained by inputting the current action and the environment information, target information, and the drone's own state information into the first critic module composed of a multi-layer perceptron is used to guide the current action; is the objective function of the second critic module defined according to the evaluation function; is the error gradient between the current action acceleration direction and the resultant force direction; Assume that the current action leads to the gradient If the value of the action increases, it means that the action is not desirable in the current environment; otherwise, it is desirable. The ultimate goal is to make the objective function J(μ θ ) The gradient during the update process is 0, that is, the action strategy is improved by optimizing the objective function through policy gradient.

9. A dynamic cooperative control system for drone swarms based on agent reinforcement learning, characterized in that: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a dynamic collaborative control method for a drone swarm based on intelligent agent reinforcement learning as described in any one of claims 1 to 8.

10. A dynamic cooperative control system for drone swarms based on agent reinforcement learning, characterized in that: It includes a readable storage medium, on which a computer program is stored. When the computer program is executed, a dynamic collaborative control method for a drone swarm based on agent reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images