UAV swarm mission planning method and device based on deep reinforcement learning

Abstract

This application provides a method and apparatus for drone swarm task planning based on deep reinforcement learning, including: randomly selecting one drone in the swarm as the first drone, and the other drones as the second drones, with the remaining drones forming the swarm; obtaining the actual task execution environment of the first drone and a drone swarm task planning model; inputting the actual task execution environment into the drone swarm task planning model to obtain the task plan for the drone swarm; wherein the drone swarm task planning model is obtained by training an improved MADDPG model using a simulated task execution environment as a training sample; the improved MADDPG model includes a MADDPG network and a mean field theory module, with the mean field theory module being disposed within the MADDPG network. Through the above method, drones can learn about global environmental changes during their movements, guiding the drone swarm to make optimal task plans in an unknown dynamic environment.

Description

Technical Field

[0001] The present invention relates to the field of drone technology, and in particular to a drone swarm mission planning method and device based on deep reinforcement learning. Background Art

[0002] Drones offer advantages such as ease of operation, flexibility, reliability, low cost, and reduced operator accidents. Recent years have seen rapid development of drones, with their capabilities for single-unit mission execution increasing, along with their autonomy and intelligence. As the scale and complexity of drone missions continue to expand, this has led to the emergence of drone swarms, where multiple drones collaborate to accomplish tasks.

[0003] Mission planning among drone swarms is fundamental to the collaborative processing of multiple drones. Traditional drone swarm mission planning methods consist of two parts: trajectory planning and task allocation. These methods operate under the assumption that the environment is essentially fixed and fully known. Under these methods, drones are unable to obtain timely global information about their environment and its changes, and are susceptible to interference from the external environment. Any incomplete environmental perception or biased estimation can lead to problems in drone swarm mission planning. Furthermore, these traditional drone swarm mission planning methods fail to fully consider the coupling between trajectory planning and task allocation. Summary of the Invention

[0004] Based on this, the present invention provides a drone cluster task planning method and device based on deep reinforcement learning, which can enable drones to timely obtain global environmental information and its changes, and guide the drone cluster to make the best decision for a specific state.

[0005] In a first aspect, the present invention provides a method for UAV swarm mission planning based on deep reinforcement learning, comprising:

[0006] Obtaining the actual mission execution environment of the drone and the drone cluster mission planning model, inputting the actual mission execution environment into the drone cluster mission planning model to obtain the mission plan of the drone cluster;

[0007] The UAV swarm mission planning is obtained by learning and training an improved MADDPG model using training samples of simulated mission execution; the improved MADDPG model includes a MADDPG network and a mean field theory module.

[0008] In a second aspect, the present invention provides a UAV swarm mission planning device based on deep reinforcement learning, comprising:

[0009] Parameter acquisition module, used to obtain the actual mission execution environment of the UAV and the UAV cluster mission planning model;

[0010] A task planning module is used to input the actual task execution environment into the UAV cluster task planning model to obtain the task plan of the UAV cluster;

[0011] The UAV swarm mission planning is obtained by learning and training an improved MADDPG model using training samples of simulated mission execution; the improved MADDPG model includes a MADDPG network and a mean field theory module.

[0012] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods for drone swarm task planning based on deep reinforcement learning in the first aspect.

[0013] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it executes any one of the drone cluster task planning methods based on deep reinforcement learning in the first aspect.

[0014] The beneficial effects of adopting the above technical solution are: based on the drone cluster task planning method based on deep reinforcement learning, a drone cluster task planning model is obtained through learning and training, so that the drone can obtain global environmental information and its changes during each action process, and can guide the drone cluster to make the optimal task planning in an unknown dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0016] Figure 1 A schematic diagram of a UAV swarm mission planning method based on deep reinforcement learning provided by one embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a MADDPG network framework provided by one embodiment of the present invention;

[0018] Figure 3 A schematic diagram illustrating an implementation of a UAV swarm mission planning method based on deep reinforcement learning according to an embodiment of the present invention;

[0019] Figure 4a A test environment for cooperative communication between drones using a deep reinforcement learning-based drone swarm mission planning method provided by one embodiment of the present invention;

[0020] Figure 4bA test environment for executing physical deception between drones using a drone swarm mission planning method based on deep reinforcement learning provided by one embodiment of the present invention;

[0021] Figure 5a-5c The UAV swarm mission planning method based on deep reinforcement learning provided by one embodiment of the present invention performs mission planning results of cooperative communication between UAVs;

[0022] Figure 6a-6c The UAV swarm mission planning method based on deep reinforcement learning provided by one embodiment of the present invention performs mission planning results of physical deception between UAVs;

[0023] Figure 7a A comparison chart of the reward values ​​obtained when performing cooperative communication tasks between drones using the deep reinforcement learning-based drone swarm task planning method according to an embodiment of the present invention and existing deep learning methods;

[0024] Figure 7b A chart comparing the success rates of the deep reinforcement learning-based UAV swarm task planning method according to an embodiment of the present invention and existing deep learning methods when performing cooperative communication tasks between UAVs;

[0025] Figure 8 A schematic diagram of a module for a drone swarm mission planning device based on deep reinforcement learning provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In order to explain the present invention in more detail, the drone swarm task planning method and device based on deep reinforcement learning provided by the present invention are specifically described below in conjunction with the drawings.

[0027] Drone swarm mission planning utilizes multiple drones to coordinate and coordinate, addressing the increasingly complex mission requirements of individual drones. Currently, when drone swarms perform area defense missions, individual drones are limited in their ability to obtain the global environment and determine the optimal strategy for mission planning. To address this issue, this application proposes a method, apparatus, storage medium, and device for drone swarm mission planning.

[0028] The embodiments of the present application provide specific application scenarios for the drone swarm mission planning method based on deep reinforcement learning. This application scenario includes the terminal device provided in the embodiments. The terminal device can be various electronic devices, including but not limited to smartphones and computers, where the computer device can be at least one of a desktop computer, portable computer, laptop computer, tablet computer, and the like. A user operates the terminal device and issues an operation instruction for drone swarm mission planning. The terminal device then executes the drone swarm mission planning method based on deep reinforcement learning of the present invention. For the specific process, please refer to the embodiment of the drone swarm mission planning method based on deep reinforcement learning.

[0029] Based on this, the embodiment of the present application provides a UAV cluster task planning method based on deep reinforcement learning, which is described by taking the application of the method to a terminal device as an example. Figure 1 Schematic diagram of the drone swarm mission planning method based on deep reinforcement learning.

[0030] In the embodiment of the present application, each drone in the drone cluster is regarded as a spherical intelligent body, and the radius of each drone is set to r uva , the initial position of the i-th UAV is set to P i =[x i ,y i , z i ] T , the initial speed of the i-th UAV is set to V i =[v i，x , v i，y , v i，z ] T , the speed of the i-th UAV at the preset time is set to where v i，x is the x-axis component of the initial velocity of the i-th UAV, v i，y is the y-axis component of the initial velocity of the i-th UAV, v i，z is the z-axis component of the initial velocity of the i-th UAV, v′ i，x is the x-axis component of the velocity of the i-th UAV at the preset time, v′ i，y is the y-axis component of the velocity of the i-th UAV at the preset time, v′ i，z is the z-axis component of the velocity of the ith UAV at the preset time, a is the acceleration of the ith UAV, and Δt is the preset time; the velocity V of any UAV i ≤V max , V max is the preset maximum speed of the flying object, and the y-axis component h of any drone position min ≤y i ≤h max , h minis the preset minimum altitude of the flying object, h max The preset maximum height of the flying object.

[0031] In the process of the UAV swarm performing a mission, there are also one or more obstacles and destinations. The obstacles are also regarded as spheres with a radius of r. adv , the initial position of the obstacle is set to P k =[x k ,y k , z k ] T , the initial velocity of the obstacle is set to V k =[v k，x , v k，y , v k，z ] T , the speed of the obstacle at the preset time is set to where v k，x is the x-axis component of the initial velocity of the obstacle, v k，y is the y-axis component of the initial velocity of the obstacle, v k，z is the z-axis component of the initial velocity of the obstacle, v′ k，x is the x-axis component of the obstacle's velocity at the preset time, v′ k，y is the y-axis component of the obstacle's velocity at the preset time, v′ k，z is the z-axis component of the obstacle's velocity at the preset time, a k is the acceleration of the obstacle, Δt is the preset time, and the speed of the obstacle V k ≤V max , V max is the preset maximum speed of the flying object, and the y-axis component h of the obstacle position min ≤y k ≤h max , h min is the preset minimum altitude of the flying object, h max The preset maximum height of the flying object.

[0032] The location of the destination is set to g = [x g ,y g , z g ] T , the radius of the destination is set to r aim .

[0033] Among them, the collision distance between the i-th UAV and the obstacle is set to D col =r uva +r adv ; When the i-th UAV arrives at the target area, the distance between the UAV and the target area is set to D aim ≤r uav +r aim .

[0034] In the embodiment of the present application, the UAV swarm mission planning can be represented by a Markov game model:<N,S,A,Γ,R,O,γ> , where N is the total number of drones in the simulated mission execution environment; S is the local state of all drones in the drone cluster; A is the action vector of all drones in the drone cluster, A=A1×A2×…×A N Γ is the probability that the UAV cluster adopts joint action to transfer to the next state in the current state, Γ: S×A1×A2×…×A N →S′, S′ is the next local state of all drones in the drone cluster; R is the joint reward of the drones, γ is the discount coefficient, r i is the reward value obtained by the interaction between the i-th UAV and the environment; O is the local state of each UAV.

[0035] Based on the physical models and motion models of the above-mentioned drones, obstacles, and destinations, the drone swarm mission planning method based on deep reinforcement learning in the embodiment of the present application specifically includes the following steps:

[0036] Step S101: randomly selecting one drone in the drone cluster as the first drone, and the other drones as the second drones, and the second drones constitute the remaining drone cluster.

[0037] In this embodiment, for ease of explanation, the first selected drone is recorded as the i-th drone in the drone cluster, the second drone is recorded as the j-th drone in the drone cluster, and the remaining drone clusters are recorded as d(i), j∈d(i).

[0038] Step S102: Acquire the actual mission execution environment of the first UAV and the UAV cluster mission planning model.

[0039] The UAV swarm mission planning is obtained by training the improved MADDPG model using the simulated mission execution environment as a training sample; the improved MADDPG model includes a MADDPG network and a mean field theory module, wherein the mean field theory module is set in the MADDPG network, and the framework of the MADDPG network is shown in the attached figure. Figure 2 The MADDPG network is a multi-agent deep deterministic policy gradient network, which is a network framework for centralized training and decentralized execution under multi-agent system planning.

[0040] Step S103: Input the actual task execution environment into the UAV cluster task planning model to obtain the UAV cluster task planning.

[0041] Specifically, the local state of each drone in the actual mission execution environment is input into the drone cluster task planning model to obtain the task planning of the drone cluster in the actual mission execution environment.

[0042] Furthermore, combined with Figure 3 , further explanation is given for the UAV cluster mission planning model used in steps S102-S103:

[0043] The UAV swarm mission planning model is obtained by training an improved MADDPG model using a simulated mission execution environment as training samples. The improved MADDPG model includes a MADDPG network and a mean field theory module. Furthermore, the MADDPG network includes a policy network and an evaluation network connected in sequence, in which the mean field theory module is nested in the evaluation network.

[0044] The simulated task execution environment can be obtained using OPENAI's gym simulation platform or universe simulation platform.

[0045] Building a UAV swarm mission planning model involves the following steps:

[0046] Step S201: Obtain training samples, which include the simulated state space s of the drone cluster at the current moment, the simulated state space s′ of the drone cluster at the next moment, the simulated rewards r of all drones in the drone cluster, the simulated action vectors a of all drones in the drone cluster, and the average action vector of the remaining drone clusters in the drone cluster.

[0047] Each training sample can be recorded as The simulated state space of the current UAV cluster includes the simulated local state of each UAV in the current UAV cluster, which is denoted as s = (o t，1 , o t，2 ,...,o t，N ), o t，i is the simulated local state of the i-th UAV in the UAV cluster at the current moment, N is the number of UAVs in the UAV cluster; the simulated state space of the UAV cluster at the next moment includes the simulated local state of each UAV in the UAV cluster at the next moment, denoted as s′=(o′ t，1 , o′ t，2 ,...,o′ t，N ), o′ t，i is the simulated local state of the i-th drone in the drone cluster at the next moment; the simulated reward of all drones in the drone cluster includes the simulated reward value of each drone in the drone cluster at the current moment, recorded as r = (r t，1 , r t，2 ,...,rt，N ), r t，i is the simulated reward value of the i-th drone in the drone cluster at the current moment; the simulated action vector of all drones in the drone cluster includes the simulated action vector of each drone in the drone cluster at the current moment, denoted as a=(a t，1 , a t，2 ,...,a t，N ), a t，i is the simulated motion vector of the i-th drone in the drone cluster at the current moment; the average motion vector of the remaining drone clusters in the drone cluster includes the average motion vector of each drone in the drone cluster corresponding to the remaining drone clusters at the current moment, recorded as is the average action vector of the i-th drone in the drone cluster corresponding to the remaining drone clusters at the current moment.

[0048] Specifically, the training samples for the UAV swarm mission planning model include the following steps:

[0049] Step S202: Obtain the simulated local state s of all drones in the drone cluster at the current moment = ( t，1 , o t，2 ,...,o t，N ), where the simulated local state of each UAV is the local state that can be observed by each UAV.

[0050] Step S203: Based on the simulated local state of each drone at the current moment t，i Calculate the simulated action vector a of each drone t，i , specifically:

[0051] The simulated local state o of the first UAV in the UAV cluster at the current moment t，i Input to the policy network μ i , get the simulated intermediate action vector μ of the first UAV at the current moment i (o t，i );

[0052] The simulated intermediate motion vector of the first UAV at the current moment is superimposed with the noise vector to obtain the simulated motion vector of the first UAV at the current moment; the expression is a t，i =μ i (o t，i )+P, where P is the noise vector. The introduction of the noise vector increases the exploratory nature of the policy function.

[0053] The simulated motion vector of the second UAV in the UAV cluster can also be calculated using the above expression, which will not be repeated here.

[0054] Step S204: The simulated local state o of each second UAV in the UAV cluster at the current moment is t，j and the simulated motion vector a of each second UAV t，j Input the mean field theory module respectively to obtain the simulated local state average value of the remaining drone cluster at the current moment and the simulated average action vector of the remaining drone cluster

[0055] Step S204 calculates the simulated local state average value of the remaining drone cluster at the current moment and the simulated average motion vector of the remaining drone cluster, including steps S301-S302:

[0056] Step 301: Superimpose the simulated local states of each second drone in the drone cluster at the current moment and calculate the average value to obtain the average value of the simulated local states of the remaining drone cluster at the current moment; the specific expression is: is the average value of the simulated local state of the remaining drone cluster at the current moment, |d(i)| is the number of drones in the remaining drone cluster, and d(i) is the number of drones in the remaining drone cluster corresponding to the i-th drone in the drone cluster.

[0057] Step S302: Superimpose the simulated motion vectors of each second drone in the drone cluster and calculate the average value to obtain the simulated average motion vector of the remaining drone cluster; the specific expression is: is the simulated average action vector of the remaining drone cluster at the current moment.

[0058] Step S205: Combine the simulated local state of the first UAV at the current moment and the average value of the simulated local states of the remaining UAV clusters at the current moment to obtain the simulated state space of the UAV cluster at the current moment, which is recorded as s t，i is the simulated state space of the drone cluster at the current moment.

[0059] Step S206: The simulated action vector of each drone in the drone cluster at the current moment interacts with the simulated task execution environment to obtain the simulated reward of each drone in the drone cluster at the current moment and the simulated local state of each drone in the drone cluster at the next moment.

[0060] Step S207: Based on the simulated local state o' of each drone at the next moment t，i Calculate the simulated action vector a′ of each drone t，i , specifically:

[0061] The simulated local state o' of the first UAV in the UAV cluster at the next moment t，i Input to the policy network μ i, get the simulated intermediate action vector μ of the first UAV at the next moment i (o′ t，i );

[0062] The simulated intermediate motion vector of the first UAV at the next moment is superimposed with the noise vector to obtain the simulated motion vector of the first UAV at the next moment; the expression is a′ t，i =μ i (o′ t，i )+P, where P is the noise vector. The introduction of the noise vector increases the exploratory nature of the policy function.

[0063] The simulated motion vector of the second UAV in the UAV cluster at the next moment can also be calculated using the above expression, which will not be repeated here.

[0064] Step 208: The simulated local state o′ of each second UAV in the UAV cluster at the next moment t，j and the simulated action vector a′ of each second drone in the drone cluster at the next moment t，j The mean field theory is input respectively to obtain the simulated local state average value of the remaining drone cluster and the simulated average action vector of the remaining drone cluster at the next moment.

[0065] Step S208 calculates the simulated local state average value of the remaining drone cluster and the simulated average motion vector of the remaining drone cluster at the next moment, including steps S303-S304:

[0066] Step 303: Superimpose the simulated local states of each second drone in the drone cluster at the next moment and calculate the average value to obtain the average value of the simulated local states of the remaining drone clusters at the next moment; the specific expression is: is the average value of the simulated local state of the remaining drone cluster at the next moment, |d(i)| is the number of drones in the remaining drone cluster, and d(i) is the number of drones in the remaining drone cluster corresponding to the i-th drone in the drone cluster.

[0067] Step S304: Superimpose the simulated motion vectors of each second drone in the drone cluster and calculate the average value to obtain the simulated average motion vector of the remaining drone cluster; the specific expression is: is the simulated average action vector of the remaining drone cluster at the next moment.

[0068] Step S209: Combine the simulated local state of the first UAV at the next moment and the average value of the simulated local states of the remaining UAV clusters at the next moment to obtain the simulated state space of the UAV cluster at the next moment, which is recorded as s′ t，i It is the simulated state space of the drone cluster at the next moment.

[0069] Step S210: After inputting the simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment into the evaluation network, the simulated reward of the first drone in the drone cluster at the current moment is superimposed to obtain the evaluation value of the evaluation network.

[0070] Specifically, step S210 of calculating the evaluation value of the evaluation network includes steps S401-S403:

[0071] Step S401: Input the simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment into the evaluation network to obtain the evaluation network action value at the next moment.

[0072] Step S402: multiplying the evaluation network action value at the next moment by the discount coefficient to obtain an intermediate evaluation network action value.

[0073] Step S403: Superimpose the intermediate evaluation network action value and the simulated reward of the drone at the current moment to obtain the evaluation value of the evaluation network. The specific expression of the evaluation value of the evaluation network is: Where y is the evaluation value of the evaluation network, r t，i is the simulated reward of the first drone in the drone cluster at the current moment, γ is the discount coefficient, is the evaluation network action value at the next moment.

[0074] Step S211: Input the simulated state space of the current UAV cluster, the simulated action vector of the first UAV at the current moment, and the simulated average action vector of the remaining UAV clusters at the current moment into the evaluation network to obtain the action value of the evaluation network at the current moment.

[0075] Step S212: Obtaining a loss function of the evaluation network according to the evaluation value of the evaluation network and the action value of the evaluation network at the current moment.

[0076] Step S212 calculates the loss function of the evaluation network and includes steps S501-S502:

[0077] Step S501: Perform difference processing on the evaluation value of the evaluation network and the action value of the evaluation network at the current moment to obtain the loss error of the evaluation network.

[0078] Step S502: Calculate the mean of the loss error after exponential processing according to the number of training samples to obtain the loss function of the evaluation network. The specific expression of the loss function of the evaluation network is: in is the loss function for evaluating the network, M is the number of training samples, Evaluate the action value of the network at the current moment.

[0079] Step S213: Obtain evaluation network parameters through the loss function.

[0080] Step S214: updating the policy network gradient according to the evaluation network parameters to obtain the policy network parameters.

[0081] Step S214 calculates the strategic network parameters including steps S601-S602:

[0082] Step S601: Obtain the current policy network policy function gradient according to the evaluation network parameters;

[0083] Step S602: After multiplying the policy function gradient of the current policy network, the action value function gradient of the current evaluation network, and the action value of the current evaluation network, the mean is calculated based on the number of training samples to obtain the policy gradient of the current policy network parameters. The specific expression is: in is the policy gradient of the policy network parameters, is the policy function gradient of the policy network, is the gradient of the current evaluation network action value function.

[0084] Step S215: Update the policy network parameters according to the proportional coefficient, and update the policy network and evaluation network according to the updated policy network coefficient until the number of updates is reached to obtain the UAV cluster mission planning model.

[0085] The update of policy network parameters is as follows: in are the updated policy network parameters, is the policy network parameter before updating, and τ is the proportional coefficient of the current policy network parameter.

[0086] In addition, calculating the simulated reward of the first drone in step 210 includes the following steps:

[0087] The simulation reward of the first drone includes the collision reward between the first drone and the obstacle. c , the arrival reward r of the first drone arriving at the destination g And the action reward r of the first drone performing the action s .

[0088] Collision Reward c The specific expression is: r col =-5, D is the distance between the drone and the obstacle, D col is the collision distance between the drone and the obstacle.

[0089] Arrival Rewards g The specific expression is: r arr =10, ε is the guidance coefficient of the UAV approaching the destination, ε=1.1, P i is the position of the i-th UAV, g is the position of the destination, D aim is the distance between the UAV and the destination, r uav is the radius of the drone, r aim The radius of the destination.

[0090] Action reward r s The specific expression is: s =-3.

[0091] The embodiment of the present application combines the mean field theory module with the MADDPG network to form a drone swarm task planning model, overcoming the problem in the prior art that the multi-agent reinforcement learning algorithm uses the set of observation values ​​obtained by each agent as the state value, and the dimension of the state value increases exponentially when the number of agents is large. The mean field theory module reduces the dimension of the state value during the model training process, accelerating the convergence of the model training process; and the MADDPG network adopts the principle of centralized training distributed decision-making, so that drones can still make task decisions efficiently under the premise of unknown environment, and adopts the experience pool and dual network structure to improve the convergence speed.

[0092] In addition, since a large amount of data is used to update and adjust the policy network and evaluation network during the training process, the final drone cluster task planning model can achieve the globally optimal planning, and realize that in an unknown dynamic three-dimensional environment, the drone cluster can simultaneously adopt centralized training and distributed execution strategies. In the training environment, drones communicate with each other and learn cooperation strategies; in the actual task execution environment, drones can make decisions based solely on the local state of their own observations, and no longer need to communicate, which greatly shortens the decision-making time.

[0093] To better demonstrate the technical effect of the UAV swarm mission planning method based on deep reinforcement learning according to the embodiment of the present invention, as shown in FIG4- Figure 6c As shown in the figure, the test environment and task planning results of the drone cluster in performing cooperative communication and physical deception tasks are demonstrated through simulation. From the simulation results of the above figures, it can be seen that the drone cluster task planning method based on deep reinforcement learning can achieve better results in cooperative communication and physical deception between drone clusters. In order to more intuitively demonstrate the superior technical effect of the drone cluster task planning method based on deep reinforcement learning in the embodiment of the present invention compared with the existing deep learning method, as shown in the attached figure, Figure 7a and attached Figure 7bAs shown, it can be clearly seen that when the drones perform cooperative communication tasks, the drone cluster task planning method based on deep reinforcement learning of the present invention can achieve higher reward values ​​during operation, and the success rate of task execution is much higher than that of the existing deep learning method.

[0094] It should be understood that although the Figure 1 The steps in the flowchart are shown sequentially as indicated by arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps must be executed; these steps may be executed in other orders. Furthermore, at least some of the steps in FIG1 may include multiple sub-steps or sub-stages. These sub-steps or stages do not necessarily need to be completed at the same time, but may be executed at different times. These sub-steps or stages do not necessarily need to be executed sequentially, but may be executed in rotation or alternation with other steps or at least a portion of their sub-steps or stages.

[0095] The embodiment disclosed in the above invention describes in detail the UAV swarm mission planning method based on deep reinforcement learning. The above method disclosed in the present invention can be implemented by various devices. Therefore, the present invention also discloses a UAV swarm mission planning device based on deep reinforcement learning corresponding to the above method. Figure 8 , specific embodiments are given below to explain in detail.

[0096] The drone selection module 701 is used to randomly select one drone in the drone cluster as the first drone and the other drones as the second drones, and the second drones constitute the remaining drone cluster.

[0097] The parameter acquisition module 702 is used to obtain the actual mission execution environment of the first UAV and the UAV cluster mission planning model.

[0098] The task planning module 703 is used to input the actual task execution environment into the UAV cluster task planning model to obtain the task planning of the UAV cluster.

[0099] Among them, the drone cluster task planning is obtained by learning and training the improved MADDPG model using a simulated task execution environment as a training sample; the improved MADDPG model includes a MADDPG network and a mean field theory module, wherein the mean field theory module is set in the MADDPG network.

[0100] For the specific definition of the UAV swarm task planning device based on deep reinforcement learning, please refer to the definition of the UAV swarm task planning method based on deep reinforcement learning above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware, and a combination thereof. The above modules can be embedded in or independent of the processor of the terminal device in hardware form, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0101] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned drone cluster task planning method based on deep reinforcement learning.

[0102] The computer-readable storage medium can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM (Erasable Programmable Read-Only Memory), a hard disk, or a ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written into one or more computer program products, and the program codes can be compressed in an appropriate form.

[0103] In one embodiment, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the above-mentioned steps of the drone cluster task planning method based on deep reinforcement learning when executing the computer program.

[0104] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to execute the above-mentioned drone swarm task planning method based on deep reinforcement learning.

[0105] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire computer device, and performs various functions of the computer device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to handle wireless communication. It is understandable that the above-mentioned modem may not be integrated into the processor and may be implemented separately through a communication chip.

[0106] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the terminal device during use, etc.

[0107] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A UAV swarm mission planning method based on deep reinforcement learning, characterized in that: The method includes: Randomly select one drone in the drone cluster as the first drone, and the other drones as the second drones, and the second drones constitute the remaining drone cluster; Obtaining the actual mission execution environment of the first UAV and the UAV swarm mission planning model; Inputting the actual task execution environment into the UAV cluster task planning model to obtain the UAV cluster task planning; The UAV swarm mission planning model is obtained by training an improved MADDPG model using a simulated mission execution environment as a training sample; the improved MADDPG model includes a MADDPG network and a mean field theory module, wherein the mean field theory module is provided in the MADDPG network; The improved MADDPG model includes a strategy network and an evaluation network connected in sequence, wherein the mean field theory module is nested in the evaluation network; Establishing the UAV swarm mission planning model includes: Obtaining training samples, where the training samples include a simulated state space of the drone cluster at the current moment, a simulated state space of the drone cluster at the next moment, simulated rewards of all drones in the drone cluster, simulated action vectors of all drones in the drone cluster, and an average action vector of the remaining drone clusters in the drone cluster; Inputting the simulated local state of the first UAV in the UAV cluster at the current moment into the policy network to obtain a simulated intermediate action vector of the first UAV at the current moment; Superimposing the simulated intermediate motion vector of the first UAV at the current moment with the noise vector to obtain a simulated motion vector of the first UAV at the current moment; Input the simulated local state of each second UAV in the UAV cluster at the current moment and the simulated action vector of each second UAV into the mean field theory module respectively, and obtain the average value of the simulated local state of the remaining UAV cluster at the current moment and the simulated average action vector of the remaining UAV cluster; The simulated local state of the first UAV at the current moment and the average value of the simulated local states of the remaining UAV clusters at the current moment are combined to obtain the simulated state space of the UAV cluster at the current moment; The simulated action vector of each drone in the drone cluster at the current moment interacts with the simulated task execution environment to obtain the simulated reward of each drone in the drone cluster at the current moment and the simulated local state of each drone in the drone cluster at the next moment; Inputting the simulated local state of the first UAV at the next moment into the policy network to obtain a simulated intermediate action vector of the first UAV at the next moment; Superimposing the simulated intermediate motion vector of the first UAV at the next moment with the noise vector to obtain a simulated motion vector of the first UAV at the next moment; Input the simulated local state of each second UAV in the UAV cluster at the next moment and the simulated action vector of each second UAV in the UAV cluster at the next moment into the mean field theory respectively, and obtain the average value of the simulated local state of the remaining UAV cluster at the next moment and the simulated average action vector of the remaining UAV cluster; The simulated local state of the first UAV at the next moment and the average value of the simulated local states of the remaining UAV clusters at the next moment are combined to obtain the simulated state space of the UAV cluster at the next moment; After inputting the simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment into the evaluation network, the simulated reward of the first drone in the drone cluster at the current moment is superimposed to obtain the evaluation value of the evaluation network; Inputting the simulated state space of the drone cluster at the current moment, the simulated action vector of the first drone at the current moment, and the simulated average action vector of the remaining drone clusters at the current moment into the evaluation network to obtain the action value of the evaluation network at the current moment; Obtaining a loss function of the evaluation network according to the evaluation value of the evaluation network and the action value of the evaluation network at the current moment; Obtaining evaluation network parameters through the loss function; Update the policy network gradient according to the evaluation network parameters to obtain the policy network parameters; The policy network parameters are updated according to the proportional coefficient, and the policy network and the evaluation network are updated according to the updated policy network coefficient until the update times are reached to obtain the UAV swarm mission planning model.

2. The UAV swarm mission planning method based on deep reinforcement learning according to claim 1, characterized in that: The simulated local state of each second UAV in the UAV cluster and the simulated action vector of each second UAV in the UAV cluster are respectively input into the mean field theory module to obtain the average value of the simulated local state of the remaining UAV cluster and the simulated average action vector of the remaining UAV cluster, including: Superimposing the simulated local states of each second UAV in the UAV cluster and calculating the average value to obtain the average value of the simulated local states of the remaining UAV clusters; The simulated motion vectors of each second UAV in the UAV cluster are superimposed and the average value is calculated to obtain the simulated average motion vector of the remaining UAV cluster.

3. The UAV swarm mission planning method based on deep reinforcement learning according to claim 1, characterized in that: The simulation rewards for the first drone include: The first drone's collision reward with an obstacle, the first drone's arrival reward when it reaches its destination, and the first drone's action reward when it performs an action.

4. The UAV swarm mission planning method based on deep reinforcement learning according to claim 1, characterized in that: After inputting the simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment into the evaluation network, the simulated reward of the first drone in the drone cluster at the current moment is superimposed to obtain the evaluation value of the evaluation network, including: The simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment are input into the evaluation network to obtain the evaluation network action value at the next moment; The evaluation network action value at the next moment is multiplied by the discount coefficient to obtain an intermediate evaluation network action value; The intermediate evaluation network action value is superimposed on the simulated reward of the drone at the current moment to obtain an evaluation value of the evaluation network.

5. The UAV swarm mission planning method based on deep reinforcement learning according to claim 1, characterized in that: The step of obtaining a loss function of the evaluation network based on the evaluation value of the evaluation network and the action value of the evaluation network at the current moment includes: Performing difference processing on the evaluation value of the evaluation network and the action value of the evaluation network at the current moment to obtain the loss error of the evaluation network; The loss function of the evaluation network is obtained by calculating the mean of the loss error after exponential processing according to the number of training samples.

6. The UAV swarm mission planning method based on deep reinforcement learning according to claim 1, characterized in that: The updating of the policy network gradient according to the evaluation network parameters to obtain the policy network parameters includes: Obtain the current policy network policy function gradient based on the evaluation network parameters; After multiplying the policy function gradient of the current policy network, the action value function gradient of the current evaluation network, and the action value of the evaluation network at the current moment, the mean is calculated according to the number of training samples to obtain the policy gradient of the current policy network parameters.

7. A UAV swarm mission planning device based on deep reinforcement learning, characterized in that: The device includes: A drone selection module, configured to select any one drone in the drone cluster as a first drone and the other drones as second drones, wherein the second drones constitute the remaining drone cluster; A parameter acquisition module, used to obtain the actual mission execution environment of the first UAV and the UAV cluster mission planning model; A task planning module is used to input the actual task execution environment into the UAV cluster task planning model to obtain the task plan of the UAV cluster; The UAV swarm mission planning is obtained by training an improved MADDPG model using a simulated mission execution environment as a training sample; the improved MADDPG model includes a MADDPG network and a mean field theory module, wherein the mean field theory module is provided in the MADDPG network; The improved MADDPG model includes a strategy network and an evaluation network connected in sequence, wherein the mean field theory module is nested in the evaluation network; Establishing the UAV swarm mission planning model includes: Obtaining training samples, where the training samples include a simulated state space of the drone cluster at the current moment, a simulated state space of the drone cluster at the next moment, simulated rewards of all drones in the drone cluster, simulated action vectors of all drones in the drone cluster, and an average action vector of the remaining drone clusters in the drone cluster; Inputting the simulated local state of the first UAV in the UAV cluster at the current moment into the policy network to obtain a simulated intermediate action vector of the first UAV at the current moment; Superimposing the simulated intermediate motion vector of the first UAV at the current moment with the noise vector to obtain a simulated motion vector of the first UAV at the current moment; Input the simulated local state of each second UAV in the UAV cluster at the current moment and the simulated action vector of each second UAV into the mean field theory module respectively, and obtain the average value of the simulated local state of the remaining UAV cluster at the current moment and the simulated average action vector of the remaining UAV cluster; The simulated local state of the first UAV at the current moment and the average value of the simulated local states of the remaining UAV clusters at the current moment are combined to obtain the simulated state space of the UAV cluster at the current moment; The simulated action vector of each drone in the drone cluster at the current moment interacts with the simulated task execution environment to obtain the simulated reward of each drone in the drone cluster at the current moment and the simulated local state of each drone in the drone cluster at the next moment; Inputting the simulated local state of the first UAV at the next moment into the policy network to obtain a simulated intermediate action vector of the first UAV at the next moment; Superimposing the simulated intermediate motion vector of the first UAV at the next moment with the noise vector to obtain a simulated motion vector of the first UAV at the next moment; Input the simulated local state of each second UAV in the UAV cluster at the next moment and the simulated action vector of each second UAV in the UAV cluster at the next moment into the mean field theory respectively, and obtain the average value of the simulated local state of the remaining UAV cluster at the next moment and the simulated average action vector of the remaining UAV cluster; The simulated local state of the first UAV at the next moment and the average value of the simulated local states of the remaining UAV clusters at the next moment are combined to obtain the simulated state space of the UAV cluster at the next moment; After inputting the simulated state space of the drone cluster at the next moment, the simulated action vector of the first drone at the next moment, and the simulated average action vector of the remaining drone clusters at the next moment into the evaluation network, the simulated reward of the first drone in the drone cluster at the current moment is superimposed to obtain the evaluation value of the evaluation network; Inputting the simulated state space of the drone cluster at the current moment, the simulated action vector of the first drone at the current moment, and the simulated average action vector of the remaining drone clusters at the current moment into the evaluation network to obtain the action value of the evaluation network at the current moment; Obtaining a loss function of the evaluation network according to the evaluation value of the evaluation network and the action value of the evaluation network at the current moment; Obtaining evaluation network parameters through the loss function; Update the policy network gradient according to the evaluation network parameters to obtain the policy network parameters; The policy network parameters are updated according to the proportional coefficient, and the policy network and the evaluation network are updated according to the updated policy network coefficient until the update times are reached to obtain the UAV swarm mission planning model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the drone swarm mission planning method based on deep reinforcement learning according to any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it executes the drone swarm mission planning method based on deep reinforcement learning according to any one of claims 1 to 6.

Images