Unmanned aerial vehicle cluster task planning method and device, storage medium and equipment

Abstract

The application provides a UAV cluster task planning method and device, a storage medium and equipment, comprising: selecting one UAV as a first UAV and other UAVs as second UAVs in a UAV cluster, and the second UAVs forming a remaining UAV cluster; obtaining an actual task execution environment of the first UAV and a UAV cluster task planning model; inputting the actual task execution environment into the UAV cluster task planning model to obtain a task planning of the UAV cluster; wherein the UAV cluster task planning model is obtained by learning and training an improved COMA model with a simulated task execution environment as a training sample; the improved COMA model comprises a COMA network and a graph convolution module, and the graph convolution module is arranged in the COMA network. The UAV cluster can guide the UAV cluster to make a globally optimal action according to the setting of the joint reward and the continuous updating of the estimated value of the evaluation function.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and more specifically to UAV swarm mission planning methods, apparatus, storage media, and devices. Background Technology

[0002] Unmanned aerial vehicles (UAVs) are characterized by low cost, high mobility, convenient deployment, and long endurance, leading to their increasingly widespread application in both military and civilian fields. They are the optimal choice for missions such as reconnaissance and patrol. However, due to the limited coverage and effectiveness of a single UAV, and the growing complexity of missions, UAV swarms working collaboratively are becoming a development trend in the UAV industry.

[0003] Existing UAV swarm task planning methods consist of two parts: trajectory planning and task allocation. Since both are pre-defined and their coupling is not considered during the planning process, these methods are unable to handle dynamic environments with unstable factors. Furthermore, existing reinforcement learning algorithms treat every UAV in the swarm equally during task planning, assuming each UAV has the same impact on the swarm's reward value. This can lead to the possibility of finding local optima in the calculated results. Moreover, when there are many UAVs in the swarm, the interaction process can easily consume significant communication and computing resources, hindering the efficiency of task planning. Summary of the Invention

[0004] Based on this, the present invention provides a method, apparatus, storage medium and device for drone swarm mission planning, which enables drones to obtain global environmental information and its changes in a timely manner, and takes into account the credit allocation problem of drones, guiding the drone swarm to make optimal decisions.

[0005] In a first aspect, the present invention provides a method for planning unmanned aerial vehicle (UAV) swarm missions, including:

[0006] In the drone swarm, one drone is randomly selected as the first drone, and the other drones are selected as the second drones. The second drones form the remaining drone swarm.

[0007] Obtain the actual mission execution environment of the first UAV and the UAV swarm mission planning model;

[0008] The actual task execution environment is input into the UAV swarm task planning model to obtain the UAV swarm task plan;

[0009] The drone swarm mission planning model is obtained by training an improved COMA model using a simulated mission execution environment as training samples. The improved COMA model includes a COMA network and a graph convolution module, wherein the graph convolution module is set in the COMA network.

[0010] In a second aspect, the present invention provides a drone swarm mission planning device, comprising:

[0011] The drone selection module is used to arbitrarily select one drone in the drone cluster as the first drone, and the other drones as the second drones. The second drones form the remaining drone cluster.

[0012] The parameter acquisition module is used to acquire the actual mission execution environment of the first UAV and the UAV cluster mission planning model.

[0013] The task planning module is used to input the actual task execution environment into the UAV swarm task planning model to obtain the task plan of the UAV swarm.

[0014] The drone swarm mission planning model is obtained by training an improved COMA model using training samples from a simulated mission execution environment. The improved COMA model includes a COMA network and a graph convolution module, wherein the graph convolution module is set in the COMA network.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the UAV swarm mission planning methods in the first aspect.

[0016] Fourthly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform any one of the UAV swarm mission planning methods in the first aspect.

[0017] The beneficial effects of adopting the above technical solution are as follows: This application constructs a drone swarm task planning model based on deep reinforcement learning of COMA and graph convolution. The drone swarm can be guided to make globally optimal actions according to the setting of joint rewards and the continuous updating of the estimated value of the evaluation function. Furthermore, the local state stacking results of neighboring drones are considered in the calculation of the estimated value of the evaluation function, which reduces the use of communication and computing resources during the interaction process and improves the efficiency of task planning. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0019] Figure 1 This is a schematic diagram of a drone swarm task planning method in one embodiment of this application;

[0020] Figure 2This is a schematic diagram of the drone swarm task planning process in one embodiment of this application;

[0021] Figure 3 This is a framework diagram of a drone swarm task planning graph convolutional module according to an embodiment of this application;

[0022] Figure 4a This is a test environment for a drone swarm to perform cooperative communication tasks according to one embodiment of this application;

[0023] Figure 4b This is a test environment for a drone swarm to perform a physical deception task according to one embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a drone swarm mission planning device according to an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. To describe the present invention in more detail, the UAV swarm mission planning method, apparatus, storage medium, and device provided by the present invention will be specifically described below with reference to the accompanying drawings.

[0026] Unmanned aerial vehicle (UAV) swarm mission planning compensates for the insufficient mission execution capabilities of a single UAV by employing multiple UAVs in a coordinated manner, addressing the increasingly complex mission processing requirements. Currently, when UAV swarms perform area defense missions, the limited information acquired by individual UAVs prevents them from timely obtaining the global environment and determining the optimal strategy during mission planning. To address this issue, this application proposes a UAV swarm mission planning method, apparatus, storage medium, and device.

[0027] This application provides specific application scenarios for the UAV swarm mission planning method. These application scenarios include the terminal device provided in the embodiments. The terminal device can be various electronic devices, including but not limited to smartphones and computer devices, wherein the computer device can be at least one of desktop computers, portable computers, laptop computers, tablet computers, etc. The user operates the terminal device, issuing operation instructions for UAV swarm mission planning. The terminal device executes the UAV swarm mission planning method of this invention. For details, please refer to the embodiments of the UAV swarm mission planning method.

[0028] Based on this, this application provides a method for planning unmanned aerial vehicle (UAV) swarm missions. The method is illustrated using an application to a terminal device as an example, in conjunction with the appendix. Figure 1The diagram shows a method for planning drone swarm missions.

[0029] In this embodiment, each drone in the drone swarm is considered a spherical intelligent agent, and the radius of each drone is set to r. uva The initial position of the i-th drone is set to P. i =[x i y i , z i ] T The initial velocity of the i-th drone is set to V. i =[v i，x v i，y v i，z ] T The speed of the i-th drone at the preset time is set to Where v i，x Let v be the x-axis component of the initial velocity of the i-th drone. i，y Let v be the y-axis component of the initial velocity of the i-th drone. i，z Let v′ be the z-axis component of the initial velocity of the i-th drone. i，x Let v′ be the x-axis component of the velocity of the i-th drone at a preset time. i，y Let v′ be the y-axis component of the velocity of the i-th drone at a preset time. i，z Let be the z-axis component of the velocity of the i-th drone at a preset time, 'a' be the acceleration of the i-th drone, and 'Δt' be the preset time; V is the velocity of any drone. i ≤V max V max Given the preset maximum speed of the flying object, the y-axis component h at any drone position min ≤y i ≤h max h min h is the preset minimum altitude for the aircraft. max This is the preset maximum altitude of the flying object.

[0030] During the mission execution by the drone swarm, one or more obstacles and a destination are also included. The obstacles are also considered as spheres with a radius of r. adv The initial position of the obstacle is set to P. k =[x k y k , z k ] T The initial velocity of the obstacle is set to V. k =[v k,x v k,y v k,z ] T The speed of the obstacle at the preset time is set to Where v k,x v is the x-axis component of the initial velocity of the obstacle. k,y v is the y-axis component of the initial velocity of the obstacle. k,z v′ is the z-axis component of the initial velocity of the obstacle. k，x Let v′ be the x-axis component of the obstacle's velocity over a preset time. k，y v′ is the y-axis component of the obstacle's velocity over a preset time. k，z a is the z-axis component of the obstacle's velocity over a preset time. k Let V be the acceleration of the obstacle, Δt be the preset time, and V be the velocity of the obstacle. k ≤V max V max The preset maximum speed of the flying object, and the y-axis component h of the obstacle position. min ≤y k ≤h max h min h is the preset minimum altitude for the aircraft. max This is the preset maximum altitude of the flying object.

[0031] The location of the destination is set to g = [x g y g , z g ] T The radius of the destination is set to r. aim .

[0032] Wherein, the collision distance between the i-th drone and the obstacle is set to D. col =r uva +r adv When the i-th drone arrives at the target area, the distance between the drone and the target area is set to D. aim ≤r uav +r aim .

[0033] In this embodiment of the application, the drone swarm task planning can be represented using a Markov game model, specifically as follows:<N,S,A,Γ,R,O,γ> Where N is the total number of drones in the simulated task execution environment; S is the local state of all drones in the drone swarm; and A is the motion vector of all drones in the drone swarm, A = A1 × A2 × … × A N Γ represents the probability that the drone swarm will transition to the next state by taking a joint action in the current state. ： S×A1×A2×…×A N →S′, where S′ is the next local state of all drones in the drone swarm; R is the joint reward for the drones. γ is the discount factor, r i Let be the reward value obtained by the i-th drone through interaction with the environment; O represents the local state of each drone.

[0034] Based on the physical and motion models of the drones, obstacles, and destination described above, the drone swarm mission planning method of this application embodiment specifically includes the following steps:

[0035] Step S101: Randomly select one drone in the drone cluster as the first drone, and the other drones as the second drones. The second drones form the remaining drone cluster.

[0036] For ease of explanation, in this embodiment, the first drone is referred to as the i-th drone in the drone cluster, and the remaining drone cluster is referred to as B. i The second drone is denoted as the j-th drone in the drone swarm, j∈B. i The drone swarm can then be denoted as B. +i .

[0037] Step S102: Obtain the actual mission execution environment of the first UAV and the UAV swarm mission planning model.

[0038] The drone swarm mission planning model is obtained by training an improved COMA model using a simulated mission execution environment as training samples. The improved COMA model includes a COMA network and a graph convolution module, wherein the graph convolution module is set in the COMA network.

[0039] Step S103: Input the actual task execution environment into the UAV swarm task planning model to obtain the UAV swarm task plan.

[0040] Specifically, the local states of each UAV in the actual mission execution environment are input into the UAV swarm mission planning model to obtain the mission plan of the UAV swarm in the actual mission execution environment.

[0041] Furthermore, the drone swarm task planning model used in steps S102-S103 is further explained:

[0042] The UAV swarm mission planning model is obtained by learning and training an improved COMA model using a simulated mission execution environment as training samples. The improved COMA model includes a COMA network and a graph convolutional module. Furthermore, the COMA network includes a policy network and an evaluation network connected in sequence, and the graph convolutional module is nested in the evaluation network.

[0043] The simulation task execution environment can be obtained using OPENAI's gym simulation platform or universe simulation platform.

[0044] As attached Figure 2The flowchart for drone swarm mission planning is shown below. Establishing the drone swarm mission planning model specifically includes the following steps:

[0045] Step S201: Obtain training samples, including the simulated local state S of all drones in the drone swarm at the current time, the simulated local state S′ of all drones in the drone swarm at the next time, the simulated action vector A of all drones in the drone swarm, and the simulated joint reward of all drones in the drone swarm. And the remaining drone cluster set B i .

[0046] Each training sample can be denoted as Where, done is the end signal of the training samples; S = {o t,1 o t,2 , ..., o t,N}, o t,i Let A represent the simulated local state of the i-th drone in the drone swarm at the current moment, where N is the number of drones in the swarm; A = {a t,1 a t,2 , ..., a t,N}, a t,i S′ is the simulated action vector of the i-th drone in the drone swarm at the current moment; S′={o′ t，1 ,o′ t，2 ,...,o′ t，N}, o′ t，i This represents the simulated local state of the i-th drone in the drone swarm at the next moment. This represents the simulated joint reward for the i-th drone in the drone swarm at the current moment.

[0047] Specifically, training the UAV swarm mission planning model using training samples involves the following steps:

[0048] Step S202: Obtain the simulated local state S = {o} of all drones in the drone cluster at the current moment. t,1 o t,2 , ..., o t,N};

[0049] Step S203: Calculate the simulated action vector for each UAV based on its simulated local state, specifically as follows:

[0050] The simulated local state of all drones in the drone swarm at the current moment. t,i The simulated intermediate action vector μ of each UAV is obtained by inputting it into the policy network. i (o t,i ).

[0051] The simulated intermediate motion vector μ of each drone i (o t,i The simulated motion vector a of each drone in the drone swarm is obtained by superimposing the vector a with the noise vector Q. t,i ′, the specific expression can be written as a t,i ′=μ i (o t,i )+N. The introduction of the noise vector increases the exploratory nature of the policy function.

[0052] Step S204: Set the simulated local state S = {o} of all UAVs at the current moment. t,1 o t,2 , ..., o t,N} and the remaining drone cluster set B i Input the graph convolution module to obtain the simulated local state features of the first UAV and the weights between the first UAV and each of the second UAVs.

[0053] As attached Figure 3 The graph convolutional module framework shown includes an observation-encoding layer, a convolutional layer, a fully connected network layer, and a ReLU nonlinear activation function layer. Specifically, in step S204, the simulated local state of the UAV is S = {o}. t,1 o t,2 , ..., o t,N} and the remaining drone cluster set B i Input graph convolution module to obtain simulated local state features h′ of the first UAV. i and the weights between the first drone and each of the second drones Includes the following steps:

[0054] Step S301: Set the simulated local state S = {o} of all UAVs at the current moment. t，1 o t，2 , ..., o t，N The observation coding layer is used to process the data separately to obtain the initial features of each UAV. Let h be the initial feature of the i-th UAV. t，i .

[0055] Step S302: Input the initial features of each UAV into the attention mechanism for processing to obtain the weights between the first UAV and each of the second UAVs.

[0056] The specific expression is: in, The weights between the first UAV and any second UAV in the m-th convolutional layer are: For the query linear mapping parameters in the attention mechanism of the m-th convolutional layer, h represents the keyword linear mapping parameter in the attention mechanism of the m-th convolutional layer. t，i Let h be the initial feature of the i-th UAV at time t. t，j Let be the initial features of the j-th UAV at time t.

[0057] Step S303: Set the initial features h of each UAV t，i and the weights By sequentially inputting fully connected network layers and ReLU nonlinear activation function layers, the simulated local state features h′ of each UAV are obtained. i .

[0058] The specific expression is: Where, h′ i Let M be the simulated local state features of the i-th UAV, and M be the number of convolutional layers. The M initial features are weighted, summed, and concatenated, and then processed by the function σ (including fully connected network layers and ReLU nonlinear activation function layers) to obtain the simulated local state features of the UAV.

[0059] Step S205: Transfer the simulated local state features h′ of the second UAV j The stacking process is performed to obtain the local state stacking result S of the second UAV. i，C .

[0060] Step S206: Stack the local states of the second UAV into a single result S. i，C The simulated motion vector a of the first UAV t，i ′, and the simulated joint reward of the first drone in the drone swarm. Input the evaluation network to obtain the policy parameters. This involves the following steps:

[0061] Step S401: Stack the local states of the second UAV into a single result S. i，C The simulated motion vector a of the first UAV is calculated. t,i Input Evaluation Network Obtain the median evaluation value of the evaluation network.

[0062] Step S402: For intermediate evaluation values After calibration, a joint reward was awarded in conjunction with the simulation of the first drone. The values ​​are superimposed to obtain the estimated value y of the evaluation network. i The specific expression for the estimated value of the evaluation network is: Where γ is the discount factor. The estimated value is obtained from the simulated action vector calculated by the policy network.

[0063] Step S403: Input the intermediate evaluation values ​​and the estimated values ​​of the evaluation network into the loss function to obtain the policy parameters θ. i .

[0064] The specific expression for the loss function is as follows: Where L(θ) i ) is the loss function. The intermediate evaluation value is obtained from the simulated action vectors directly obtained from the training samples.

[0065] Step S207: Based on the strategy parameter θ i Calculate the policy gradient The policy network and evaluation network are updated according to the policy gradient until the required number of updates is reached, so as to obtain the UAV swarm mission planning model.

[0066] The specific expression for the policy gradient of the policy network is:

[0067]

[0068] Where G is the number of training samples. The gradient of the policy function of the policy network. To evaluate the gradient of the network action-value function, μ i (o i Let ) be the local state vector of the policy network. i The action value selected at that time For the state space s t，i Take action a i =μ i (o i Evaluation function when ).

[0069] The difference between the final estimated value obtained by calculating the action value through the above policy network and the evaluation function calculated from the action value in the training sample is used to continuously update the policy parameters. Furthermore, the policy network and evaluation network are updated based on the continuously updated policy parameters, so that the updated policy network and evaluation network converge continuously, and the calculated evaluation value becomes closer and closer to the evaluation function value in the sample.

[0070] In addition, the joint reward calculation for the first drone includes the following steps:

[0071] Step S501: The first reward value r obtained by the interaction between the first UAV and the simulated mission execution environment. t，i The multiple second reward values ​​r obtained by each second UAV interacting with the simulated mission execution environment t，j .

[0072] The first reward value includes the collision reward r between the first drone and the obstacle.c The arrival reward r for the first drone to reach its destination g And the action reward r for the first drone to perform the action s .

[0073] Collision reward r c The specific expression is: r col = -5, where D is the distance between the drone and the obstacle. col This represents the collision distance between the drone and the obstacle.

[0074] Arrival Rewards g The specific expression is: r arr =10, ε is the guidance coefficient for the drone to approach the destination, ε=1.1, P i Let g be the location of the i-th drone, g be the location of the destination, and D be the location of the i-th drone. aim r represents the distance between the drone and its destination. uav r is the radius of the drone. aim The radius of the destination area.

[0075] The specific expression for the action reward rs is: r s =-3.

[0076] Similarly, we can obtain multiple second reward values ​​obtained by the interaction between each second drone and the simulated mission execution environment, which will not be elaborated here.

[0077] Step S502: Transfer the plurality of second reward values ​​r t，j and the weights of the first UAV and each of the second UAVs The weighted reward value of the remaining drone cluster is obtained by performing weighted processing.

[0078] Step S503: Combine the first reward value and the weighted reward value to obtain the joint reward of the first UAV. The specific expression is:

[0079] The drone swarm tasks performed in this embodiment of the invention include drone swarm cooperative communication and drone swarm physical deception, as shown in the appendix. Figure 4a and attached Figure 4b As shown, a test environment for drone swarms to perform cooperative communication tasks and physical deception is presented through simulation.

[0080] This application employs deep reinforcement learning with COMA and graph convolution to construct a drone swarm task planning model. This model allows the drone swarm to continuously update and be guided to make globally optimal actions based on the joint reward settings and the estimated value of the evaluation function. The training method combines a policy network and an evaluation network. The policy network of each drone is trained separately, and the action values ​​obtained from each drone's policy network are aggregated into the evaluation network to obtain an estimated value. This estimated value is then compared with the evaluation network value calculated from the training samples. The policy network and evaluation network are adjusted accordingly, ensuring that the updated drone swarm task planning model yields calculation results that more closely resemble the training sample values.

[0081] Furthermore, by using a large amount of data to update and adjust the policy network and evaluation network during training, the final UAV swarm mission planning model can achieve globally optimal planning. This enables the UAV swarm to simultaneously adopt centralized training and distributed execution strategies in an unknown dynamic 3D environment. In the training environment, UAVs communicate with each other and learn cooperative strategies. In the actual mission execution environment, UAVs can make decisions based solely on their own observed local states without the need for communication, which greatly shortens the decision-making time.

[0082] It should be understood that, although attached Figure 1 The steps in the flowchart are shown sequentially according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, [the following is a list of steps]. Figure 1 At least some of the steps in the process may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0083] The above-described embodiments of the present invention provide a detailed description of a drone swarm mission planning method. Since this method can be implemented using various types of devices, the present invention also discloses a drone swarm mission planning apparatus corresponding to the above method. Figure 5 The following are specific embodiments for detailed explanation.

[0084] The drone selection module 601 is used to arbitrarily select one drone in the drone cluster as the first drone, and the other drones as the second drones, with the second drones forming the remaining drone cluster.

[0085] The parameter acquisition module 602 is used to acquire the actual mission execution environment of the first UAV and the UAV cluster mission planning model.

[0086] The task planning module 603 is used to input the actual task execution environment into the UAV swarm task planning model to obtain the task plan of the UAV swarm.

[0087] The drone swarm mission planning model is obtained by training an improved COMA model using training samples from a simulated mission execution environment. The improved COMA model includes a COMA network and a graph convolution module, wherein the graph convolution module is set in the COMA network.

[0088] Specific limitations regarding the UAV swarm mission planning device can be found in the limitations of the UAV swarm mission planning method described above, and will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the terminal device's processor, or stored in software within the terminal device's memory, allowing the processor to call and execute the corresponding operations of each module.

[0089] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described UAV swarm mission planning method.

[0090] The computer-readable storage medium may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), hard disk, or ROM. Optionally, the computer-readable storage medium includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products, and the program code may be compressed in an appropriate form.

[0091] In one embodiment, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the steps of the above-described UAV swarm mission planning method.

[0092] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs can be stored in the memory and configured to be executed by one or more processors, and the one or more applications are configured to perform the above-described UAV swarm mission planning method.

[0093] A processor may include one or more processing cores. The processor connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0094] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the terminal device during use.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for planning unmanned aerial vehicle (UAV) swarm missions, characterized in that, The method includes: In the drone swarm, one drone is randomly selected as the first drone, and the other drones are selected as the second drones. The second drones form the remaining drone swarm. Obtain the actual mission execution environment of the first UAV and the UAV swarm mission planning model; The actual task execution environment is input into the UAV swarm task planning model to obtain the UAV swarm task plan; The drone swarm mission planning model is obtained by training an improved COMA model using a simulated mission execution environment as training samples. The improved COMA model includes a COMA network and a graph convolutional module, wherein the graph convolutional module is set in the COMA network. The COMA network includes a policy network and an evaluation network connected in sequence, wherein the graph convolutional module is nested in the evaluation network; establishing the UAV swarm mission planning model includes: Obtain training samples, which include the simulated local state of all drones in the drone swarm at the current moment, the simulated local state of all drones in the drone swarm at the next moment, the simulated action vectors of all drones in the drone swarm, and the simulated joint reward of all drones in the drone swarm. The simulated local states of all drones in the drone cluster at the current moment are input into the policy network to obtain the simulated intermediate action vector of each drone. The simulated intermediate motion vector of each UAV is superimposed with the noise vector to obtain the simulated motion vector of each UAV in the UAV swarm. Input the simulated local states of all UAVs at the current moment and the set of remaining UAV clusters into the graph convolution module to obtain the simulated local state features of the first UAV and the weights between the first UAV and each of the second UAVs. The simulated local state features of the second UAV are stacked to obtain the local state stacking result of the second UAV. The local state stacking results of the second UAV, the simulated action vector of the first UAV calculated by the policy network, and the simulated joint reward of the first UAV in the UAV cluster are input into the evaluation network to obtain policy parameters. The policy gradient is calculated based on the policy parameters, and the policy network and evaluation network are updated based on the policy gradient until the required number of updates is reached, so as to obtain the UAV swarm task planning model.

2. The UAV swarm mission planning method as described in claim 1, characterized in that, The graph convolution module includes an observation-encoding layer, a convolutional layer, a fully connected network layer, and a ReLU nonlinear activation function layer. The step of inputting the simulated local states of all UAVs at the current moment and the remaining UAV cluster set into the graph convolution model to obtain the simulated local state features of the first UAV and the weights between the first UAV and each of the second UAVs includes: The simulated local states of all UAVs at the current moment are processed by the observation coding layer to obtain the initial features of each UAV. The initial features of each UAV are input into the attention mechanism for processing to obtain the weights between the first UAV and each of the second UAVs; The initial features and weights of each UAV are sequentially input into a fully connected network layer and a ReLU nonlinear activation function layer to obtain the simulated local state features of each UAV.

3. The UAV swarm mission planning method as described in claim 1, characterized in that, The process of inputting the local state stacking result of the second UAV, the simulated action vector of the first UAV calculated by the policy network, and the simulated joint reward of the first UAV in the UAV swarm into the evaluation network to obtain policy parameters includes: The local state stacking result of the second UAV and the simulated action vector of the first UAV calculated by the policy network are input into the evaluation network to obtain the intermediate evaluation value of the evaluation network. The intermediate evaluation value is multiplied by the discount factor and then superimposed with the simulated joint reward of the first UAV to obtain the estimated value of the evaluation network. The intermediate evaluation values ​​and estimated values ​​of the evaluation network are input into the loss function to obtain the policy parameters.

4. The UAV swarm mission planning method as described in claim 1, characterized in that, The calculation of the joint reward for the first drone includes: The first reward value obtained by the interaction between the first UAV and the simulated mission execution environment; and the multiple second reward values ​​obtained by the interaction between each second UAV and the simulated mission execution environment. The weighted reward values ​​of the remaining drone cluster are obtained by weighting the multiple second reward values ​​and the weights of the first drone and each second drone. The first reward value and the weighted reward value are combined to obtain the joint reward for the first UAV.

5. The UAV swarm mission planning method as described in claim 4, characterized in that, The first reward value includes: The first drone's collision reward, the first drone's arrival reward, and the first drone's action reward are all based on their actions.

6. The UAV swarm mission planning method as described in claim 1, characterized in that, Calculating the simulated local state of the first drone in the drone swarm at the next moment includes: The simulated motion vector of the first UAV is interacted with the simulated task execution environment to obtain the simulated local state of the first UAV at the next moment.

7. A drone swarm mission planning device, characterized in that, The device includes: The drone selection module is used to arbitrarily select one drone in the drone cluster as the first drone, and the other drones as the second drones. The second drones form the remaining drone cluster. The parameter acquisition module is used to acquire the actual mission execution environment of the first UAV and the UAV cluster mission planning model. The task planning module is used to input the actual task execution environment into the UAV swarm task planning model to obtain the task plan of the UAV swarm. The drone swarm mission planning model is obtained by training an improved COMA model using training samples from a simulated mission execution environment. The improved COMA model includes a COMA network and a graph convolutional module, wherein the graph convolutional module is set in the COMA network. The COMA network includes a policy network and an evaluation network connected in sequence, wherein the graph convolutional module is nested in the evaluation network; establishing the UAV swarm mission planning model includes: Obtain training samples, which include the simulated local state of all drones in the drone swarm at the current moment, the simulated local state of all drones in the drone swarm at the next moment, the simulated action vectors of all drones in the drone swarm, and the simulated joint reward of all drones in the drone swarm. The simulated local states of all drones in the drone cluster at the current moment are input into the policy network to obtain the simulated intermediate action vector of each drone. The simulated intermediate motion vector of each UAV is superimposed with the noise vector to obtain the simulated motion vector of each UAV in the UAV swarm. Input the simulated local states of all UAVs at the current moment and the set of remaining UAV clusters into the graph convolution module to obtain the simulated local state features of the first UAV and the weights between the first UAV and each of the second UAVs. The simulated local state features of the second UAV are stacked to obtain the local state stacking result of the second UAV. The local state stacking results of the second UAV, the simulated action vector of the first UAV calculated by the policy network, and the simulated joint reward of the first UAV in the UAV cluster are input into the evaluation network to obtain policy parameters. The policy gradient is calculated based on the policy parameters, and the policy network and evaluation network are updated based on the policy gradient until the required number of updates is reached, so as to obtain the UAV swarm task planning model.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of any one of the UAV swarm mission planning methods according to claims 1-6.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs any one of the UAV swarm mission planning methods according to claims 1-6.

Images