Unmanned aerial vehicle cluster formation method, device, equipment and storage medium

By using Markov decision-making process and Actor-Critic network model in the drone cluster combined with the maximum entropy model-free algorithm and reward function, the problem of formation adjustment of the drone cluster in a changing environment is solved, and a more stable formation is achieved and stronger coping skills.

CN119088076AActive Publication Date: 2024-12-06CHINA ORDNANCE EQUIP GRP AUTOMATION RES INST CO LTD

Patent Information

Application Number
CN202411149224.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-06
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

The existing drone cluster formation method cannot adjust the formation in time when facing a changing environment, resulting in mission failure.

Method used

Based on Markov's decision-making process, the drone motion model is established, and the Actor-Critic network model is combined with the maximum entropy model-free algorithm and reward function to achieve dynamic adjustment and stability of the drone cluster formation.

Benefits of technology

It enhances the ability of drone clusters to respond to environmental changes, avoids local optimal traps, and improves the stability of the formation and the ability to respond to external changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088076A_ABST
    Figure CN119088076A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned aerial vehicle cluster formation method, device and equipment and a storage medium, the method has the capability of coping with environment changes, and meanwhile, the method designs a reward function for the formation shape problem of the formation, and improves the formation stability capability. And by adopting the SAC algorithm, the environment is prevented from repeatedly falling into local optimum, and the capacity of the system for coping with the external environment change is enhanced. The reward function is adopted for formation, and the formation stability is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of drone cluster technology, and in particular to a drone cluster formation method, device, equipment and storage medium based on deep reinforcement learning. Background Art

[0002] In the mission execution of UAV formation, formation control is an important research topic. Existing formation methods mainly complete control through hierarchical dimension reduction. Commonly used algorithms are generally genetic algorithms, simulated annealing algorithms, ant colony optimization algorithms, etc.

[0003] However, the formation method in the existing technology relies on environmental information obtained in advance. When the drone cluster faces a changing environment, it cannot make formation adjustments in time, resulting in mission failure. Summary of the invention

[0004] In view of the above problems, the present invention provides a method, device, equipment and storage medium for overcoming the above problems or at least partially solving the above problems. The method is based on the model established by the Markov decision process, and by setting dynamic obstacles, the ability of the system to face a changing environment is enhanced, and by setting a formation reward function, the formation of the drone cluster is maintained.

[0005] The present invention provides the following scheme:

[0006] A drone swarm formation method, comprising:

[0007] Establishing a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set;

[0008] The information of the UAV motion model is brought into the Actor-Critic network model, and the network parameters of the Actor network and the Critic network are updated according to the maximum entropy model-free algorithm using a centralized training-distributed decision-making architecture;

[0009] Substituting the obtained network parameters into the reward function of the reward mechanism to obtain a reward value;

[0010] The Actor network and the Critic network perform calculations cyclically until the reward value of the reward function converges, and the model ends the calculation to obtain the action parameters required to be executed by each drone.

[0011] Preferably: the environmental information includes the two-dimensional positions of all drones, the headings of all drones, the speeds of all drones, and the target point positions of all drones;

[0012] The observation information includes the two-dimensional position of the current drone; the heading of the current drone; the speed of the current drone; the distance between the current drone and nearby drones;

[0013] The action set includes move left, move right, move forward, move backward, and hold.

[0014] Preferably: the reward mechanism design principle is that the closer the drone is to the target point, the greater the reward value; the closer the distance between the two drones is to the optimal distance, the greater the reward value. The reward mechanism includes a reward function for guiding the drone to fly to the target point, a reward function when the drone collides with an obstacle, a distance reward function between the drone and its nearby drones, and a reward function for the drone to arrive near the target point.

[0015] Preferably: the reward function for guiding the heading of the drone is expressed by the following formula:

[0016]

[0017] Where: θ represents the flight angle of the UAV guided by the target point;

[0018] The reward function of the distance between the target point and the destination is expressed as follows:

[0019]

[0020] Where: d_target represents the Euclidean distance between the current UAV and the target point, and d_pre represents the Euclidean distance between the previous UAV and the target point;

[0021] The reward function for guiding the drone to avoid obstacles is expressed as follows:

[0022]

[0023] The reward function for guiding the UAV to maintain the formation with its nearby UAVs is expressed as follows:

[0024]

[0025] Where: d min Represents the safe distance between drones, d target represents the optimal distance between drones, d max Indicates the maximum distance between drones, d ij Indicates the distance between the drone and its nearby drones, d err Indicates the error threshold between two drones. This reward and punishment system is designed to maintain the formation of the drones.

[0026] The sparse reward function of the drone arriving near the target point is expressed as follows:

[0027]

[0028] In the formula, δd represents the set distance threshold;

[0029] The reward function that the drone relies on is as follows:

[0030] R i =k1r1+k2r2+k3r3+k4r4+k5r5

[0031] In the formula, k1, k2, k3, k4 are weighted coefficients, and r1, r2, r3, r4 are guiding rewards.

[0032] Preferably: the model-free algorithm based on maximum entropy updates the parameters of the Actor network and the Critic network, the Actor network inputs the observation information of the drone and outputs the action performed by the drone, the Critic network inputs the observation information and action information of the drone and outputs the evaluation Q value.

[0033] Preferably: the maximum entropy model-free algorithm adopts a centralized training and distributed execution architecture:

[0034] The state information and actions of the drone are input into the Critic network, and the calculation formula for the evaluation Q value is as follows:

[0035]

[0036] Where i represents the i-th drone, i = 1, 2, 3, ..., N, j represents the number of Critic networks, j = 1, 2, r is the reward value, γ is the discount factor, d is the round termination signal, s ′ Indicates the environmental state of the current drone at the next moment. represents the action of other drones at the next moment, θ represents the network parameters of Actor, represents the network parameters of Critic, π θ (a|s) is the probability that the strategy takes action a in state s at the next moment, It represents the target Q value calculated by the drone when performing action a in the environment state s;

[0037] According to the gradient descent method, the parameters are updated and the loss function is calculated as follows:

[0038]

[0039] In the formula, a i ={…,a N-1 ,…}, represents the Q value calculated when the drone takes action a in environment s;

[0040] The Actor network inputs the observation information of the drone, outputs the action performed by the drone, and updates the parameters according to the gradient ascent method. The loss function calculation formula is:

[0041]

[0042] a other Indicates the action of the drone at the current moment;

[0043] Use the gradient descent method to update the entropy regularization coefficient α:

[0044]

[0045] In the formula, H represents the initial value of entropy

[0046] Soft update target network parameters:

[0047]

[0048] Where τ represents the soft update parameter.

[0049] Preferably, the output action is updated according to the policy function, the policy network inputs the drone observation information and policy network parameters at the current moment, and outputs the action executed by the drone at the current moment.

[0050] A drone swarm formation device, used to execute the above-mentioned drone swarm formation method, the device comprising:

[0051] A motion model building unit, used to build a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set;

[0052] The parameter input unit is used to bring the information of the UAV motion model into the Actor-Critic network model. The centralized training-distributed decision-making architecture is adopted to update the network parameters of the Actor network and the Critic network according to the maximum entropy model-free algorithm.

[0053] A reward function calculation unit, used for bringing the obtained network parameters into the reward function of the reward mechanism to obtain a reward value;

[0054] The action parameter acquisition unit is used for the Actor network and the Critic network to perform cyclic calculations until the reward value of the reward function converges and the model ends the calculation to obtain the actions that each drone needs to perform.

[0055] A drone swarm formation device, the device comprising a processor and a memory:

[0056] The memory is used to store program code and transmit the program code to the processor;

[0057] The processor is used to execute the above-mentioned drone cluster formation method according to the instructions in the program code.

[0058] A computer-readable storage medium is used to store program code, and the program code is used to execute the above-mentioned drone cluster formation method.

[0059] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0060] The embodiment of the present application provides a method, device, equipment and storage medium for drone swarm formation, which has the ability to cope with environmental changes. At the same time, the method designs a reward function for the formation problem of the formation, thereby improving the stability of the formation. The SAC algorithm is used to avoid the environment from repeatedly falling into the local optimum, and the system's ability to cope with changes in the external environment is enhanced. The reward function is used for formation to achieve the stability of the formation.

[0061] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1 is a flow chart of a method for forming a swarm of unmanned aerial vehicles based on reinforcement learning provided by an embodiment of the present invention;

[0064] Figure 2 is a structural diagram of a model-free deep learning algorithm for maximum entropy in an embodiment of the present invention;

[0065] Figure 3 is a schematic diagram of a drone cluster formation device provided by an embodiment of the present invention;

[0066] Figure 4 It is a schematic diagram of a drone cluster formation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0068] See also Figure 1-Figure 2 , a drone cluster formation method provided by an embodiment of the present invention, the method may include:

[0069] S101: Establishing a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information, and an action set; in specific implementation, the embodiment of the present application can provide that the environmental information includes the two-dimensional position of all UAVs; the heading of all UAVs; the speed of all UAVs; and the target point position of all UAVs;

[0070] The observation information includes the two-dimensional position of the current drone; the heading of the current drone; the speed of the current drone; the distance between the current drone and nearby drones;

[0071] The action set includes move left, move right, move forward, move backward, and hold.

[0072] S102: Bring the UAV motion model information described in S101 into the Actor-Critic network model, adopt a centralized training-distributed decision-making architecture, and update the network parameters of the Actor network and the Critic network according to the maximum entropy model-free algorithm; the algorithm includes 1 Actor network, 1 Actor target network, and selects the smaller value of the two Critic as the target Q value; in specific implementation, input the status information and action of the UAV into the Critic network, and the calculation formula for the evaluation Q value is as follows:

[0073]

[0074] Where i represents the i-th drone, i = 1, 2, 3, ..., N, j represents the number of Critic networks, j = 1, 2, r is the reward value, γ is the discount factor, d is the round termination signal, s ′ Indicates the environmental state of the current drone at the next moment. represents the action of other drones at the next moment, θ represents the network parameters of Actor, represents the network parameters of Critic, π θ (a|s) is the probability that the strategy takes action a in state s at the next moment, It represents the target Q value calculated by the drone when performing action a in the environment state s;

[0075] According to the gradient descent method, the parameters are updated and the loss function is calculated as follows:

[0076]

[0077] In the formula, a i ={…,a N-1 ,…}, represents the Q value calculated when the drone takes action a in environment s;

[0078] The Actor network inputs the observation information of the drone, outputs the action performed by the drone, and updates the parameters according to the gradient ascent method. The loss function calculation formula is:

[0079]

[0080] a other Indicates the action of the drone at the current moment;

[0081] Use the gradient descent method to update the entropy regularization coefficient α:

[0082]

[0083] In the formula, H represents the initial value of entropy

[0084] Soft update target network parameters:

[0085]

[0086] Where τ represents the soft update parameter.

[0087] S103: Bring the network parameters obtained in step S102 into the reward function of the reward mechanism to obtain the reward value; the design principle of the reward mechanism is that the closer the drone is to the target point, the greater the reward value; the closer the distance between the two drones is to the optimal distance, the greater the reward value, which mainly includes the reward function for guiding the drone to fly to the target point, the reward function when the drone collides with an obstacle, the distance reward function between the drone and its nearby drones, and the reward function when the drone arrives near the target point. The specific implementation is:

[0088] The reward function for guiding the drone's heading is expressed as follows:

[0089]

[0090] Where: θ represents the flight angle of the UAV guided by the target point;

[0091] The reward function of the distance between the guiding target point and the destination is expressed as follows:

[0092]

[0093] Where: d_target represents the Euclidean distance between the current drone and the target point, and d_pre represents the Euclidean distance between the drone and the target point in the previous step. When the drone moves toward the target point, it is rewarded, and when the drone moves away from the target point or remains stationary, it is punished;

[0094] The reward function for guiding the drone to avoid obstacles is expressed as follows:

[0095]

[0096] The reward function for guiding the UAV to maintain the formation with its nearby UAVs is expressed as follows:

[0097]

[0098] Where: d min Represents the safe distance between drones, d target represents the optimal distance between drones, d max Indicates the maximum distance between drones, d ij Indicates the distance between the drone and its nearby drones, d err Indicates the error threshold between two drones. This reward and penalty system is to maintain the formation of the drones.

[0099] The sparse reward function of the drone arriving near the target point is expressed as follows:

[0100]

[0101] In the formula, δd represents the set distance threshold. If the reward is triggered, the drone ends the current round of training.

[0102] To summarize, the reward function that the drone relies on is as follows:

[0103] R i =k1r1+k2r2+k3r3+k4r4+k5r5

[0104] Where k1, k2, k3, and k4 are weighted coefficients, r1, r2, r3, and r4 are guiding rewards. Each time the drone changes state, it will obtain guiding rewards. r4 is a sparse reward. When triggered, the drone will end the training of this round.

[0105] S104: The Actor network and the Critic network perform cyclic calculations until convergence, and each drone outputs the action parameters to be executed according to the strategy function.

[0106] In summary, the drone cluster formation method provided by this application has the ability to cope with environmental changes. At the same time, this method adopts a reward function to improve the formation ability in response to the problem of autonomous change of the formation. The SAC algorithm is used to avoid the environment from repeatedly falling into the local optimum, and the system's ability to cope with changes in the external environment is enhanced. The use of a reward function makes the drone cluster formation more stable and improves the formation ability.

[0107] See also Figure 3 , the embodiment of the present application can also provide a drone cluster formation device, such as Figure 4 As shown, for executing the above-mentioned drone cluster formation method, the device may include:

[0108] The motion model building unit 301 is used to build a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set;

[0109] The parameter input unit 302 is used to bring the information of the UAV motion model into the Actor-Critic network model, adopt a centralized training-distributed decision-making architecture, and update the network parameters of the Actor network and the Critic network according to the maximum entropy model-free algorithm;

[0110] A reward function calculation unit 303 is used to bring the obtained network parameters into the reward function of the reward mechanism to obtain a reward value;

[0111] The action parameter acquisition unit 304 is used for the Actor network and the Critic network to perform cyclic calculations until the reward value of the reward function converges, the model ends the calculation, and the actions that each drone needs to perform are obtained.

[0112] The embodiment of the present application may also provide a drone cluster formation device, the device comprising a processor and a memory:

[0113] The memory is used to store program code and transmit the program code to the processor;

[0114] The processor is used to execute the steps of the above-mentioned drone cluster formation method according to the instructions in the program code.

[0115] like Figure 4 As shown, a drone cluster formation device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other through the communication bus 13.

[0116] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0117] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the drone cluster formation method.

[0118] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0119] Establishing a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set;

[0120] The information of the UAV motion model is brought into the Actor-Critic network model, and the network parameters of the Actor network and the Critic network are updated according to the maximum entropy model-free algorithm using a centralized training-distributed decision-making architecture;

[0121] Substituting the obtained network parameters into the reward function of the reward mechanism to obtain a reward value;

[0122] The Actor network and the Critic network perform calculations cyclically until the reward value of the reward function converges, and the model ends the calculation to obtain the action parameters required to be executed by each drone.

[0123] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0124] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0125] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0126] Of course, it should be noted that Figure 4 The structure shown does not constitute a limitation on the drone cluster formation device in the embodiment of the present application. In actual applications, the drone cluster formation device may include Figure 4More or fewer components than shown, or combinations of certain components.

[0127] An embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the steps of the above-mentioned drone cluster formation method.

[0128] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0129] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0130] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for forming a swarm of drones, characterized in that: include: Establishing a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set; The information of the UAV motion model is brought into the Actor-Critic network model, and the network parameters of the Actor network and the Critic network are updated according to the maximum entropy model-free algorithm using a centralized training-distributed decision-making architecture; Substituting the obtained network parameters into the reward function of the reward mechanism to obtain a reward value; The Actor network and the Critic network perform calculations cyclically until the reward value of the reward function converges, and the model ends the calculation to obtain the action parameters required to be executed by each drone.

2. The drone swarm formation method according to claim 1, characterized in that: The environmental information includes the two-dimensional position of all drones, the heading of all drones, the speed of all drones, and the target point position of all drones; The observation information includes the two-dimensional position of the current drone; the heading of the current drone; the speed of the current drone; the distance between the current drone and nearby drones; The action set includes move left, move right, move forward, move backward, and hold.

3. The drone swarm formation method according to claim 1, characterized in that: The design principle of the reward mechanism is that the closer the drone is to the target point, the greater the reward value; the closer the distance between the two drones is to the optimal distance, the greater the reward value. The reward mechanism includes a reward function for guiding the drone to fly to the target point, a reward function when the drone collides with an obstacle, a distance reward function between the drone and its nearby drones, and a reward function for the drone to arrive near the target point.

4. The drone swarm formation method according to claim 3, characterized in that: The reward function for guiding the drone's heading is expressed as follows: Where: θ represents the flight angle of the UAV guided by the target point; The reward function of the distance between the target point and the destination is expressed as follows: Where: d_target represents the Euclidean distance between the current UAV and the target point, and d_pre represents the Euclidean distance between the previous UAV and the target point; The reward function for guiding the drone to avoid obstacles is expressed as follows: The reward function for guiding the UAV to maintain the formation with its nearby UAVs is expressed as follows: Where: d min Represents the safe distance between drones, d target represents the optimal distance between drones, d max Indicates the maximum distance between drones, d ij Indicates the distance between the drone and its nearby drones, d err Indicates the error threshold between two drones. This reward and punishment system is designed to maintain the formation of the drones. The sparse reward function of the drone arriving near the target point is expressed as follows: In the formula, δd represents the set distance threshold; The reward function that the drone relies on is as follows: <h2 style=";text-align:left;direction:ltr">R<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =k1r1+k2r2+k3r3+k4r4+k5r5 In the formula, k1, k2, k3, k4 are weighted coefficients, and r1, r2, r3, r4 are guiding rewards.

5. The drone swarm formation method according to claim 1, characterized in that: The model-free algorithm based on maximum entropy updates the parameters of the Actor network and the Critic network. The Actor network inputs the observation information of the drone and outputs the action performed by the drone. The Critic network inputs the observation information and action information of the drone and outputs the evaluation Q value.

6. The drone swarm formation method according to claim 5, characterized in that: The model-free algorithm for maximum entropy uses a centralized training and distributed execution architecture: The state information and actions of the drone are input into the Critic network, and the calculation formula for the evaluation Q value is as follows: Where i represents the i-th drone, i = 1, 2, 3, ..., N, j represents the number of Critic networks, j = 1, 2, r is the reward value, γ is the discount factor, d is the round termination signal, s ′ Indicates the environmental state of the current drone at the next moment. represents the action of other drones at the next moment, θ represents the network parameters of Actor, represents the network parameters of Critic, π θ (a|s) is the probability that the strategy takes action a in state s at the next moment, It represents the target Q value calculated by the drone when performing action a in the environment state s; According to the gradient descent method, the parameters are updated and the loss function is calculated as follows: In the formula, a i =*…,a N-1 ,…+, represents the Q value calculated when the drone takes action a in environment s; The Actor network inputs the observation information of the drone, outputs the action performed by the drone, and updates the parameters according to the gradient ascent method. The loss function calculation formula is: a other Indicates the action of the drone at the current moment; Use the gradient descent method to update the entropy regularization coefficient α: In the formula, H represents the initial value of entropy Soft update target network parameters: Where τ represents the soft update parameter.

7. The drone output action according to claim 1, characterized in that: The output action is updated according to the policy function, and the policy network inputs the drone observation information and policy network parameters at the current moment, and outputs the action performed by the drone at the current moment.

8. A drone cluster formation device, characterized in that: Used to execute the drone swarm formation method according to any one of claims 1 to 7, the device comprises: A motion model building unit, used to build a UAV motion model based on a Markov decision process; the UAV motion model includes environmental information, observation information and an action set; The parameter input unit is used to bring the information of the UAV motion model into the Actor-Critic network model. The centralized training-distributed decision-making architecture is adopted to update the network parameters of the Actor network and the Critic network according to the maximum entropy model-free algorithm. A reward function calculation unit, used for bringing the obtained network parameters into the reward function of the reward mechanism to obtain a reward value; The action parameter acquisition unit is used for the Actor network and the Critic network to perform cyclic calculations until the reward value of the reward function converges and the model ends the calculation to obtain the actions that each drone needs to perform.

9. A drone cluster formation device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the drone cluster formation method described in any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the drone cluster formation method described in any one of claims 1-7.

Citation Information

Patent Citations

  • Formation changing method for intensive autonomous formations of unmanned aerial vehicle

    CN103941728A

  • Distributed formation method of unmanned aerial vehicle cluster based on reinforcement learning

    CN110007688A

  • Unmanned aerial vehicle formation collision avoidance method based on deep reinforcement learning

    CN114020013A

  • Deep reinforcement learning formation transformation method and system based on dynamic target allocation

    CN114237293A

  • Unmanned aerial vehicle formation path planning method based on reinforcement learning

    CN116774731A

Cited By

  • Multi-quadrotor safe formation control method and system based on event triggering self-locking prevention

    CN122018559A

  • A multi-four-rotor safe formation control method and system based on event-triggered anti-locking

    CN122018559B