Distributed uav cluster cooperative task planning method based on knowledge enhancement and reinforcement learning

Abstract

This application discloses a distributed UAV swarm cooperative task planning method based on knowledge enhancement and reinforcement learning, comprising: constructing an original expert database by interacting with an expert policy and a simulation environment and performing intermittent data collection; performing distributed preprocessing on the original expert database to obtain a preprocessed expert dataset; cloning and pre-training multiple initialized policy networks based on the preprocessed expert dataset to obtain a pretrained policy network set; interacting with the simulation environment based on the pretrained policy network set to collect reinforcement learning experience data and storing it in an experience replay pool to obtain an updated experience replay pool; and jointly training the pretrained policy network set and the initialized value network based on the updated experience replay pool to obtain a UAV swarm cooperative task planning policy. This application effectively combines prior knowledge and reinforcement learning, enabling efficient and stable UAV swarm cooperative task planning.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) swarm control technology, and in particular to a distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning. Background Technology

[0002] With the widespread application of drone swarms in military reconnaissance, coordinated strikes, and disaster relief, achieving efficient and intelligent collaborative mission planning in complex and dynamic environments has become a key technological challenge. Traditional methods mainly rely on centralized pre-programming or rule engines, which are difficult to adapt to highly dynamic and uncertain battlefield environments.

[0003] In existing technologies, there are two main technical approaches: one is expert policy generation based on large language models, which can quickly output executable policies, but its decision-making relies on text reasoning, has limited ability to model high-dimensional continuous state spaces, and lacks closed-loop feedback optimization with the environment, resulting in an upper limit to policy performance; the other is methods based on multi-agent reinforcement learning, such as multi-agent proximal policy optimization algorithms, which can adapt to complex environments through trial and error learning, but suffer from problems such as low exploration efficiency in the early stages of training, slow convergence speed, and easy getting trapped in local optima. Summary of the Invention

[0004] The main purpose of this application is to provide a distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning, which aims to solve the technical problems of low training efficiency and instability of reinforcement learning and limited performance of expert policies in UAV swarm task planning in the prior art.

[0005] To achieve the above objectives, this application proposes a distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning. The method includes: An original expert database is constructed by interacting with the expert strategy and simulation environment and collecting intermittent data. The original expert database is subjected to distributed preprocessing to obtain a preprocessed expert dataset; Based on the preprocessed expert dataset, multiple initialized policy networks are cloned and pre-trained to obtain a set of pre-trained policy networks, where each policy network corresponds to a UAV agent. Based on the pre-trained policy network set, the simulation environment is interacted to collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool. Based on the updated experience replay pool, the pre-trained policy network set and the initialized value network are jointly trained to obtain the UAV swarm collaborative task planning strategy.

[0006] Furthermore, to achieve the above objectives, this application also proposes a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning. The distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning includes: The data acquisition module is used to interact with the simulation environment based on expert strategies and to perform intermittent data acquisition to build the original expert database. The preprocessing module is used to perform distributed preprocessing on the original expert database to obtain a preprocessed expert dataset. The pre-training module is used to clone and pre-train multiple initialized policy networks based on the preprocessed expert dataset to obtain a set of pre-trained policy networks, wherein each policy network corresponds to a UAV agent. The interaction module is used to interact with the simulation environment based on the pre-trained policy network set, collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool. The joint training module is used to jointly train the pre-trained policy network set and the initialized value network based on the updated experience replay pool to obtain the UAV swarm collaborative task planning strategy.

[0007] Furthermore, to achieve the above objectives, this application also proposes a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described above.

[0008] In addition, to achieve the above objectives, this application also proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described above.

[0009] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described above.

[0010] The one or more technical solutions proposed in this application significantly improve training efficiency, policy stability, and multi-machine collaborative capabilities by combining expert strategies with simulation environments for interactive data acquisition, utilizing behavioral cloning to pre-train policy networks, and then optimizing policy networks and value networks through joint training, thereby achieving efficient and stable UAV swarm collaborative task planning. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating an embodiment of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in this application. Figure 2 This is a framework diagram of expert data preprocessing provided in an embodiment of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in this application. Figure 3 This is a diagram of the training framework for a drone swarm agent based on behavior cloning, provided in an embodiment of the distributed drone swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in this application. Figure 4 This is a flowchart of a knowledge-enhanced UAV swarm collaborative task planning algorithm provided in an embodiment of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in this application.

[0014] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0016] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0017] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone; or an electronic device capable of performing the above functions, such as a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning. The following description uses a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning as an example to illustrate this embodiment and the subsequent embodiments.

[0018] Based on this, embodiments of this application provide a distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in this application.

[0019] In this embodiment, the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning includes steps S10~S50: Step S10: Based on the expert strategy, interact with the simulation environment and perform intermittent data collection to build the original expert database.

[0020] It should be noted that expert strategy refers to a high-level intelligent agent strategy module built on a large language model that can generate reasonable collaborative decisions based on the overall battlefield situation. It can guide the exploration direction in the early stage of reinforcement learning, alleviate the sparse reward problem, and constitute the core source of the "knowledge enhancement" mechanism.

[0021] The simulation environment refers to a high-fidelity multi-agent air combat simulation platform used to simulate the dynamic interaction between a swarm of drones and an enemy air defense system. The simulation environment includes the following core components: our entities, namely N controllable drones, each with a position p. i ∈R 2 Orientation θ i ∈[0,2π), speed, weapon status, and survival flag l i ∈{0,1}; Enemy entities, i.e., several fixed or moving anti-aircraft towers (or enemy aircraft), possessing detection range, attack capability, and health points; State transition function, i.e., given the current global state s t and combined action a t The environment outputs the next state s with deterministic / random characteristics. t+1 Reward mechanisms that support multi-level feedback (such as hit rewards, sacrifice contributions, task completion, etc.); observability design, i.e., global state s t The expert strategy and value network are visible, but only partial observations are available for individual UAV strategies. The simulation environment provides a safe, repeatable, and low-cost training and evaluation platform, supports large-scale parallel data acquisition, and enables digital twin tactical verification.

[0022] Intermittent data acquisition refers to a non-continuous, conditionally triggered expert trajectory sampling mechanism designed to avoid redundant storage and improve data efficiency and diversity. Intermittent data acquisition includes action change determination and frequency control determination, which can avoid a large number of repetitive state-action pairs, thereby reducing storage overhead, increasing sample information content, and preventing overfitting of static behavior.

[0023] The original expert database is a structured data set generated by an intermittent acquisition process, with global environment state-expert joint action pairs as the basic unit. The database size is controlled by the preset number of expert data acquisition rounds, such as 500 rounds. The original expert database serves as a carrier of prior knowledge, realizing "knowledge injection" and providing initial feasible solutions for the policy network, which greatly shortens the convergence time of reinforcement learning.

[0024] In one feasible implementation, step S10 may include: acquiring an initial global environment state of the simulation environment; inputting the initial global state into an expert policy constructed based on a large language model; acquiring initial expert joint actions generated by the expert policy, wherein the expert joint actions include actions that each UAV in the UAV cluster should perform; constructing a first global environment state-expert joint action expert data pair based on the initial global environment state and the initial expert joint actions; executing the initial expert joint actions in the simulation environment to obtain an updated global environment state; inputting the updated global environment state back into the expert policy to obtain a next expert joint action generated by the expert policy; determining whether there is a difference between the next expert joint action and the initial expert joint action; if there is, determining whether the current sampling frequency has reached a preset sampling frequency; if it has, constructing a second global environment state-expert joint action expert data pair based on the updated global environment state and the next expert joint action; repeating multiple rounds of simulation tasks until a preset number of expert data acquisition rounds are completed to obtain an original expert database containing multiple global environment state-expert joint action expert data pairs.

[0025] It should be noted that in this implementation, the expert strategy is fine-tuned by a large language model or constructed through prompting engineering. Its input is the complete global environment state, and its output is the coordinated actions of the drone swarm. This strategy simulates the high-order collaborative logic of human commanders or tactical AI, such as: grouping and encircling enemy targets, assigning decoy drones to draw fire, dynamically allocating strike missions, and avoiding friendly fire collisions. Its advantage lies in the fact that it eliminates the need for manually written rules; instead, it automatically generates structured action instructions through the LLM's understanding of tactical semantics.

[0026] Understandably, expert data acquisition primarily involves interaction between expert strategies and the simulation environment to construct corresponding <data, label> pairs. However, since the environmental state between adjacent steps in the simulation environment is not significantly different, collecting data for every step would result in excessive redundancy. Therefore, an intermittent expert data acquisition strategy is proposed. This strategy controls the variability of sampled data by adjusting the actions given by the expert strategy and the sampling frequency, thereby reducing the redundancy of the final generated expert data.

[0027] In this embodiment, an intermittent expert data acquisition method is designed to reduce redundant data, and global environmental information is converted into local observation inputs for each UAV, combined with one-hot encoding of the UAV number. The effectiveness of the sampled data is improved by dual constraints of action differences and sampling frequency; local observation mapping maintains the independent perspective and differences of each agent, so that the training samples meet the requirements of centralized training-distributed execution, fundamentally improving the generalization performance of the cooperative strategy.

[0028] It is worth noting that the simulation environment reads the complete battlefield situation information at the start of the mission, i.e., the initial global environment state, as the input basis for expert strategy decision-making. The initial global environment state is a structured vector containing: the two-dimensional position (x, y) of each entity; the orientation angle θ ∈ [0, 2π); the survival status l ∈ {0, 1} (1 indicates survival, 0 indicates destruction); and may also include extended attributes such as speed and remaining weapon quantity, which are not specifically limited in this embodiment. The initial global environment state can ensure that the generated actions are tactically reasonable, such as avoiding ordering dead drones to perform missions.

[0029] The initial global environment state is encoded into natural language or structured prompts and input into a large language model (LLM) that has been fine-tuned or optimized using prompt engineering. The LLM outputs a set of coordinated action instructions for the entire drone swarm, i.e., the initial expert joint action. This instruction set covers the specific actions that each drone in the swarm should perform, ensuring the coordination and consistency of the swarm's actions. For example, the LLM might output: "Drone 1 attacks anti-aircraft tower A, Drone 2 provides cover, Drone 3 lures the enemy, the rest maintain patrol." This output is then parsed into a legal action vector 'a' in the discrete action space. expert =[a (1) ,a (2) ,...,a (N) ].

[0030] Pair the current state with the corresponding action to construct <state, action>, forming a supervised learning sample. The global environment state and expert joint action data pairs are generated and stored in the original expert database. The actions generated by the experts are converted into physical control commands, input into the simulation engine, and the environmental state is advanced to the next time step. t+1 That is, to obtain the updated global environment state s t+1 Since the real environment has deterministic or noisy state transition functions, after an action is performed, the environment returns to a new global state, including changes caused by combat, movement, destruction, etc.

[0031] Update the global environment state s t+1The input expert strategy generates new expert joint actions, which are then subject to action change determination and frequency control determination. Action change determination refers to determining whether a new expert joint action is affected by a change in frequency. Compared to the previous action Sampling is only considered when there are differences; frequency control judgment means that even if the action changes, the current simulation step number must meet the preset sampling frequency, such as sampling once every 5 steps, so as to construct a new global environment state-expert joint action expert data pair. As the simulation task progresses, the above process is repeated until the preset number of rounds is completed, and finally a raw expert database containing rich tactical scenarios is formed.

[0032] It is worth noting that the differences in the expert data collection process mainly include two aspects: one is the difference in the simulation situation data given by the simulation environment, and the other is the difference in the actions given by the expert strategy. Therefore, considering these two differences, the system ensures that when the expert strategy gives different actions, the data can be saved into the expert database by controlling the sampling frequency. When the simulation environment situation changes after running a certain number of steps, the new data can be added to the expert database, and finally, the expert database can be persisted.

[0033] Step S20: Perform distributed preprocessing on the original expert database to obtain the preprocessed expert dataset.

[0034] It should be noted that distributed preprocessing refers to the process of decoupling and converting the state-action pairs recorded in the original expert database from a global perspective into local observation-individual action pairs for each UAV agent. Although this process is executed centrally during the training phase, its output structure is naturally adapted to the decentralized execution architecture. Therefore, the expert dataset after "distributed" preprocessing is a structured data set with local observation-individual expert action pairs as the basic unit.

[0035] It is understandable that the expert database collected through expert strategies is a direct observation of the environment and the actions performed by each drone. If training is carried out directly through this database, the features of each agent during training will be completely consistent, which does not conform to the observation input of the agent in reinforcement learning, and will lose the unique features of each agent, causing the trained agent policies to tend to be consistent. Therefore, preprocessing is performed on the basis of the original collected expert database to generate unique features of each drone agent in the drone swarm.

[0036] In one feasible implementation, step S20 may include: extracting global environment state-expert joint action expert data pair samples from the original expert database, wherein the global environment state-expert joint action expert data pair samples consist of a global environment state and corresponding expert joint actions, the global environment state including the position, attitude, and survival status of all friendly UAVs and enemy air defense towers; calculating the relative position and relative angle between the target UAV and all other entities in the UAV cluster based on the global environment state; constructing an initial local observation vector for the target UAV based on the relative position, relative angle, and the status information of all other entities; obtaining the identification information of the target UAV and performing one-hot encoding based on the identification information to obtain a one-hot encoded vector; concatenating the initial local observation vector and the one-hot encoded vector to obtain the target local observation vector of the target UAV; extracting the individual expert actions corresponding to the target UAV from the expert joint actions; constructing local observation vector-individual expert action expert data pairs based on the target local observation vector and the individual expert actions; repeating the above preprocessing process to process each global environment state-expert joint action expert data pair sample in the original expert database to obtain a preprocessed expert dataset.

[0037] It should be noted that the expert data collected through expert strategies is composed of the following formula: in, This represents the global simulation situation information provided by the simulation environment, and 'a' represents the joint actions of the entire UAV swarm.

[0038] As can be seen, each global environment state-expert joint action expert data pair sample in the original expert database consists of a global environment state and a corresponding expert joint action. The global environment state is a structured vector that fully describes the physical and logical state of all entities on the battlefield, specifically including the position, attitude, and survival status of all friendly drones and enemy air defense towers.

[0039] In the training process of multi-agent reinforcement learning, each agent has its own independent observations, and each agent represents an independent individual. Therefore, the aforementioned expert database is split into multiple data entries, each corresponding to the observations and actions of each agent. Specifically, the splitting method simplifies the original global environment state arrangement into the field-of-view information relative to other equipment in the environment from each agent's perspective. Taking the i-th UAV as an example, its observations include the relative position, yaw and pitch angles, and state information of other UAVs and the defending equipment relative to the i-th UAV's state. The formulas for calculating position, yaw, and pitch angles are: in, Represents the location coordinates of other friendly drones or anti-aircraft guns. Represents the position coordinates of the i-th drone. This represents the position of other friendly equipment or anti-aircraft guns relative to the i-th drone. This represents the difference in yaw and pitch angles between other friendly equipment or anti-aircraft guns and the i-th UAV. and Let be the orientation vector of the i-th UAV and the position pointing vectors of other equipment, respectively.

[0040] The above calculations can transform the global environment state into the relative position and angle information of the i-th drone. Simultaneously, combined with the survival status information of other drones and defensive equipment, this forms the initial local observation vector for the i-th drone. Based on this, adding a one-hot encoding of the i-th drone's number to the initial local observation vector allows the agent to better understand its surrounding environment. The one-hot encoding method is as follows: in, Let be a uniquely hot vector of length n for the i-th drone, where only the i-th position is 1 and the rest are 0.

[0041] This encoding method clearly identifies the unique identity of each drone, ensuring that each agent can make differentiated decisions based on its unique observation information during subsequent training. The introduction of one-hot encoded vectors not only enhances the expressive power of local observation vectors but also provides the reinforcement learning model with explicit information about the drone's identity, helping the model learn more targeted strategies.

[0042] The final preprocessed expert dataset is as follows: Where n represents the number of drones. This represents the independent observation information of each UAV obtained from the global state of the simulation environment. 'result' represents the individual expert action of the i-th drone, and 'result' represents the final preprocessed result, which is a data pair of local observations and actions of each drone within the drone swarm.

[0043] This distributed preprocessing approach also helps improve training efficiency. Since each agent has its own independent observation and action space, training can be performed in parallel, thus fully utilizing computing resources and shortening training time. Simultaneously, the preprocessed expert dataset provides a high-quality data foundation for subsequent reinforcement learning training. This data not only contains rich tactical scenarios and action information but also enhances its expressive power through methods such as one-hot encoding, enabling the model to better learn the cooperative strategies of the drone swarm.

[0044] like Figure 2 As shown, Figure 2 This is a framework diagram for expert data preprocessing. Sample data is obtained by sampling from the expert database. The sample data consists of environmental states and corresponding expert actions, such as the state of UAV 1 and its corresponding actions. The states of each UAV in the environmental states are processed to obtain the preprocessed observations of each UAV, including its relative position with all other entities, such as the relative position of the first UAV to the second UAV, the relative position of the first UAV to the first air defense tower, etc. The preprocessed observations of each UAV are paired with their corresponding expert actions. For example, the observation of the first UAV and its corresponding expert action form one data pair, the observation of the second UAV and its corresponding expert action form another data pair, and so on, until the observations of all UAVs and their corresponding expert actions are paired. These data pairs together constitute the preprocessed expert dataset.

[0045] Step S30: Based on the preprocessed expert dataset, clone and pretrain multiple initialized policy networks to obtain a set of pretrained policy networks, where each policy network corresponds to a UAV agent.

[0046] It should be noted that the multiple policy networks initialized are created independently for each drone agent in the drone swarm before training begins, with the same structure but randomly initialized parameters. This is denoted as π. ϕ1 ,π ϕ2 ,...,π ϕN , where N is the cluster size and ϕi is the learnable parameter of the i-th network.

[0047] Understandably, pre-training cloning, or Behavior Cloning (BC), is a supervised learning method that uses a preprocessed expert dataset as input-label pairs to train a policy network to mimic expert actions.

[0048] The pre-trained policy network set refers to the collection of N policy networks obtained after completing behavior cloning training, each of which has the preliminary ability to perform collaborative tasks.

[0049] It is worth noting that this implementation proposes a two-stage algorithm framework of "behavioral cloning pre-training + expert-guided reinforcement learning training". The first stage utilizes a dataset generated by expert policies for behavioral cloning pre-training, providing high-quality initial policies for the UAV agent. The second stage introduces expert policy cross-entropy constraints in multi-agent proximal policy optimization and employs an annealing mechanism to gradually weaken the expert influence. This structure leverages expert knowledge to improve exploration efficiency and convergence stability in the early stages of training, while releasing the agent's autonomous exploration capabilities in the later stages, achieving a smooth transition from expert-led to reinforcement learning-led learning, thus balancing training stability and policy optimality.

[0050] In its implementation, the training of UAV swarm agents based on behavior cloning involves randomly sampling from a shuffled expert dataset for collaborative task planning among UAV swarms. Environmental observations of each agent are used as features, and the agent's actions are used as labels for training. Finally, the difference between the agent's actions and the expert's actions is used as a loss function value for backpropagation training. Through multiple iterations, the agents can fully acquire the knowledge from the expert dataset, thereby distilling the expert strategy for collaborative task planning among UAV swarms into the agent model of the UAV swarm.

[0051] In one feasible implementation, step S30 may include: dividing the preprocessed expert dataset into a training subset and a validation subset; randomly selecting a number of local observation vector-individual expert action data pair samples from the training subset, and determining the corresponding policy network according to the UAV number; inputting the local observation vector in each local observation vector-individual expert action data pair sample into the corresponding policy network to obtain the predicted action probability distribution; calculating the weighted cross-entropy loss based on the individual expert actions in the local observation vector-individual expert action data pair, the predicted action probability distribution, and the death mask matrix generated by the UAV survival status; performing backpropagation updates on each policy network based on the weighted cross-entropy loss, and adjusting the current learning rate using an exponential decay method; after each preset training round, using the validation subset to evaluate the validation loss of each policy network, and outputting the pre-trained policy network with satisfactory performance; repeating the above training and validation process until the preset number of behavior cloning training rounds is completed to obtain the pre-trained policy network set.

[0052] It should be noted that the expert dataset obtained after distributed preprocessing is randomly divided into a training subset and a validation subset in a ratio such as 8:2. The training subset is used for parameter updates, and the validation subset is used to monitor generalization performance and prevent overfitting.

[0053] Understandably, each training iteration samples a batch from the training subset, for example, 32 samples, each sample being in the form of ( , In this context, the superscript i indicates that the sample belongs to the i-th drone. Based on the number i, the sample is routed to the corresponding policy network π. ϕi After forward computation, the policy network outputs a probability distribution over a discrete action space.

[0054] In the context of drone swarm collaborative task planning, agent death scenarios are involved. Including this data in the training process would interfere with the agent's training, leading to poor training results. Therefore, a loss function calculation method based on the mask matrix of drone deaths is introduced during training. This method eliminates the influence of using expert data as training data when drones die, thereby improving training effectiveness. Therefore, the formula for calculating the weighted cross-entropy loss is: in, Let represent the independent observations of the i-th agent, and n represent the number of agents in the UAV swarm. The actions of the drone swarm agent representing the i-th batch of data. This represents the actual action of the j-th agent in the i-th batch of data. This represents the predicted action of the j-th agent under the i-th batch of data. This represents the mask value of the j-th agent in the i-th batch of data, where 1 represents death and 0 represents survival. The number of samples included in each batch.

[0055] Meanwhile, overfitting may occur during the training of agent neural networks, i.e., overfitting the training dataset and lacking generalization ability. To address this, a normalization layer is added to the design of agent neural networks to prevent overfitting.

[0056] In the training process of neural networks, the choice of learning rate has a significant impact on the training process. A small learning rate will lead to slow convergence, while a large learning rate may cause instability in the training. Therefore, an exponential decay method is used to adjust the learning rate during the training of UAV swarm agents. The learning rate can be gradually reduced as the training process progresses. The specific formula for adjustment is as follows: in, This represents the learning rate at the epoch-level training round, where gamma is the decay coefficient. This represents the learning rate at epoch+1.

[0057] This formula allows for a gradual reduction of the learning rate during training, enabling the model to converge quickly in the early stages and fine-tune steadily in the later stages, avoiding oscillations caused by an excessively large learning rate or slow convergence caused by an excessively small learning rate. Furthermore, to further improve training stability, this method introduces an early stopping mechanism in the behavioral cloning phase. This mechanism monitors the loss changes of the validation subset; when the validation loss no longer decreases significantly or even begins to increase after several consecutive training rounds, it indicates that the model may be overfitting the training data, and the training process is immediately terminated. This mechanism effectively prevents the model from losing its generalization ability due to overfitting in the later stages of training, while also saving unnecessary computational resources.

[0058] It is worth noting that a death mask matrix is ​​introduced to shield failed samples, and an exponential decay learning rate and normalization layer structure are employed to prevent training fluctuations and overfitting. The masking mechanism ensures the effectiveness of the supervision signal; the learning rate decay improves the later convergence stability; and the normalization layer balances the feature distribution, improving the model's fitting and generalization ability to expert policies, thereby obtaining a more robust initial agent model.

[0059] In practice, a patience value can be set. If the validation loss does not decrease in consecutive patience rounds of training, an early stopping condition is triggered. Furthermore, to avoid false stopping due to data fluctuations, a sliding window averaging method can be introduced to smooth the validation loss before making a judgment. Through the synergistic effect of the early stopping mechanism and the exponentially decaying learning rate, the generalization performance and stability of the model are significantly improved while ensuring training efficiency.

[0060] like Figure 3 As shown, Figure 3 The training framework diagram for a drone swarm agent based on behavior cloning is shown. Two sets of key information are extracted from the expert database: the observation status of the drone swarm. Expert actions with drone swarms The drone swarm intelligence agent consists of n policy networks, from Actor1 to Actor2. n The structure consists of each Actor receiving the corresponding observation state (obs). i and output a predicted action a. i The action a output by each Actor i It will interact with expert actions from the expert database. By comparing the two, the action difference between them is calculated. Based on this action difference, the parameters of the corresponding Actor are updated so that the action output by the model gradually approaches the action of the expert.

[0061] Step S40: Based on the pre-trained policy network set, interact with the simulation environment to collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool.

[0062] It should be noted that the local observations of each UAV are input into its corresponding pre-trained policy network, sampling actions are executed in the simulation environment, and environmental feedback is obtained, including new states, rewards, and survival states. This reinforcement learning experience data obtained from these interactions is stored in the experience replay pool. The experience replay pool acts as a data buffer, capable of storing a large amount of historical experience data. The experience replay mechanism breaks down data correlations, improving training stability. The replay pool supports batch sampling for joint training of the policy and value networks.

[0063] The updated experience replay pool is created after one or more rounds of simulation interaction. The newly collected reinforcement learning experience data is added to the experience replay pool, and its content is updated so that it can be used for subsequent training iterations.

[0064] In one feasible implementation, step S40 may include: acquiring the current global environment state of the simulation environment, and generating the current local observation vector of each UAV in real time based on the current global environment state; inputting the local observations of each UAV into the pre-trained policy network corresponding to the pre-trained policy network set to obtain the agent actions and corresponding action selection probabilities of each UAV; synchronously invoking the expert policy, inputting the current global environment state into the expert policy, and generating corresponding expert reference joint actions; converting the agent actions into physical control commands and inputting them into the simulation environment for execution to obtain the reward value of the environment feedback and the survival status of each UAV, wherein the reward value is calculated by a multi-level reward function, the multi-level reward function including at least: task completion reward, task failure penalty, target hit reward, time step penalty, proximity to target reward, and local sacrifice contribution reward; encapsulating the current global environment state, the current local observation vector of each UAV, agent actions, action selection probabilities, expert reference joint actions, reward values, and survival status into reinforcement learning experience data; and storing the reinforcement learning experience data into an experience replay pool to obtain an updated experience replay pool.

[0065] It should be noted that the UAV swarm reinforcement learning algorithm used in this implementation is a multi-agent proximal policy optimization algorithm with a CTDE architecture, which is used to train the multi-agent UAV swarm. Based on this algorithm, a cross-entropy loss function between the expert policy and the agent policy is introduced to accelerate the convergence speed and improve the performance of the multi-agent reinforcement learning algorithm. The specific design of the reinforcement learning algorithm includes state space design, action space design, and reward function design.

[0066] The state space design is consistent with the unique observations of each UAV in the preprocessed expert data. It is composed of UAV information, enemy air defense tower information, and finally, the one-hot encoded information of the UAV ID is added to form the final UAV observation space. In reinforcement learning, each interaction starts with the global state to generate local observations and expert reference actions. Therefore, based on the current global environment state of the simulation environment, the relative position and relative angle of the target UAV and all other entities in the UAV swarm are calculated. Then, combined with the state information of other entities and the one-hot encoded information of the UAV ID, the current local observation vector of each UAV is generated in real time. This design ensures that the agent can acquire observation information similar to that of the expert data in the simulation environment, thus more effectively mimicking expert behavior.

[0067] The motion space design is consistent with the motion information ultimately received from the simulation environment. The drone's motion is controlled by adjusting the changes in its yaw and pitch angles. The specific drone motion formula is as follows: Among them, (x N ,y N ,z N ) represents the three-dimensional coordinates of the UAV at time N, v N Ω represents the velocity of the drone at time N, Δt represents the time step, and Ω represents the velocity of the drone at time N. t This represents the rotation matrix within time step t. Represents the unit vector of the direction of the UAV at time N, (x N+1 ,y N+1 ,z N+1 ) represents the three-dimensional coordinates of the UAV at time N+1.

[0068] The reward function is the ultimate optimization goal of the reinforcement learning algorithm. Therefore, the design of the reward function in reinforcement learning algorithms is crucial, as it directly affects the final training effect. The reward function designed in this section mainly consists of two parts: a target reward, used to encourage the drone swarm to complete the final task objective, and a guiding reward, used to guide the actions of individual agents in the drone swarm under most conditions. The final designed reward function is as follows: 1. Mission Completion Reward: The mission completion reward is used to encourage the drone swarm to complete the final mission objective. Specifically, when our drone swarm destroys all enemy air defense facilities within the specified time, the agent receives a substantial reward. The formula is: in, This is a positive constant representing the reward for completing the mission. The condition for mission completion is that the drone swarm destroys all enemy anti-aircraft towers. No reward is given if the mission is not completed.

[0069] 2. Mission Failure Penalty: The mission failure penalty is used to punish drone swarms for failing to complete their mission objectives. This includes situations where our drone swarm is destroyed by enemy anti-aircraft towers or exceeds the mission time limit. The formula is: in, This represents the reward constant given when the task is completed, and the negative value of this constant is given when the task fails.

[0070] 3. Target Strike Reward: A reward function awarded after a drone launches a missile and destroys an enemy anti-aircraft tower. The formula is: in, This represents a positive constant (reward) for successfully striking a target, thus incentivizing drones to engage enemy targets.

[0071] 4. Time Step Penalty: To guide the drone swarm to complete the final mission objective more quickly, a time step penalty based on the current simulation step size is introduced. The formula is: in, This is a time step penalty constant (negative), where t represents the current simulation step size. As the simulation runtime increases, the time step penalty for each step also increases, with a maximum value of [value missing]. Multiply by the maximum simulation step size.

[0072] 5. Proximity Reward: To guide the drone swarm towards the enemy target, a proximity reward function is introduced. First, the closest enemy target to the drone at the previous simulation step size is calculated, as follows: Where m represents the number of task objectives. Represents the location coordinates of the drone. This represents the position coordinates of the i-th enemy target. This represents the enemy target closest to the drone.

[0073] Based on the enemy target, the distance change between the simulated step lengths of the two steps is calculated. This change is then used to calculate the final reward function for approaching the target. The calculation formula is: in, The positive constant represents the reward function constant for approaching the target, and i represents the simulation step size at step i. and These represent the coordinates of the UAV at the (i-1)th simulation step and the i-th simulation step, respectively. and These represent the coordinates of the task target at the (i-1)th step and the ith step, respectively. The reward for getting closer to the target is calculated by subtracting the current distance from the distance to the target in the previous step.

[0074] 6. Local Sacrifice Contribution Reward: In collaborative operations of drone swarms, agents often need to sacrifice themselves to ensure the entire swarm achieves its mission objective. Therefore, a local sacrifice contribution reward is designed to calculate the contribution value given to the entire drone swarm when a drone is destroyed. It first calculates whether other drones are within the range of an anti-aircraft tower when the drone is shot down. The formula for this determination is: in, Represents the position coordinates of the i-th drone. Indicates the location coordinates of the target air defense tower. This indicates the maximum attack range of the anti-aircraft tower; the distance between the current drone position and the anti-aircraft tower position is less than... "At" indicates that the anti-aircraft tower is within its strike range.

[0075] By summing the judgment results of all drones, a local sacrifice contribution reward function is obtained. As shown in the following formula: in, This represents a positive constant for the contribution reward of local sacrifice. When a drone is destroyed, if there are no other drones within the range of the anti-aircraft tower, a negative reward is given. If there are other drones, a reward is given based on the relative distance between the drone and the target, in order to assess the contribution of the destroyed drone to the team.

[0076] The final multi-level reward function for the entire drone swarm collaboration is: : Here, σ1, σ2, and σ3 represent the weight coefficients of the time step penalty, the reward for getting closer to the target, and the reward for local sacrifice contribution, respectively. These coefficients can be adjusted according to the actual task requirements to balance the impact of different reward items on the overall training effect.

[0077] This multi-layered reward function design guides the drone swarm to complete mission objectives more efficiently during reinforcement learning, while also enhancing the swarm's collaborative combat capabilities. During training, each agent continuously adjusts its strategy based on feedback from the reward function to maximize cumulative rewards, thereby gradually optimizing the overall performance of the drone swarm.

[0078] Multi-agent proximal policy optimization algorithms are primarily trained by collecting data from an experience replay pool. The data stored in the experience replay pool can be represented as follows:<obs,s,v,a,a_logprob,r,d,a'> Where obs represents the local observation data of each drone, s represents the global state of the entire battlefield, v represents the global value function under the current battlefield state, a represents the action of the drone swarm, a_logprob selects the probability of the drone swarm choosing action a, r represents the reward function value, d represents the survival state of each drone in the environment, and a' represents the joint action of experts under the current state.

[0079] The experience replay pool normalizes the reward function value during data collection, which can improve the convergence efficiency of the algorithm. The normalization process is as follows: When the first set of data is collected in the experience replay pool, the mean and standard deviation of the reward function values ​​are initialized. The initialization formula is: Where R represents the first reward value collected in the experience replay pool. This represents the mean of the reward function. This represents the standard deviation of the reward function.

[0080] Subsequently, as new data is collected, the mean and standard deviation of the reward values ​​are updated using the following formula: Where i represents the i-th reward value collected in the experience replay pool. and Let R represent the mean of the reward function of the experience replay pool when collecting data at step i and step i-1, and let R represent the reward value collected at step i. and These are intermediate variables required for the calculation process.

[0081] Finally, based on the standard deviation and variance of the reward value, the normalized value of the reward value is calculated as follows: Where R represents the reward value collected in the experience replay pool. This represents the mean of the reward function. The standard deviation of the reward function, This represents the normalized reward value.

[0082] Step S50: Based on the updated experience replay pool, jointly train the pre-trained policy network set and the initialized value network to obtain the UAV swarm collaborative task planning strategy.

[0083] It should be noted that the policy network set and value network are updated based on data sampled from the experience replay pool. The update of the policy network set mainly follows the principles of the multi-agent proximal policy optimization algorithm. The parameters of the policy network are adjusted by calculating the probability ratio between the current policy and the old policy, combined with the normalized reward value and the advantage function estimate. The advantage function estimate is obtained by calculating the difference between the value function in the current state and the target value function, aiming to measure the advantage of taking the current action compared to the average action. When updating the policy network set, the probability ratio between the new and old policies is first calculated, reflecting the direction and magnitude of the policy update. Subsequently, the normalized reward value, reflecting the contribution of the current action to the overall task, and the advantage function estimate are combined to form the objective function for policy update. By minimizing this objective function, the policy network set can gradually learn better task planning strategies, enabling the UAV swarm to make more rational decisions in collaborative operations.

[0084] The updating of the value network primarily relies on the Bellman equation, achieved by minimizing the mean squared error between the predicted value function and the target value function. The predicted value function is derived from the current value network output based on the current state, while the target value function is calculated by combining the predicted value function for the next state and the normalized reward value. When updating the value network, the current state is first input to obtain the predicted value function. Then, based on the next state and the normalized reward value, the target value function is calculated using the Bellman equation. The parameters of the value network are adjusted by comparing the difference between the predicted and target value functions, i.e., the mean squared error. This process aims to enable the value network to more accurately evaluate the value under different states, thereby providing more reliable guidance for the policy network.

[0085] During joint training, the policy network set and the value network collaborate to optimize the collaborative task planning capabilities of the UAV swarm. The policy network set is responsible for generating the action policies of the UAV swarm, while the value network evaluates the potential long-term benefits of these policies. Through continuous interaction and feedback, they form a co-evolutionary mechanism.

[0086] In its implementation, during each training iteration, the policy network generates a set of actions based on the current state and executes these actions. After execution, the drone swarm enters a new state and receives a corresponding reward. These states and reward values ​​are stored in an experience replay pool for subsequent training. Simultaneously, the value network calculates the predicted value function and the target value function based on the current and next states, respectively. The policy network updates its parameters using the normalized reward value and the estimated advantage function, combined with the probability ratio of the old and new policies, to generate better action policies. The value network continuously adjusts its parameters by minimizing the mean squared error between the predicted and target value functions, improving the accuracy of state value assessment. As training iterations continue, the performance of both the policy network and the value network gradually improves. The action policies generated by the policy network become increasingly rational, guiding the drone swarm to efficiently coordinate operations in complex environments and better achieve various mission objectives. The value network's assessment of state values ​​also becomes increasingly accurate, providing the policy network with reliable decision-making support, enabling the drone swarm to make more informed choices in different situations.

[0087] It is worth noting that the knowledge-enhanced UAV swarm cooperative task planning algorithm in this embodiment includes two stages. Both stages fully utilize expert knowledge from the expert policies in the UAV swarm cooperative task planning to enhance the algorithm and address the instability problem during reinforcement learning training. First, the algorithm uses a large language model to generate UAV swarm cooperative task planning policies for environmental exploration, generating expert data and constructing an expert dataset. Then, it initializes an agent policy for each UAV within the swarm and pre-trains the UAV swarm agents using behavior cloning based on the expert dataset. Next, it proposes an online reinforcement learning training method incorporating expert policies. During the training stage of the UAV swarm using a multi-agent reinforcement learning algorithm, the calculation method of the loss function for maximizing single rewards is optimized. A cross-entropy loss function between the UAV swarm agent policies and the expert policies of the UAV swarm cooperative task planning is introduced to better guide the training process of the UAV swarm agents. Simultaneously, annealing is used to gradually transition the UAV swarm agents into a reinforcement learning agent training process. Figure 4 As shown, Figure 4 The diagram shows a knowledge-enhanced UAV swarm collaborative task planning algorithm. A set of "environmental observation" and "expert action" pairs is randomly sampled from an expert database and input into the corresponding UAV swarm agents (Actor network) for policy distillation. The UAV swarm agents include Actor1-Actor2. nUsing the distance between expert actions and agent actions as a loss function, parameters are updated by minimizing this gap to achieve behavior cloning, i.e., imitating expert behavior. This stage completes distributed pre-training, enabling each UAV to possess basic collaborative capabilities and avoiding the inefficiency and high risk of starting from scratch. The simulation environment and the UAV swarm collaborative task planning expert strategy interact bidirectionally to obtain expert action suggestions. An experience replay pool stores interaction experience data from the simulation environment. Batch data from the experience replay pool is then input into the human-machine swarm intelligent agents (Actor1-Actor). n ) and the Critic1-Critic network n In the context of drone swarm intelligence agents (Actor1-Actor) n The agent receives the current environmental state, generates agent actions, and updates these actions based on two factors: the expert action gap and the advantage function value. These two factors are added together to form a composite loss function, driving policy optimization. (Critic1-Critic network) n It receives the current state and outputs a state value estimate. It compares the value estimate with the TD error (time difference error) to calculate the MSE loss, thereby updating the critic network parameters and improving the accuracy of the value estimate.

[0088] This embodiment provides a distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning. By combining expert policies and simulation environment for interactive data collection, and using behavior cloning to pre-train the policy network, and then optimizing the policy network and value network through joint training, the method significantly improves training efficiency, policy stability and multi-UAV collaborative capability, and achieves efficient and stable UAV swarm collaborative task planning.

[0089] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S40 includes steps S501 to S505: Step S501: Randomly select a number of reinforcement learning experience data samples from the updated experience replay pool, and determine the global environment state, local observation vectors of each UAV, agent actions, historical action selection probability, expert reference joint actions, reward value and survival status based on the reinforcement learning experience data samples.

[0090] It should be noted that a batch random sampling method is used to extract B experience tuples from the experience replay pool, which are reinforcement learning experience data samples. From each experience sample, structured fields such as global environment state, local observation vectors of each UAV, agent actions, historical action selection probabilities, expert reference joint actions, reward value, and survival status are extracted.

[0091] Step S502: Input the global environment state into the initialized value network for forward propagation, and output the corresponding state value estimate.

[0092] It should be noted that this implementation adopts a centralized value network, that is, a shared network processes the global state, inputs the global environment state into the initialized value network, and outputs a scalar state value estimate.

[0093] Step S503: Construct a value loss function and a policy loss function based on the state value estimation, the local observation vectors of each UAV, agent actions, historical action selection probabilities, expert reference joint actions, reward values, and survival states.

[0094] It's important to note that the value loss function is constructed to measure the accuracy of the value network's predictions. It typically takes the form of mean squared error, comparing the predicted state value estimate with the target value estimate calculated using the Bellman equation. Specifically, the target value estimate combines the value estimate of the next state with the normalized reward value, reflecting the expected long-term gains from taking a specific action in the current state. By minimizing the value loss function, the value network can continuously adjust its parameters to improve its ability to assess state value.

[0095] The construction of the policy loss function focuses on guiding the policy network set to generate better action policies. It combines the probability ratio of the old and new policies, the normalized reward value, and the advantage function estimate. The probability ratio of the old and new policies reflects the direction and magnitude of policy updates, the normalized reward value reflects the contribution of the current action to the overall task, and the advantage function estimate measures the degree of advantage of taking the current action compared to the average action. By minimizing the policy loss function, the policy network set can gradually learn to make more reasonable decisions in complex environments, thereby improving the collaborative combat effectiveness of the UAV swarm.

[0096] It is understandable that behavioral cloning can be used to train an agent model with basic intelligence, i.e., a pre-trained policy network set, with the help of expert datasets. However, since the expert dataset for collaborative mission planning of UAV swarms cannot cover all the situations that UAV swarms encounter in actual combat, the dependence on expert policies is gradually reduced during the training process by introducing the cross-entropy loss function of expert policies and agent policies and by annealing.

[0097] It is worth noting that this implementation proposes an online reinforcement learning training method incorporating expert strategies. During the training phase of the UAV swarm using a multi-agent reinforcement learning algorithm, the calculation method of the loss function for maximizing single rewards is optimized. A cross-entropy loss function between the UAV swarm agent strategies and the expert strategies for collaborative task planning is introduced, thereby better guiding the training process of the UAV swarm agents. Simultaneously, annealing is used to gradually transition the training process to a reinforcement learning agent training process. Specifically, in the initial training phase, due to the limited understanding of task planning by the agent model, expert strategies dominate. The cross-entropy loss function effectively transfers the experience of expert strategies to the agent, enabling it to quickly learn basic task planning capabilities. As training progresses, the annealing mechanism gradually takes effect, reducing the weight of expert strategies in the loss function. This allows the agent more room for autonomous exploration and learning, relying more on the feedback mechanism of reinforcement learning to optimize its strategies. Ultimately, this achieves a smooth transition from relying on expert strategies to relying on its own reinforcement learning capabilities, improving the adaptability and accuracy of task planning for UAV swarms in different real-world scenarios. This online reinforcement learning training method, incorporating expert strategies, not only fully utilizes the rich experience of experts in task planning, avoiding the waste of time and resources due to blind exploration in the early stages of training, but also stimulates the agent's autonomous learning capabilities in the later stages of training, enabling it to flexibly adjust its strategies according to changes in the actual environment, further improving the quality and efficiency of task planning. Furthermore, this method has strong versatility and scalability, and can be applied to collaborative task planning scenarios for UAV swarms of different sizes and types, providing an effective technical means to solve the problem of collaborative task planning for UAV swarms in complex environments.

[0098] In one feasible implementation, step S503 may include: calculating the cumulative expected return based on the reward value, and determining the advantage function value based on the cumulative expected return and the state value estimate; constructing a value loss function based on the cumulative expected return, the state value estimate, and the survival state; inputting the local observation vectors of each UAV into the policy network corresponding to the pre-trained policy network set to obtain the current action selection probability; constructing a policy gradient loss term based on the historical action selection probability, the current action selection probability, and the advantage function value; calculating the cross-entropy loss between the agent's action and the corresponding individual action in the expert reference joint action, and constructing an expert constraint loss term based on the cross-entropy loss and the current annealing weight; and constructing a policy loss function based on the policy gradient loss term, the expert constraint loss term, and the policy entropy regularization term.

[0099] It should be noted that in the multi-agent proximal policy optimization algorithm, the optimization objective of the policy network is to maximize the advantage function. In this embodiment, the optimization objective of the policy network, based on the multi-agent proximal policy optimization algorithm, introduces the cross-entropy between the expert policy and the agent policy. By minimizing this value, the agent can be made to move closer to the expert policy during training.

[0100] The advantage function value is calculated using the generalized advantage estimation method. The formula for generalized advantage estimation is as follows: in, and To calculate the hyperparameters of GAE, It is a discount factor between 0 and 1, used to assess the importance of future rewards. Used to weigh the bias and variance of GAE and V represents the global state of the environment at time t+1 and time t, and V represents the state value function. Based on the global state of the environment, it returns the cumulative expected reward of the environment in the future. This represents the reward value that the environment gives to the agent at time t. This represents the advantage estimation error at time t. The value represents the advantage estimation error at time t+n. This represents the estimate of generalized advantage.

[0101] Next, the dominant function value is normalized, and the calculation formula is as follows: Where n represents the amount of data retrieved from the experience replay pool each time. This represents a vector consisting of the advantage function estimates in the experience replay pool. for The mean, express The standard deviation.

[0102] Finally, by introducing a policy entropy loss function based on maximizing the advantage function, the exploration of the agent can be reduced. The policy network in the multi-agent proximal policy optimization algorithm is optimized by maximizing both the advantage function and the agent's policy entropy. The optimization objective is: in, Indicates the current intelligent agent in Select in status The probability of an action. Let B represent the probability of the old strategy, B represent the size of the batch data, and n represent the number of drone swarm agents. This represents the dominant function value, and clip is a clipping function that clips... Limited to and middle, This is the cutting control factor. S represents the hyperparameter for controlling the magnitude of policy entropy. The strategy entropy, with the ultimate goal of maximizing it. .

[0103] Based on this optimization objective, the policy differences between expert policies and agent policies are incorporated into the final training optimization objective to better guide the training direction of the agent during the training process, as shown in the following equation: in, This refers to the action generated at a given observation based on an expert strategy. This represents the action output by the neural network of a reinforcement learning agent under a given observation, with the ultimate optimization objective being to minimize... .

[0104] Taking into account both optimization objectives, the final optimization objective of the policy network is as follows: in, To control the hyperparameters that influence the expert policy on the agent model's policy, the optimization objective is ultimately minimized. .

[0105] Expert-guided reinforcement learning agent models can significantly improve training efficiency in the early stages of reinforcement learning training. However, as reinforcement learning progresses, the guidance of expert policies severely limits the agent's exploration. Therefore, this implementation proposes an annealing-based hyperparameter adaptive adjustment method. This method weakens the influence of expert policies on the reinforcement learning agent's training as the number of training steps increases. The specific adaptive adjustment formula is as follows: Where s represents the number of training rounds in reinforcement learning, , These are adaptively adjusted hyperparameters; when the number of training steps is less than... Choose the smaller value To ensure training stability, the number of training steps is within [a certain range]. and During this period, the number of training steps gradually increases. The value of this value enhances exploration; when the training steps reach [a certain number], [the value of this value is adjusted]. The original algorithm is transformed into a multi-agent reinforcement learning algorithm.

[0106] In multi-agent proximal policy optimization algorithms, the optimization objective of the value network is to better fit the state value function. Its role is to guide the policy network update direction and help it train in a direction that maximizes the state value function.

[0107] The cumulative expected return is calculated as the sum of the discounted reward values ​​obtained from a certain state to the end of the final environment run, as shown in the following formula: Where i represents the current running time of the environment, and n represents the time until the environment finishes running. This is the reward function value obtained at step k.

[0108] Based on the discounted reward, it can be approximated as the target value of the state value function in a certain state. Using the distance between the target value and the actual output value as the final optimization objective allows the value network to better fit the state value function during training, thus better guiding the agent's training. Building on this, a state value function pruning mechanism is introduced to limit the update magnitude of the value function during training. The formula for calculating the pruned state value function is: in, Indicates in The new state value function under the given state. Indicates in The old state value function under the given state, is the clipping coefficient of the state value function.

[0109] The final formula for calculating the optimization objective of the value network is: Where B represents the size of the batch data, and n represents the number of drone swarm agents. Indicates the state Determine the new state value function. This represents the cumulative expected return obtained from state i until the end of the environment's operation.

[0110] The optimization objective is to guide the value network to better fit the true state value function by minimizing the gap between the predicted state value and the actual cumulative expected return, while considering the pruned state value function. This design not only improves the learning efficiency of the value network but also helps prevent overfitting during training, thereby enhancing the overall task planning performance. In the early stages of training, the value network gradually approximates the true state value function by continuously adjusting its parameters, providing the policy network with accurate update directions. As training progresses, the value network and policy network collaborate to optimize the collaborative task planning capabilities of the UAV swarm, enabling it to make rapid and accurate decisions in complex and ever-changing environments and efficiently complete various collaborative tasks.

[0111] Step S504: Update the gradient of the value network based on the value loss function, and update the gradient of the pre-trained policy network set based on the policy loss function.

[0112] It should be noted that during gradient updates, a mini-batch gradient descent approach is used, randomly sampling a certain size of data samples from the experience replay pool. For the value network, based on the calculated value loss function, the gradient of each layer's parameters is calculated using the backpropagation algorithm. Then, the parameters of the value network are updated along the gradient descent direction according to the set learning rate, allowing the value network to gradually fit the true state-value function. For the pre-trained policy network set, the gradient of each layer's parameters is obtained through backpropagation based on the policy loss function. The parameters of the policy network set are then updated according to the learning rate, prompting the agent to continuously optimize its action selection policy during training, improving the accuracy of task planning. The learning rate setting is crucial during gradient updates. An excessively large learning rate may lead to unstable network parameter updates or even failure to converge; while an excessively small learning rate will slow down the training process and result in excessively long convergence times. Therefore, a dynamic learning rate adjustment method is typically used. A larger learning rate is used in the early stages of training to accelerate convergence, and the learning rate is gradually reduced as training progresses, allowing for more precise adjustment of network parameters and ultimately achieving better training results. Furthermore, to further improve the stability and performance of training, optimization algorithms can be introduced, such as the stochastic gradient descent algorithm with momentum. This algorithm considers the direction of previous gradients during gradient updates, accelerating convergence and reducing oscillations. By continuously updating the gradients of the value network and the policy network, the two networks cooperate and promote each other, thereby continuously improving the collaborative task planning capability of the UAV swarm in complex environments and enabling it to complete various tasks more efficiently and accurately.

[0113] Step S505: Repeat the above training process until the preset training rounds are reached or the convergence condition is met, and use the trained policy network set as the UAV swarm collaborative task planning strategy.

[0114] It's important to note that during repeated training, each round enhances the collaborative task planning capabilities of the drone swarm. As the number of training rounds increases, the value network and policy network are continuously optimized, gradually improving the agent's decision-making ability in complex environments. The preset number of training rounds ensures the model has sufficient time to learn, while the satisfaction of the convergence condition indicates that the model has reached a relatively stable state, with limited potential improvement from further training. The training process ends when the preset number of training rounds is reached or the convergence condition is met, at which point the policy network set possesses a high level of collaborative task planning capability. Deploying this trained policy network set to a real-world drone swarm allows it to exhibit higher efficiency and accuracy when performing various collaborative tasks.

[0115] In this embodiment, by combining expert policy guidance, adaptive hyperparameter adjustment through annealing, and collaborative optimization of the value network and policy network, efficient collaborative task planning of UAV swarms in complex environments is achieved, improving training efficiency. The annealing mechanism avoids the limitation of expert policies on the agent's exploration capabilities, ensuring flexibility and adaptability in the later stages of training. At the same time, the gradient update mechanism of the value network and policy network enables the UAV swarm to make accurate decisions quickly when facing complex and ever-changing task environments, effectively improving the efficiency and success rate of collaborative task planning.

[0116] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning. Any simple modifications based on this technical concept are within the protection scope of this application.

[0117] This application also provides a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning. The distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning includes: The data acquisition module 10 is used to interact with the simulation environment based on expert strategies and to perform intermittent data acquisition to build an original expert database.

[0118] The preprocessing module 20 is used to perform distributed preprocessing on the original expert database to obtain a preprocessed expert dataset.

[0119] The pre-training module 30 is used to clone and pre-train multiple initialized policy networks based on the preprocessed expert dataset to obtain a set of pre-trained policy networks, wherein each policy network corresponds to a UAV agent.

[0120] The interaction module 40 is used to interact with the simulation environment based on the pre-trained policy network set, collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool.

[0121] The joint training module 50 is used to jointly train the pre-trained policy network set and the initialized value network based on the updated experience replay pool to obtain the UAV swarm collaborative task planning strategy.

[0122] The distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning provided in this application adopts the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in the above embodiments, which can solve the technical problems of low training efficiency and instability of reinforcement learning and limited performance of expert strategies in UAV swarm task planning in the prior art. Compared with the prior art, the beneficial effects of the distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning provided in this application are the same as the beneficial effects of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning provided in the above embodiments, and other technical features in the distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0123] This application provides a distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning. The distributed UAV swarm collaborative task planning device based on knowledge enhancement and reinforcement learning includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning in the above embodiment 1.

[0124] This application provides a non-transitory computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the distributed UAV swarm cooperative task planning method based on knowledge enhancement and reinforcement learning in the above embodiments.

[0125] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described above.

[0126] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning, characterized in that, The method includes: An original expert database is constructed by interacting with the expert strategy and simulation environment and collecting intermittent data. The original expert database is subjected to distributed preprocessing to obtain a preprocessed expert dataset; Based on the preprocessed expert dataset, multiple initialized policy networks are cloned and pre-trained to obtain a set of pre-trained policy networks, where each policy network corresponds to a UAV agent. Based on the pre-trained policy network set, the simulation environment is interacted to collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool. Based on the updated experience replay pool, the pre-trained policy network set and the initialized value network are jointly trained to obtain the UAV swarm collaborative task planning strategy.

2. The method as described in claim 1, characterized in that, The process of interacting with the simulation environment based on expert strategies and conducting intermittent data collection to construct an original expert database includes: The initial global environment state of the simulation environment is obtained, and the initial global state is input into the expert policy built based on the large language model to obtain the initial expert joint action generated by the expert policy, wherein the expert joint action includes the action that each drone in the drone cluster should perform; Construct a first global environment state-expert joint action expert data pair based on the initial global environment state and the initial expert joint action; The initial joint expert action is executed in the simulation environment to obtain the updated global environment state; The updated global environment state is input into the expert policy again to obtain the next joint expert action generated by the expert policy. Determine whether the next joint expert action differs from the initial joint expert action; If it exists, determine whether the current sampling frequency has reached the preset sampling frequency; If achieved, a second global environment state-expert joint action expert data pair is constructed based on the updated global environment state and the next expert joint action; Repeatedly execute multiple rounds of simulation tasks until the preset number of expert data collection rounds is completed, resulting in an original expert database containing multiple global environment state-expert joint action expert data pairs.

3. The method as described in claim 1, characterized in that, The distributed preprocessing of the original expert database to obtain the preprocessed expert dataset includes: Extract global environment state-expert joint action expert data pairs from the original expert database. The global environment state-expert joint action expert data pairs consist of global environment state and corresponding expert joint actions. The global environment state includes the position, attitude and survival status of all friendly UAVs and enemy air defense towers. Calculate the relative position and relative angle between the target drone and all other entities in the drone swarm based on the global environmental state; Based on the relative position, relative angle, and state information of all other entities, an initial local observation vector for the target UAV is constructed. Obtain the identification number of the target UAV, and perform one-hot encoding based on the identification number to obtain a one-hot encoded vector; The initial local observation vector and the one-hot encoded vector are concatenated to obtain the target local observation vector of the target UAV. Extract the individual expert actions corresponding to the target UAV from the joint expert actions; Construct local observation vector-individual expert action expert data pairs based on the target local observation vector and individual expert actions; Repeat the above preprocessing process to process the sample of each global environment state-expert joint action expert data in the original expert database to obtain the preprocessed expert dataset.

4. The method as described in claim 1, characterized in that, The process involves cloning and pre-training multiple initialized policy networks based on the preprocessed expert dataset to obtain a set of pre-trained policy networks, including: The preprocessed expert dataset is divided into a training subset and a validation subset; A number of local observation vector-individual expert action expert data pairs are randomly selected from the training subset, and the corresponding policy network is determined according to the UAV number. Each local observation vector-individual expert action expert data pair is input into the corresponding policy network to obtain the predicted action probability distribution. The weighted cross-entropy loss is calculated based on the individual expert actions in the local observation vector-individual expert action expert data pair, the probability distribution of the predicted actions, and the death mask matrix generated by the UAV survival status. The weighted cross-entropy loss is used to backpropagate and update each policy network, and the current learning rate is adjusted using an exponential decay method. After each preset training round, the validation loss of each policy network is evaluated using the validation subset, and the pre-trained policy network with satisfactory performance is output. Repeat the above training and verification process until the preset number of behavior cloning training rounds are completed, and the pre-trained policy network set is obtained.

5. The method as described in claim 1, characterized in that, The process involves interacting with the pre-trained policy network set and the simulation environment to collect reinforcement learning experience data and store it in the experience replay pool, resulting in an updated experience replay pool, including: The current global environment state of the simulation environment is obtained, and the current local observation vector of each UAV is generated in real time based on the current global environment state; Based on the local observations of each UAV, the data is input into the pre-trained policy network corresponding to the pre-trained policy network set to obtain the agent actions of each UAV and the corresponding action selection probabilities. Synchronously invoke the expert strategy, input the current global environment state into the expert strategy, and generate the corresponding expert reference joint action; The actions of the intelligent agent are converted into physical control commands and input into the simulation environment for execution, so as to obtain the reward value of environmental feedback and the survival status of each UAV. The reward value is calculated by a multi-level reward function, which includes at least: task completion reward, task failure penalty, target hit reward, time step penalty, approaching the target reward and local sacrifice contribution reward. The current global environment state, the current local observation vectors of each UAV, agent actions, action selection probabilities, expert reference joint actions, reward values, and survival status are encapsulated into reinforcement learning experience data. The reinforcement learning experience data is stored in the experience replay pool to obtain the updated experience replay pool.

6. The method as described in claim 1, characterized in that, The method of jointly training the pre-trained policy network set and the initialized value network based on the updated experience replay pool to obtain the UAV swarm cooperative task planning strategy includes: A number of reinforcement learning experience data samples are randomly selected from the updated experience replay pool, and the global environment state, local observation vectors of each UAV, agent actions, historical action selection probability, expert reference joint actions, reward value and survival status are determined based on the reinforcement learning experience data samples. The global environment state is input into the initialized value network for forward propagation, and the corresponding state value estimate is output. Based on the state value estimation, the local observation vectors of each UAV, agent actions, historical action selection probabilities, expert reference joint actions, reward values, and survival states, a value loss function and a policy loss function are constructed. The value network is updated with gradients based on the value loss function, and the pre-trained policy network set is updated with gradients based on the policy loss function. Repeat the above training process until the preset number of training rounds is reached or the convergence condition is met. Use the trained policy network set as the planning strategy for UAV swarm collaborative tasks.

7. The method as described in claim 6, characterized in that, The construction of the value loss function and policy loss function based on the state value estimation, the local observation vectors of each UAV, agent actions, historical action selection probabilities, expert reference joint actions, reward values, and survival states includes: The cumulative expected return is calculated based on the reward value, and the advantage function value is determined based on the cumulative expected return and the state value estimate. A value loss function is constructed based on the cumulative expected return, the state value estimate, and the survival state. The local observation vectors of each UAV are input into the policy network corresponding to the pre-trained policy network set to obtain the current action selection probability; Based on the historical action selection probability, the current action selection probability, and the advantage function value, a policy gradient loss term is constructed. Calculate the cross-entropy loss between the agent's action and the corresponding individual action in the expert reference joint action, and construct an expert constraint loss term based on the cross-entropy loss and the current annealing weight; The policy loss function is constructed based on the policy gradient loss term, the expert constraint loss term, and the policy entropy regularization term.

8. A distributed unmanned aerial vehicle (UAV) swarm collaborative task planning device based on knowledge enhancement and reinforcement learning, characterized in that, The device includes: The data acquisition module is used to interact with the simulation environment based on expert strategies and to perform intermittent data acquisition to build the original expert database. The preprocessing module is used to perform distributed preprocessing on the original expert database to obtain a preprocessed expert dataset. The pre-training module is used to clone and pre-train multiple initialized policy networks based on the preprocessed expert dataset to obtain a set of pre-trained policy networks, wherein each policy network corresponds to a UAV agent. The interaction module is used to interact with the simulation environment based on the pre-trained policy network set, collect reinforcement learning experience data and store it in the experience replay pool to obtain the updated experience replay pool. The joint training module is used to jointly train the pre-trained policy network set and the initialized value network based on the updated experience replay pool to obtain the UAV swarm collaborative task planning strategy.

9. A distributed unmanned aerial vehicle (UAV) swarm collaborative task planning device based on knowledge enhancement and reinforcement learning, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the distributed unmanned aerial vehicle swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a distributed UAV swarm collaborative task planning program based on knowledge enhancement and reinforcement learning. When the distributed UAV swarm collaborative task planning program based on knowledge enhancement and reinforcement learning is executed by the processor, it implements the distributed UAV swarm collaborative task planning method based on knowledge enhancement and reinforcement learning as described in any one of claims 1 to 7.

Images