Unmanned aerial vehicle group cooperation method based on reinforcement learning in observation limited environment

Abstract

The invention discloses an unmanned aerial vehicle group cooperation method based on reinforcement learning in an observation limited environment, and the method builds a reinforcement learning environment network of each unmanned aerial vehicle, can help each unmanned aerial vehicle to make a more sufficient understanding of the environment, and improves the limitation on the decision of an unmanned aerial vehicle group caused by a too small observation range. Meanwhile, the unmanned aerial vehicle can obtain more diversified experience and strategies through the high observation range expert network and the low observation range expert network with different observation radiuses. The observation value of the strategy network is obtained by obtaining the global state under the current time step, so that the output of the strategy network can fully consider the behaviors of other unmanned aerial vehicles in the system and the dynamic change of the whole system. Therefore, the made decision can give full consideration to the global perspective and perspectiveness, and the cooperation efficiency of the unmanned aerial vehicle cluster and the final decision effect are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned swarm mission planning, and in particular to a UAV swarm collaboration method based on reinforcement learning in an observation-restricted environment. Background Art

[0002] At present, drones have made significant progress in many fields, including civilian aerial photography, search and rescue, monitoring, and military reconnaissance and strike missions. With the continuous advancement of drone technology in manufacturing, communication and control, their role in military and civilian operations has become increasingly important. However, due to the limitations of its information collection and single-machine performance, a single drone is difficult to cope with complex and changing tasks. Therefore, the use of multiple drones to work together can significantly improve the overall operational efficiency by sharing battlefield information and utilizing their different capabilities. This approach has made the collaborative operation of drone swarms a hot topic in current research. In the context of this swarm operation, drones need to autonomously analyze the operating site environment and make adaptive decisions in real time while minimizing human intervention.

[0003] Fully distributed multi-agent reinforcement learning is a training framework suitable for multi-UAV collaborative tasks. In the fully distributed multi-agent reinforcement learning training framework, each UAV can be regarded as an agent, and each UAV makes decisions based on the information it observes during the learning process without relying on communication between multiple UAVs. Therefore, this type of method has greater flexibility and adaptability. When the number of UAVs in the system decreases or new UAVs are added, this type of method can more easily adapt to such changes without redesigning the centralized training process, so the failure of one UAV will not directly affect other UAVs. The method under the fully distributed framework is less dependent on training resources. This decentralized decision-making structure can also reduce the complexity of communication and coordination, and improve the decision-making efficiency of multi-UAV systems.

[0004] In a fully distributed framework, the partial observability problem faced by the agent will have a significant impact on its decision-making effect, that is, each drone cannot observe the global state of the entire mission environment, but can only observe partial information. Especially in collaborative formation, search and rescue and other environmental conditions, due to the large scope of the overall mission, the observation range of each drone is greatly limited, which leads to huge challenges in understanding the entire environmental system. When the observation range of the drone is greatly reduced, the information that can be obtained becomes extremely limited, which seriously restricts its ability to make effective decisions. Since drones can only rely on limited local information to make decisions, they often cannot fully consider the behavior of other drones in the system and the dynamic changes of the entire system. This information limitation may cause the decisions made by a single drone to lack a global perspective and foresight, thereby affecting the collaborative efficiency of the entire group system and the final decision-making effect. Summary of the invention

[0005] The present invention discloses a UAV swarm collaboration method based on reinforcement learning in an observation-restricted environment to overcome the above-mentioned technical problems.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] A UAV swarm collaboration method based on reinforcement learning in an observation-constrained environment includes the following steps:

[0008] S1: Establish a drone cluster containing I drones, build a reinforcement learning environment network for the drone cluster and initialize it;

[0009] The reinforcement learning environment network includes a policy network of the ith UAV, a high observation range expert network of the ith UAV, a low observation range expert network of the ith UAV, a target network of the high observation range expert network of the ith UAV, and a target network of the low observation range expert network of the ith UAV; wherein i=1...I, i represents the index number of the UAV, and I represents the total number of UAVs;

[0010] Among them, the mission radius of the drone group>the observation radius of the expert network with high observation range of the i-th drone>the observation radius of the expert network with low observation range of the i-th drone;

[0011] S2: Get the global state s at the current time step t t ; and according to the policy network, obtain the observation value of the policy network on the global state and the action value of the drone at the current time step t To obtain the local action-historical observation data pair of the policy network Then, according to the local action-historical observation data pair of the strategy network and the action value of the drone at the current time step t Get the Q value of the policy network;

[0012] S3: Based on the local action-historical observation data pairs of the high observation range expert network and the policy network Obtain the observation value of the global state of the current environment by the high observation range expert network To obtain local action-historical observation data pairs of high observation range expert network And according to the action value of the drone at the current time step t Obtaining the Q value of the high observation range expert network

[0013] Based on the local action-historical observation data pairs of the low-observation range expert network and the policy network Obtain the observation value of the low observation range expert network on the global state of the current environment To obtain local action-historical observation data pairs of low-observation range expert network And according to the action value of the drone at the current time step t Get the Q value of the low observation range expert network

[0014] S4: Based on the Q value of the policy network and the Q value of the high observation range expert network Q-value of low observation range expert network Based on the loss function of the supervised learning policy network, the loss function of the high observation range expert network and the loss function of the low observation range expert network, the policy network, the high observation range expert network and the low observation range expert network are trained respectively; at the same time, the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network are updated; to obtain the final reinforcement learning environment network;

[0015] S5: Based on the policy network in the final reinforcement learning environment network, the predicted value of the drone action is obtained to achieve collaborative control of the drone swarm in an observation-constrained environment.

[0016] Furthermore, the reinforcement learning environment network is initialized, including initializing a discount factor γ, a soft update parameter λ of the target network, and an experience replay pool.

[0017] Furthermore, the experience replay pool stores Tuple data of s t represents the global state at time step t; Represents the observation value of the policy network of all drones on the global state of the current environment, Represents the observation value of the policy network of the i-th UAV on the global state of the current environment; represents the observation value of the high observation range expert network of all drones on the global state of the current environment, where represents the observation value of the high observation range expert network of the i-th UAV on the global state of the current environment; Represents the observation value of all drones and the low observation range expert network on the global state of the current environment, where represents the observation value of the low observation range expert network of the i-th UAV on the global state of the current environment; Represents the action value of all drones, represents the action value of the i-th drone; r t Indicates execution of action a t The total reward value given by the post-reinforcement learning environment network.

[0018] Furthermore, the Q value of the expert network with high observation range is obtained The formula used is as follows:

[0019]

[0020] In the formula, Indicates the action taken by the expert network with high observation range of the i-th UAV under the current observation value The obtained Q value; r t is the reward function value; γ is the discount factor; Indicates the action taken in the high observation range expert network The maximum Q value that can be obtained after entering the t+1 time step;

[0021] Get the Q value of the low observation range expert network The formula used is as follows:

[0022]

[0023] In the formula, Indicates the action taken by the expert network of the low observation range of the i-th UAV under the current observation value The obtained Q value; Indicates the action taken in the low observation range expert network The maximum Q value that can be achieved after entering the t+1 time step.

[0024] Furthermore, the high observation range expert network and the low observation range expert network are both trained using a reinforcement learning method;

[0025] The policy network is trained using an imitation learning method.

[0026] Furthermore, the loss function of the high observation range expert network is expressed as follows:

[0027]

[0028] in, represents the loss function of the high observation range expert network; E bs represents the expected function; represents the parameters of the high observation range expert network; represents the Q value of the target network of the expert network with high observation range;

[0029]

[0030] in, Parameters of the target network representing the high-observation-range expert network; Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the high observation range expert network; r t is the reward function value; γ is the discount factor;

[0031] The loss function of the low observation range expert network is expressed as follows:

[0032]

[0033] in, represents the loss function of the low observation range expert network; represents the parameters of the low observation range expert network; E bs represents the expected function; represents the Q value of the target network of the low observation range expert network,

[0034]

[0035] in, represents the parameters of the target network of the low-observation-range expert network, Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the low observation range expert network.

[0036] Furthermore, the loss function of the strategy network is expressed as follows:

[0037]

[0038] in, represents the loss function of the policy network; θ i is the parameter of the policy network; α is a custom coefficient. bs [·] indicates the expected function.

[0039] Furthermore, the formula used to update the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network is as follows:

[0040]

[0041] in, The parameters of the target network representing the low-observation-range expert network; represents the parameters of the target network of the high observation range expert network; λ is the soft update coefficient; represents the parameters of the target network of the updated low-observation-range expert network; Represents the parameters of the target network after the update of the high observation range expert network.

[0042] Beneficial effects: The present invention provides a method for drone swarm collaboration based on reinforcement learning in an observation-constrained environment, which builds a reinforcement learning environment network for each drone, and can help each drone to have a more thorough understanding of the environment, and improve the limitations on drone swarm decision-making caused by a small observation range. At the same time, drones can obtain more diverse experiences and strategies through high-observation range expert networks and low-observation range expert networks with different observation radii. By obtaining the global state at the current time step to obtain the observation value of the policy network, the output of the policy network can fully consider the behavior of other drones in the system and the dynamic changes of the entire system. The decisions made can fully take into account the global perspective and foresight, greatly improving the collaborative efficiency of the drone cluster and the final decision-making effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 It is a flow chart of the UAV swarm collaboration method in an observation-restricted environment of the present invention;

[0045] Figure 2 It is a schematic diagram of a drone swarm collaboration method based on reinforcement learning and imitation learning in an observation-constrained environment in an embodiment of the present invention;

[0046] Figure 3 A network structure diagram of a drone swarm collaboration method based on reinforcement learning and imitation learning in an observation-constrained environment in an embodiment of the present invention;

[0047] Figure 4 This is a flowchart of a drone swarm collaboration method based on reinforcement learning and imitation learning in an observation-constrained environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] This embodiment introduces a UAV swarm collaboration method based on reinforcement learning in an observation-constrained environment. Figure 1 , Figure 3 and Figure 4 As shown, the following steps are included:

[0050] S1: Establish a drone cluster containing I drones, and build a reinforcement learning environment network for the drone cluster; initialize the reinforcement learning environment network and initialize the experience replay pool;

[0051] The reinforcement learning environment network includes a policy network of the ith UAV, a high observation range expert network of the ith UAV, a low observation range expert network of the ith UAV, a target network of the high observation range expert network of the ith UAV, and a target network of the low observation range expert network of the ith UAV; wherein i=1...I, i represents the index number of the UAV, and I represents the total number of UAVs;

[0052] Among them, the observation radius of the expert network with high observation range of the i-th UAV is greater than the observation radius of the expert network with low observation range of the i-th UAV;

[0053] Among them, initializing the reinforcement learning environment network includes initializing the discount factor γ and the soft update parameter λ of the target network;

[0054] Preferably, the experience replay pool stores Tuple data of s t represents the global state at time step t; Represents the observation value of the policy network of all drones on the global state of the current environment, Represents the observation value of the policy network of the i-th UAV on the global state of the current environment; represents the observation value of the high observation range expert network of all drones on the global state of the current environment, where represents the observation value of the high observation range expert network of the i-th UAV on the global state of the current environment; Represents the observation value of all drones and the low observation range expert network on the global state of the current environment, where represents the observation value of the low observation range expert network of the i-th UAV on the global state of the current environment; Represents the action value of all drones, represents the action value of the i-th drone; r t Indicates execution of action a t The total reward value given by the post-reinforcement learning environment network;

[0055] Specifically, the experience replay pool D in this embodiment stores Tuple data composed of t represents the global state at time step t; Represents the observation value of the global state of the current environment obtained by the policy network of all drones according to the observation function, Represents the observation value of the global state of the current environment obtained by the strategy network of the i-th UAV according to the observation function; and They represent the observation values ​​of the global state of the current environment obtained by the high observation range expert network and the low observation range expert network of all drones according to the observation function, where represents the observation value of the global state of the current environment obtained by the high observation range expert network of the i-th UAV according to the observation function, represents the observation value of the global state of the current environment obtained by the low observation range expert network of the i-th UAV according to the observation function; Represents the action value of all drones, represents the action value of the i-th drone; r t Indicates execution of action a t The total reward value given by the environment is used by each drone for updating; t+1 It represents the global state after all drones execute actions and enter the next time step, that is, the global state at time step t+1. Represents the observation value obtained by the strategy network of all drones at time step t+1 according to the observation function; and They represent the corresponding observation values ​​obtained by the high observation range expert network and the low observation range expert network of all drones at time step t+1 according to the observation function; i represents the index number of the drone;

[0056] Specifically, this embodiment builds a reinforcement learning environment network of a drone cluster consisting of I drones, and initializes the reinforcement learning environment network; wherein the reinforcement learning environment network includes the strategy network agent_learner of the i-th drone i 、The high observation range expert network agent_upper of the i-th UAV i 、The low observation range expert network agent_lower of the i-th UAV i , the target network of the high observation range expert network of the ith UAV and the target network of the low observation range expert network of the ith UAV; In this embodiment, the constructed reinforcement learning environment network and a plurality of UAVs interacting therewith are combined to obtain a UAV swarm system, and the UAV swarm system is regarded as a multi-agent system, in which each UAV is regarded as an agent;

[0057] In this embodiment, in a drone swarm system including 1 drone, a communication network is designed for each drone, including a strategy network agent_learner i 、High Observation Range Expert Network agent_upper i , low observation range expert network agent_lower i , the network structures of the strategy network, the high observation range expert network and the low observation range expert network are the same, but the input observation values ​​are different. This embodiment uses a high observation range expert network and a low observation range expert network with different observation radiuses to expand the observation range of the drone. Compared with using a single observation network, it can provide more diverse drone cooperation strategies under limited observation conditions, such as how the drone moves to observe more content, or attacking at the current time step to strike the enemy, etc. At the same time, it can take into account longer-term goals, such as a drone can move in a certain direction, or approach a certain drone, etc., and can work with other drones at a certain moment in the future to complete tasks such as destroying enemy drones. Or although the current time step is far away, it can survive longer, etc. Among them, the task radius of the drone group>the observation radius of the high observation range expert network of the i-th drone>the observation radius of the low observation range expert network of the i-th drone.

[0058] like Figure 2 As shown in the figure, the observation values ​​of the policy network, high observation range expert network and low observation range expert network of the i-th UAV are o i , and In addition, each drone's agent_upper i Network and agent_lower i Each network maintains a target network with the same network structure as its own.

[0059] Specifically, after the reinforcement learning environment network is built, it is initialized, including initializing the discount factor γ, the soft update parameter λ of the target network, and the experience replay pool;

[0060] The policy network, high observation range expert network, low observation range expert network, target network of high observation range expert network and target network of low observation range expert network in this embodiment all adopt the deep recurrent Q network, i.e., DRQN ​​structure. This structure takes into account that in the reinforcement learning environment network of the drone cluster, each drone only obtains partial observation information of the environment, so the network introduces the gated recurrent unit structure (Gated Recurrent Unit, GRU) on the basis of DQN to make more effective use of the observed information. The DRQN ​​structure effectively retains historical input information, so that the intelligent agent has the ability to remember historical states and learn the observed information more fully. The DRQN ​​structure has three layers, the first layer is a fully connected layer, the activation function is ReLU, the second layer is a GRU layer, the third layer is a fully connected layer, the optimizer is Adam, and the network width is set to 64.

[0061] Specifically, this embodiment initializes the policy network, high observation range expert network, low observation range expert network, target network of high observation range expert network and target network of low observation range expert network of each drone, wherein the target network of high observation range expert network has the same initial parameters as the high observation range expert network, and the target network of low observation range expert network has the same initial parameters as the low observation range expert network. Specifically including:

[0062] First: Initialize the policy network agent_learner for the i-th drone i The value function Q(θ i ), high observation range expert network agent_upper i The value function and low observation range expert network agent_lower i The value function Among them, θ i is the parameter of the policy network; Q(θ i ) represents the value function of the policy network of the i-th UAV; are the parameters of the high observation range expert network, represents the value function of the high observation range expert network of the i-th UAV; is the expert network parameter for low observation range; represents the value function of the low observation range expert network of the i-th UAV;

[0063] Then: initialize the target network of the high observation range expert network and the target network of the low observation range expert network for the i-th UAV, and assign the parameters of the high observation range expert network to the target network of the high observation range expert network: And assign the parameters of the low observation range expert network to the target network of the low observation range expert network: in: is the target Q network corresponding to the high observation range expert network, is the parameter of the target network corresponding to the high observation range expert network, and the initial parameters are same, is the target Q network corresponding to the low observation range expert network, is the parameter of the target network of the low observation range expert network, and the initial parameters are same.

[0064] ← indicates assignment;

[0065] S2: Get the global state s at the current time step t t ; and according to the policy network, obtain the observation value of the policy network on the global state and the action value of the drone at the current time step t To obtain the local action-historical observation data pair of the policy network Then, according to the local action-historical observation data pair of the strategy network and the action value of the drone at the current time step t Get the Q value of the policy network;

[0066] The global state of this embodiment includes the location of obstacles in the environment, the location of the target point, the location of the drone, the flight direction, the flight speed, etc., including the information detected by all drones in the drone cluster system and the information of the drone itself. S3: Based on the local action-historical observation data of the high observation range expert network and the strategy network Obtain the observation value of the global state of the current environment by the high observation range expert network To obtain local action-historical observation data pairs of high observation range expert network And according to the action value of the drone at the current time step t Obtaining the Q value of the high observation range expert network

[0067] Based on the local action-historical observation data pairs of the low-observation range expert network and the policy network Obtain the observation value of the low observation range expert network on the global state of the current environment To obtain local action-historical observation data pairs of low-observation range expert network And according to the action value of the drone at the current time step t Get the Q value of the low observation range expert network

[0068] In this embodiment, the strategy network, the high observation range expert network, the low observation range expert network and their respective target networks are all uses of existing technologies in the field, and this embodiment only uses them to implement the functions of this embodiment. Therefore, they will not be described in detail here.

[0069] Preferably, the formula used to obtain the Q value of the high observation range expert network is as follows:

[0070] Specifically, the high observation range expert network and the low observation range expert network of this embodiment respectively input the corresponding local action-historical observation value data pair and the action value of the drone at the strategy network time step t , output the corresponding value function, that is, the Q value function of the high observation range expert network. During the learning process, randomly select states and actions, and iterate repeatedly using the Bellman equation to find the true Q value of each state and action. When making decisions, each state selects the action with the largest Q value, and iterates repeatedly to update the Q value. The iterative process of the high observation range expert network adopted by the i-th drone is as follows:

[0071]

[0072] In the formula, Indicates the action taken by the expert network with high observation range of the i-th UAV under the current observation value The obtained Q value; r t is the reward function value; γ is the discount factor; Indicates the action taken in the high observation range expert network The maximum Q value that can be obtained after entering the t+1 time step;

[0073] Similarly, the formula used to obtain the Q value of the low observation range expert network is as follows: that is, under the iterative process of the low observation range expert network of the i-th drone:

[0074]

[0075] In the formula, Indicates the action taken by the expert network of the low observation range of the i-th UAV under the current observation value The obtained Q value; Indicates the action taken in the low observation range expert network The maximum Q value that can be achieved after entering the t+1 time step.

[0076] S4: Based on the Q value of the policy network and the Q value of the high observation range expert network Q-value of low observation range expert network Based on the loss function of the supervised learning policy network, the loss function of the high observation range expert network and the loss function of the low observation range expert network, the policy network, the high observation range expert network and the low observation range expert network are trained respectively; at the same time, the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network are updated; to obtain the final reinforcement learning environment network;

[0077] Preferably, in this embodiment, the drone interacts with the reinforcement learning environment network, the high observation range expert network and the low observation range expert network are trained by the reinforcement learning method, and the policy network is trained by the imitation learning method. Until convergence, a trained drone swarm collaboration model, i.e., the reinforcement learning environment network, is obtained. Among them, the reinforcement learning method is used to train the high observation range expert network and the low observation range expert network respectively, and the imitation learning method is used to train the policy network. The methods are conventional methods in the field and will not be described here.

[0078] S5: Based on the policy network in the final reinforcement learning environment network, the predicted value of the drone action is obtained to achieve collaborative control of the drone swarm in an observation-constrained environment.

[0079] Specifically, according to the global state of the ith UAV at time step t and the policy network of the ith UAV, obtain the observation value of the policy network of the ith UAV on the global state of the current environment Then, the local action-historical observation data pairs of the policy network of the i-th UAV are obtained, that is, a sequence formed by the action values ​​of the UAV at multiple time steps in the history of the UAV and the observation value pairs of the historical policy network.

[0080] Specifically, in a partially observable environment, the i-th UAV is based on the current global state s t , through the policy network agent_learner i Observation function to obtain the observation value of the policy network and the action value of the drone at time step t Then obtain the local action-historical observation data pair of the strategy network in, Represents the local action-historical observation data pair of the policy network of the i-th drone;

[0081] Specifically, the policy network can perform environmental feedback and experience storage. The i-th drone will pair the local action-historical observation data of the policy network Input to the policy network agent_learner i:Q(θ i ),according to Select action value and execute.

[0082] When all drones select actions and execute them, the system obtains the global reward r at the current time step t t , and enter the next time step, that is, the t+1 time step, to obtain the global state s of the next time step t+1 Each drone has a global state s in the next time step. t+1 , repeat the above steps.

[0083] Specifically, storage To the experience replay pool D, if the time step reaches the set threshold, the reinforcement learning environment network is reset. Samples of size bs are sampled from the experience replay pool D to train the reinforcement learning environment network. Where bs represents the batch size of the sampling;

[0084] Preferably, according to the Bellman equation, the loss function of the high observation range expert network is expressed as follows:

[0085]

[0086] in, represents the loss function of the high observation range expert network; E bs represents the expectation function, which is the expectation after calculating the mean square error between the target network and the expert network with a high observation range; Represents the parameters of the high observation range expert network; represents the Q value of the target network of the high observation range expert network, that is, the target network of the high observation range expert network is introduced, and the Q value of the target network is calculated using the target network of the high observation range expert network. The representation is as follows:

[0087]

[0088] in, Parameters of the target network representing the high-observation-range expert network; Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the high observation range expert network; r t is the reward function value; γ is the discount factor;

[0089] Preferably, the loss function of the low observation range expert network is expressed as follows:

[0090]

[0091] in, represents the loss function of the low observation range expert network; represents the parameters of the low observation range expert network; E bs represents the expectation function, which is the expectation after calculating the mean square error between the target network and the expert network with low observation range; represents the Q value of the target network of the low observation range expert network, The representation is as follows:

[0092]

[0093] in, represents the parameters of the target network of the low-observation-range expert network, Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the low observation range expert network.

[0094] Preferably, the loss function of the policy network is expressed as follows:

[0095] Among them, the strategy network agent_learner of the i-th drone i Using supervised learning methods, we can imitate the high observation range expert network agent_upper of the i-th UAV i and the low observation range expert network agent_lower of the i-th UAV i The loss is calculated using mean square error, and the loss function is expressed as follows:

[0096]

[0097] in, represents the loss function of the policy network; θ i is the parameter of the policy network; α is a custom coefficient. bs [·] represents the expected function, which is the weighted expectation of the mean square error of the value function of the i-th UAV;

[0098] Preferably, the formula used to update the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network is as follows:

[0099] Specifically, the target network of the high observation range expert network and the target network of the low observation range expert network are updated by using the soft update parameter update method, that is, after the set number of training times, they are as follows:

[0100]

[0101]

[0102] in, The parameters of the target network representing the low-observation-range expert network; represents the parameters of the target network of the high observation range expert network; λ is the soft update coefficient; represents the parameters of the target network of the updated low-observation-range expert network; Indicates the parameters of the target network of the updated high observation range expert network. Specifically, in this embodiment, the parameters of the target network are updated only after a set number of training times, so that the entire reinforcement learning environment network can be more stable.

[0103] The parameter update method of the soft update of this embodiment can improve the stability of the algorithm. Specifically, in a drone swarm system of 1 drone, each drone adopts the above method for learning.

[0104] In this embodiment, after iterative training until convergence, the model is saved, and the subsequent drone swarm collaboration problem with limited observation environment directly calls the model output result without repeated training. Calling the policy network of the trained reinforcement learning environment network can solve the drone swarm collaboration problem in the observation limited environment of this embodiment.

[0105] This embodiment introduces two expert agents, a high observation range expert network and a low observation range expert network, and combines the idea of ​​imitation learning with the distributed multi-agent reinforcement learning method, which can help each drone to have a more complete understanding of the environment and improve the limitations of the drone group decision-making caused by the small observation range. At the same time, expert agents with different observation ranges are set up to enable drones to obtain more diverse experiences and strategies. By obtaining the global state at the current time step, the observation value of the policy network is obtained, so that the output of the policy network can fully consider the behavior of other drones in the system and the dynamic changes of the entire system. The decisions made can fully take into account the global perspective and foresight, greatly improving the collaborative efficiency of the drone cluster and the final decision-making effect.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A UAV swarm collaboration method based on reinforcement learning in an observation-constrained environment, characterized in that: The steps include: S1: Establish a drone cluster containing I drones, build a reinforcement learning environment network for the drone cluster and initialize it; The reinforcement learning environment network includes a policy network of the ith UAV, a high observation range expert network of the ith UAV, a low observation range expert network of the ith UAV, a target network of the high observation range expert network of the ith UAV, and a target network of the low observation range expert network of the ith UAV; wherein i=1...I, i represents the index number of the UAV, and I represents the total number of UAVs; Among them, the mission radius of the drone group>the observation radius of the expert network with high observation range of the i-th drone>the observation radius of the expert network with low observation range of the i-th drone; S2: Get the global state s at the current time step t t ; and according to the policy network, obtain the observation value of the policy network on the global state and the action value of the drone at the current time step t To obtain the local action-historical observation data pair of the policy network Then, according to the local action-historical observation data pair of the strategy network and the action value of the drone at the current time step t Get the Q value of the policy network; S3: Based on the local action-historical observation data pairs of the high observation range expert network and the policy network Obtain the observation value of the global state of the current environment by the high observation range expert network To obtain local action-historical observation data pairs of high observation range expert network And according to the action value of the drone at the current time step t Obtaining the Q value of the high observation range expert network Based on the local action-historical observation data pairs of the low-observation range expert network and the policy network Obtain the observation value of the low observation range expert network on the global state of the current environment To obtain local action-historical observation data pairs of low-observation range expert network And according to the action value of the drone at the current time step t Get the Q value of the low observation range expert network S4: Based on the Q value of the policy network and the Q value of the high observation range expert network Q-value of low observation range expert network Based on the loss function of the supervised learning policy network, the loss function of the high observation range expert network and the loss function of the low observation range expert network, the policy network, the high observation range expert network and the low observation range expert network are trained respectively; at the same time, the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network are updated; to obtain the final reinforcement learning environment network; S5: Based on the policy network in the final reinforcement learning environment network, the predicted value of the drone action is obtained to achieve collaborative control of the drone swarm in an observation-constrained environment.

2. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: Initializing the reinforcement learning environment network includes initializing a discount factor γ, a soft update parameter λ of the target network, and an experience replay pool.

3. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 2, characterized in that: The experience replay pool is stored Tuple data of s t represents the global state at time step t; Represents the observation value of the policy network of all drones on the global state of the current environment, Represents the observation value of the policy network of the i-th UAV on the global state of the current environment; represents the observation value of the high observation range expert network of all drones on the global state of the current environment, where represents the observation value of the high observation range expert network of the i-th UAV on the global state of the current environment; Represents the observation value of all drones and the low observation range expert network on the global state of the current environment, where represents the observation value of the low observation range expert network of the i-th UAV on the global state of the current environment; Represents the action value of all drones, represents the action value of the i-th drone; r t Indicates execution of action a t The total reward value given by the post-reinforcement learning environment network.

4. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: Obtaining the Q value of the high observation range expert network The formula used is as follows: In the formula, Indicates the action taken by the expert network with high observation range of the i-th UAV under the current observation value The obtained Q value; r t is the reward function value; γ is the discount factor; Indicates the action taken in the high observation range expert network The maximum Q value that can be obtained after entering the t+1 time step; Get the Q value of the low observation range expert network The formula used is as follows: In the formula, Indicates the action taken by the expert network of the low observation range of the i-th UAV under the current observation value The obtained Q value; Indicates the action taken in the low observation range expert network The maximum Q value that can be achieved after entering the t+1 time step.

5. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: The high observation range expert network and the low observation range expert network are both trained using a reinforcement learning method; The policy network is trained using an imitation learning method.

6. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: The loss function of the high observation range expert network is expressed as follows: in, represents the loss function of the high observation range expert network; E bs represents the expected function; Represents the parameters of the high observation range expert network; represents the Q value of the target network of the expert network with high observation range; in, Parameters of the target network representing the high-observation-range expert network; Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the high observation range expert network; r t is the reward function value; γ is the discount factor; The loss function of the low observation range expert network is expressed as follows: in, represents the loss function of the low observation range expert network; represents the parameters of the low observation range expert network; E bs represents the expected function; represents the Q value of the target network of the low observation range expert network, in, represents the parameters of the target network of the low-observation-range expert network, Indicates taking action After entering the t+1 time step, the maximum Q value that can be obtained based on the parameters of the target network of the low observation range expert network.

7. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: The loss function of the policy network is expressed as follows: in, represents the loss function of the policy network; θ i is the parameter of the policy network; α is a custom coefficient; E bs [·] indicates the expected function.

8. The method for UAV swarm collaboration based on reinforcement learning in an observation-constrained environment according to claim 1, characterized in that: The formula used to update the parameters of the target network of the high observation range expert network and the parameters of the target network of the low observation range expert network is as follows: in, The parameters of the target network representing the low-observation-range expert network; represents the parameters of the target network of the high observation range expert network; λ is the soft update coefficient; represents the parameters of the target network of the updated low-observation-range expert network; Represents the parameters of the target network after the update of the high observation range expert network.

Images