A multi-agent swarm cooperative control method based on reinforcement learning

Abstract

This invention relates to a multi-agent cluster cooperative control method based on reinforcement learning, comprising: obtaining normalized global state information based on a first normalized relative distance and a first normalized relative velocity between an agent and a virtual leader point, and a second normalized relative distance and a second normalized relative velocity between an agent and neighboring agents; calculating the individual reward of each agent based on an individual reward function, and obtaining the total reward for a preset interval based on the individual rewards of all agents; training an Actor-Critic network model based on the normalized global state information at the current moment, the actions performed by the agents, the total reward for the preset interval, and the normalized global state information at the current moment, to obtain a trained Actor-Critic network model; and deploying the trained Actor network onto the agents to obtain the final acceleration command to control the agent's movement. This invention has the advantages of high sample utilization, strong generalization, excellent cooperative effect, and stable training.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and automatic control technology, specifically to a multi-agent cluster cooperative control method based on reinforcement learning. Background Technology

[0002] With the rapid development of automatic control technology and artificial intelligence algorithms, UAV swarm cooperative control, as a distributed cooperative solution with high robustness and high operational efficiency, has been widely used in low-altitude inspection, emergency rescue, and military reconnaissance. Current practical engineering projects place higher demands on the autonomous aggregation, safe collision avoidance, and stable cooperative following of UAV swarms, especially in scenarios with limited perception and dynamically changing environments, posing severe challenges to the adaptability, sample utilization, and convergence performance of control algorithms.

[0003] In practical applications of collaborative control of UAV swarms, the commonly used traditional control methods mainly include the Boids model and the artificial potential field method. These methods rely on accurate environmental modeling and manually designed motion rules. Although they are simple to implement, they have significant limitations in actual UAV operation scenarios: due to the limited perception range of UAVs, the limited communication bandwidth, and the real-time changes in the environment, traditional methods are difficult to adjust flexibly and are prone to problems such as swarm failure, high collision risk, or unstable following.

[0004] Existing multi-agent deep deterministic policy gradient algorithms based on reinforcement learning, while capable of learning control policies through interaction with the environment, still suffer from three core problems when directly applied to UAV scenarios: First, they fail to fully consider the perception range constraints of UAVs, resulting in poor policy generalization ability in real-world environments with limited information; second, the algorithms do not design effective incentives for the characteristics of UAV swarm tasks, leading to low sample utilization and slow convergence speed; and third, the reward function lacks positive guidance for the aggregation target, making it easy for UAVs to explore without direction in their initial dispersed state, hindering efficient safe aggregation and stable following.

[0005] The aforementioned problems severely limit the engineering implementation of traditional drone swarm control methods in complex real-world scenarios. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a multi-agent cluster cooperative control method based on reinforcement learning. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a multi-agent swarm cooperative control method based on reinforcement learning, comprising: S1. For each agent in the agent cluster, based on the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point within the perception range, and the second normalized relative distance and the second normalized relative velocity between the agent and the neighboring agents within the perception range, normalized global state information is obtained. S2. During the interaction between the agent and the environment, the individual reward of each agent is calculated based on the agent's individual reward function, and the total reward for a preset range is obtained based on the individual rewards of all agents. S3, based on the current situation The normalized global state information at each moment, the action performed by the agent, the total reward of the preset interval, and The normalized global state information at each time step is used to train the Actor-Critic network model based on the gradient of a multi-agent deep deterministic policy until the model converges, thus obtaining the trained Actor-Critic network model. S4. Deploy the trained Actor network onto the agent, obtain the final acceleration command through the trained Actor network, and control the movement of the agent through the final acceleration command.

[0007] In one embodiment of the present invention, S1 includes: For each agent in the cluster, calculate the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point located within the perception range, respectively expressed as:

[0008] in, In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative distance between them In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative velocity between them In order to be in Virtual leadership point at all times Location, In order to be in Virtual leadership point at all times speed, In order to be in Time-based intelligent agent Location, In order to be in Time-based intelligent agent speed, For maximum sensing distance, Maximum speed constraint; The second normalized relative distance and the second normalized relative velocity between the agent and its neighboring agents within the perception range are calculated and expressed as follows:

[0009] in, In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative distance between them In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative velocity between them In order to be in Time Neighbor Intelligent Agent Location, In order to be in Time Neighbor Intelligent Agent speed; Based on the first normalized relative distance, the first normalized relative velocity, the second normalized relative distance, and the second normalized relative velocity, the local state information of the current agent is constructed, represented as:

[0010] in, In order to be in Time-based intelligent agent Local state information; According to the intelligent agent cluster The local state information of each agent is used to construct the global state information, and the global state information is normalized to the interval [-1, 1] to obtain the normalized global state information, which is represented as follows:

[0011] in, This is global status information. , This represents the total number of agents in the agent cluster.

[0012] In one embodiment of the present invention, S1 further includes: Determine the relationship between the Euclidean distance between the current agent and the neighboring agents and the maximum perception distance, and determine the relationship between the Euclidean distance between the current agent and the virtual leader point and the maximum perception distance; If the Euclidean distance between the current agent and the neighboring agent is less than or equal to the maximum perception distance, it means that the neighboring agent is within the perception range of the current agent. If the Euclidean distance between the current agent and the neighboring agent is greater than the maximum perception distance, the relative state between the current agent and the neighboring agent is set to 0. If the Euclidean distance between the current agent and the virtual leader point is less than or equal to the maximum perception distance, it means that the virtual leader point is within the perception range of the current agent. If the Euclidean distance between the current agent and the virtual leader point is greater than the maximum perception distance, the relative state between the current agent and the virtual leader point is set to 0. In one embodiment of the present invention, S2 includes: During the interaction between the agent and the environment, the distance constraint reward between agents is obtained by constructing a distance constraint reward function between agents based on the relationship between the Euclidean distance between the agents and the preset expected safe distance and maximum perception distance between agents. The distance constraint reward function between the agent and the virtual leader point is constructed based on the relationship between the Euclidean distance between the agents and the preset expected safe distance between the agents, and the distance constraint reward between the agent and the virtual leader point is obtained. Based on the L2 norm, a speed collaborative reward is obtained by constructing a speed collaborative reward function based on the speed of the agent and the speed of the virtual leader point; When all agents are in an initial dispersed state, and when the first normalized relative distance of each agent to the virtual leader point is less than a preset aggregation threshold, and this state lasts for at least a first preset duration, an initial aggregation achievement reward is provided to each agent using the initial aggregation achievement reward function. After the initial aggregation achievement reward is triggered, when the first normalized relative speed between each agent and the virtual leader point is less than a preset speed deviation threshold, and this state lasts for at least a second preset duration, a stable following achievement reward is provided to each agent using the stable following achievement reward function. The individual reward for each agent is obtained based on the agent-to-agent distance constraint reward, the agent-to-virtual-leader point distance constraint reward, the speed cooperation reward, the initial aggregation achievement reward, and the stable following achievement reward. The initial total reward is obtained by summing the individual rewards of all the agents. pass The initial total reward is pruned to obtain the total reward for the preset interval. In one embodiment of the present invention, the distance-constrained reward function between agents is expressed as:

[0013] in, Rewards are based on distance constraints between agents. Let be the Euclidean distance between the agents. The preset expected safe distance between agents, For maximum sensing distance, , All are positive constant coefficients; The distance-constrained reward function between the agent and the virtual leader point is expressed as follows:

[0014] in, Rewards are constrained by the distance between the agent and the virtual leader point. Positive constant coefficients; The speed-cooperative reward function is expressed as follows:

[0015] in, For speed-coordinated rewards, The velocity vector of the virtual leader point. For intelligent agents The velocity vector, for Norm, It is a positive constant coefficient. In one embodiment of the present invention, the initial aggregation achievement reward function is expressed as:

[0016] in, As an initial achievement reward, This represents the total number of agents in the agent cluster. In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative distance between them The preset expected safe distance between agents, This is the initial aggregation reward coefficient; The stable following achievement reward function is expressed as follows:

[0017] in, To ensure consistent achievement rewards, In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative velocity between them To ensure stable follow-up reward coefficients. In one embodiment of the present invention, the individual reward function of the agent is expressed as:

[0018] in, For intelligent agents Individual rewards Rewards are based on distance constraints between agents. Rewards are constrained by the distance between the agent and the virtual leader point. For speed-coordinated rewards, As an initial achievement reward, To ensure consistent achievement rewards, This represents the total number of agents in the agent cluster. In one embodiment of the present invention, the Actor-Critic network model includes and N Each of the aforementioned intelligent agents corresponds to one-to-one N There are one Actor network and one Critic network, where: No. The input to the Actor network is the agent. The local state information is used to obtain the agent's information. The initial acceleration command is obtained and mapped to [the target value] through an activation function. 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command; The Critic network takes normalized global state information as input and outputs a state value function. In one embodiment of the present invention, S3 includes: Will the current The normalized global state information at each moment, the action performed by the agent, the total reward of the preset interval, and The normalized global state information at each moment is stored in a shared global experience replay pool; Randomly sample batch data from the global experience replay pool, update the parameters of the Critic network by minimizing the mean squared error loss of the Critic network, and update the parameters of the Actor network of each agent by maximizing the policy gradient of the Actor network. Each fixed number of steps is proportional τUpdate the target Actor and target Critic networks until the model converges, resulting in a trained Actor-Critic network model. In one embodiment of the present invention, S4 includes: The first A trained Actor network is deployed to the agent. The airborne controller inputs intelligent agents The agent obtains local state information in real time. The initial acceleration command is obtained and mapped to [the target value] through an activation function. 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command is used to control the intelligent agent. The movement.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention has a high sample utilization rate: through the experience sharing mechanism, it integrates the interaction samples of all intelligent agents, improves the sample reuse rate, and accelerates the algorithm convergence speed.

[0020] The present invention has strong generalization ability: by using relative state information and normalization processing, the strategy can be applied in different environmental sizes and location scenarios.

[0021] The perception-limited adaptation of this invention restricts state information to the perception range of the intelligent agent, which meets the needs of scenarios with limited communication perception in practical applications.

[0022] The present invention has excellent collaborative effects: it can effectively realize the aggregation, collision avoidance and collaborative movement of intelligent agents following virtual leader points, and is applicable to a variety of multi-agent carriers.

[0023] This invention is adapted to sparse scenarios: it constructs a composite reward function of "basic constraints + stage incentives", and the stage achievement reward provides effective positive feedback for sparse reward scenarios, solves the problem of the agent's initial directionless exploration and training stagnation, and improves the achievement rate and efficiency of stage goals.

[0024] The training stability of this invention is achieved by updating the target network parameters through soft updates, avoiding oscillations during training and ensuring the model's convergence stability.

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] Figure 1This is a flowchart illustrating a multi-agent cluster cooperative control method based on reinforcement learning provided by the present invention. Figure 2 This is a diagram of the MADDPG and experience-sharing mechanism algorithm framework provided by the present invention; Figure 3 This is a system architecture diagram provided by the present invention; Figure 4 This invention provides a reward curve diagram for a training process; Figure 5 This invention provides a cluster motion trajectory diagram. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0028] Example 1 The existing technical solution describes a collaborative control method for multi-UAV swarm collaborative motion with Multi-Agent Deep Deterministic Policy Gradient (MADDPG) as the core algorithm framework. It realizes multi-agent swarm collaborative control with the basic MADDPG algorithm as the core. Although it can complete basic aggregation and following collaborative actions, it is not adaptable to actual application scenarios. The core defects are concentrated in the design of reward function and the adaptation of perception scenario. Specifically: (1) The reward function design is simple and the reward signal is sparse, which easily leads to training stagnation: Only continuous negative rewards based on distance / speed constraints between agents and between agents and virtual leader points are designed. There is no positive incentive mechanism. In the scenario where agents are initially dispersed, agents cannot obtain effective positive feedback for a long time. The exploration behavior has no clear direction and training stagnation is very likely to occur. The algorithm training efficiency is low and the convergence stability is poor. Moreover, the reward value is not pruned. Extreme reward values ​​are easy to cause training oscillation and further affect the model convergence effect. (2) The state observation is not combined with the actual needs of perception limitation, resulting in poor engineering adaptability: Although the relative position and relative velocity are normalized, the observation information is not limited to the perception range of the agent, which is inconsistent with the scenario where the agent's communication and perception capabilities are limited in actual applications. This leads to poor generalization of the trained strategy in real engineering scenarios, making it difficult to achieve efficient collaborative control under perception limitation conditions.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-agent cluster cooperative control method based on reinforcement learning provided by the present invention. The present invention provides a multi-agent cluster cooperative control method based on reinforcement learning, which includes: Step 1: For each agent in the agent cluster, obtain normalized global state information based on the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point within the perception range, and the second normalized relative distance and the second normalized relative velocity between the agent and its neighboring agents within the perception range.

[0030] Specifically, the intelligent agent cluster includes N An intelligent agent, for the current intelligent agent In other words, it is possible to determine the intelligent agent. Normalized relative distance and relative velocity between the agent and the virtual leader point within the perception range, and the agent's ability to determine the agent's position. The normalized relative distances and relative velocities between the agent and other neighboring agents within the perception range can be used to obtain the agent's information. Local state information, and the intelligent agent cluster N The global state information is obtained by concatenating the local state information of each agent, and then normalized to obtain the normalized global state information. The perception range refers to the effective distance at which an agent (e.g., a drone) can effectively acquire state information, defined by the maximum perception distance. The virtual leader point is a preset target trajectory point (not a physical entity) of the cluster, used to guide the cluster of agents (e.g., a drone cluster) to complete the task of gathering and collaborative following. Its position and speed are preset by the task trajectory. The maximum perception distance refers to the farthest effective distance at which each agent (e.g., a drone) can effectively collect the state information of other agents or virtual leader points when constructing its own state observation. This maximum perception distance is a preset value. The maximum perception distance can be adjusted according to experimental needs in the simulation scenario (e.g., 80m, 100m, or 120m). In the actual scenario, the maximum effective perception distance of the drone hardware is directly adopted to ensure the adaptability of the algorithm to the real scenario.

[0031] In this embodiment, the intelligent agent cluster can be a drone cluster, which includes... N The drone swarm can also be any other swarm that can be controlled using the control method provided in this embodiment, and this embodiment does not specifically limit this. For example, this embodiment selects 5 small fixed-wing drones to form a drone swarm, which completes the collaborative task of "initial dispersion and aggregation → stable following of the virtual leader point" in a two-dimensional planar space. The boundary of the two-dimensional operation space is [-10, 80] × [-10, 80] meters, the number of drones is 5, the initial position of the virtual leader point is (0, 0) meters, the movement speed is 0.6 m / s, the expected safe distance of the drones is 5 meters, the maximum sensing distance is 10 meters, the minimum speed constraint of the drones is 0 m / s, the maximum speed constraint is 1.5 m / s, and the acceleration constraint is ±1 m / s. 2The state update step is 0.1 seconds, and the virtual leader point moves along a preset straight trajectory at a constant speed.

[0032] In this embodiment, the agent motion model satisfies:

[0033] in, In order to be in Time-based intelligent agent Location, In order to be in Time-based intelligent agent Location, In order to be in Time-based intelligent agent speed, In order to be in Time-based intelligent agent speed, For time intervals, For intelligent agents The weight, In order to be in Time-based intelligent agent Control of external forces, For minimum speed constraints, For maximum speed constraints, This is the truncation function, and its formula is: .

[0034] In one specific embodiment, step 1 includes: Step 1.1: For each agent in the cluster, calculate the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point within its perception range, expressed as follows:

[0035] in, In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative distance between them In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative velocity between them In order to be in Virtual leadership point at all times Location, In order to be in Virtual leadership point at all times speed, In order to be in Time-based intelligent agent Location, In order to be in Time-based intelligent agent speed, This represents the maximum sensing distance.

[0036] Step 1.2: Calculate the second normalized relative distance and the second normalized relative velocity between the agent and its neighboring agents within the perception range, expressed as follows:

[0037] in, In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative distance between them In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative velocity between them In order to be in Time Neighbor Intelligent Agent Location, In order to be in Time Neighbor Intelligent Agent speed; Step 1.3: Construct the local state information of the current agent based on the first normalized relative distance, the first normalized relative velocity, the second normalized relative distance, and the second normalized relative velocity.

[0038] Specifically, for any intelligent agent In terms of its The observation information (local state information) at any given time consists of two parts: the agent's... Interactions with other intelligent agents and among intelligent agents The state between the virtual leader point and the virtual leader point is defined as follows:

[0039] in, In order to be in Time-based intelligent agent Local state information, Indicates for intelligent agents The number of other intelligent agents within the perception range.

[0040] Step 1.4: Based on the intelligent agent cluster... The local state information of each agent is used to construct the global state information, and the global state information is normalized to the interval [-1,1] to obtain the normalized global state information.

[0041] Specifically, the global state information of the entire system can be obtained by integrating the local state information of individual agents, as follows:

[0042] in, This is global status information. , This represents the total number of agents in the agent cluster.

[0043] Then, the global state information is normalized to the interval [-1,1] to obtain the normalized global state information.

[0044] In an optional embodiment, step 1 of this embodiment requires performing state masking processing first, and then performing other steps based on the state masking processing result. The state masking processing steps include: Determine the relationship between the Euclidean distance and the maximum perception distance between the current agent and its neighboring agents, and the relationship between the Euclidean distance and the maximum perception distance between the current agent and the virtual leader point. If the Euclidean distance between the current agent and its neighboring agents is less than or equal to the maximum perception distance, it means that the neighboring agent is within the current agent's perception range. If the Euclidean distance between the current agent and its neighboring agents is greater than the maximum perception distance, then set the relative state between the current agent and its neighboring agents to 0. Similarly, if the Euclidean distance between the current agent and the virtual leader point is less than or equal to the maximum perception distance, it means that the virtual leader point is within the current agent's perception range. If the Euclidean distance between the current agent and the virtual leader point is greater than the maximum perception distance, then set the relative state between the current agent and the virtual leader point to 0.

[0045] Specifically, calculate the current intelligent agent Intelligent agents with neighbors Euclidean distance between them, current agent With virtual leadership points The Euclidean distance between them is expressed as:

[0046]

[0047] in, For intelligent agents Intelligent agents with neighbors The Euclidean distance between them For intelligent agents With virtual leadership points The Euclidean distance between them, For intelligent agents Position coordinates, ( For Neighbor Intelligent Agent Position coordinates, ( Virtual leadership point The location coordinates.

[0048] Therefore, if Then retain the corresponding first normalized relative distance. First normalized relative velocity ,like Then the neighboring intelligent agent Corresponding state features , Set to 0, if Then retain the corresponding second normalized relative distance. Second normalized relative velocity ,like Then the neighboring intelligent agent Corresponding state features , Set to 0 to implement perceived constrained constraints.

[0049] It should be noted that the core of the global state information in this embodiment is the relative distance (2D) and relative speed (2D) with the virtual leader point. If there are neighboring intelligent agents within the perception range, the relative distance (2D) and relative speed (2D) with the neighboring intelligent agents can be added to the extended dimensions of the global state information.

[0050] Furthermore, considering the effectiveness of the action space, the actions of an intelligent agent in a two-dimensional planar space cannot exceed a defined reasonable range. Therefore, it is necessary to constrain the magnitude of the actions. ,in, For intelligent agents acceleration, Represents the lower bound of acceleration. The upper bound represents the acceleration. The action of the entire system consists of the effective actions of all agents, represented as follows: .

[0051] Step 2: During the interaction between the agent and the environment, calculate the individual reward of each agent based on the agent's individual reward function, and obtain the total reward for the preset interval based on the individual rewards of all agents.

[0052] In one specific embodiment, step 2 includes: Step 2.1: During the interaction between the agent and the environment, the distance constraint reward function between agents is constructed based on the relationship between the Euclidean distance between agents and the preset expected safe distance and maximum perception distance between agents, so as to obtain the distance constraint reward between agents.

[0053] Here, the distance-constrained reward function between agents is expressed as:

[0054] in, Rewards are based on distance constraints between agents. This represents the Euclidean distance between agents, specifically the Euclidean distance between the current agent and its neighboring agents. The preset expected safe distance between agents, For maximum sensing distance, , All are positive constant coefficients.

[0055] Step 2.2: Construct a distance constraint reward function between an agent and a virtual leader point based on the relationship between the Euclidean distance between agents and the preset expected safe distance between agents, and obtain the distance constraint reward between the agent and the virtual leader point.

[0056] Here, the distance-constrained reward function between the agent and the virtual leader point is expressed as:

[0057] in, Rewards are constrained by the distance between the agent and the virtual leader point. It is a positive constant coefficient.

[0058] Step 2.3: Based on the L2 norm, construct a speed collaborative reward function according to the speed of the agent and the speed of the virtual leader point to obtain the speed collaborative reward.

[0059] Here, the speed-coordinated reward function is expressed as:

[0060] in, For speed-coordinated rewards, The velocity vector of the virtual leader point. For intelligent agents The velocity vector, for Norm, It is a positive constant coefficient.

[0061] Speed-coordinated rewards are used to penalize speed deviations and guide the agent's speed to align with the virtual leader point.

[0062] Step 2.4: When all agents are in the initial dispersed state, and when the first normalized relative distance of each agent to the virtual leader point is less than the preset aggregation threshold, and this state lasts for at least the first preset duration, the initial aggregation achievement reward is provided to each agent using the initial aggregation achievement reward function.

[0063] Specifically, when all agents are in an initial dispersed state, the agents The first normalized relative distance to the virtual leadership point All are less than a preset aggregation threshold (the preset aggregation threshold is, for example, the desired safe distance between agents). When the aggregation condition is 1.2 times that of the target (and the target condition is 1.2 times that of the target), and this state lasts for no less than the first preset duration (e.g., 3 time steps), each agent is given a one-time positive reward, namely the initial aggregation achievement reward. This initial aggregation achievement reward is a global reward. When all agents meet the aggregation conditions, each agent receives this one-time positive reward, which is used to incentivize the entire cluster to complete aggregation.

[0064] Here, the initial aggregation achievement reward function is expressed as:

[0065] in, As an initial achievement reward, This represents the initial aggregation reward coefficient.

[0066] Initial cluster achievement rewards are triggered only when the conditions are met for the first time during the training period to avoid repeated incentives. Negative rewards are given when drones are too close to each other to avoid collisions within the cluster.

[0067] Step 2.5: After triggering the initial aggregation achievement reward, when the first normalized relative speed between each agent and the virtual leader point is less than the preset speed deviation threshold, and this state lasts for at least the second preset duration, the stable following achievement reward is provided to each agent using the stable following achievement reward function.

[0068] Specifically, after the cluster completes its initial aggregation, i.e. after step 2.4 is completed, the first normalized relative velocity of all agents and the virtual leader point is... All are less than the preset speed deviation threshold (for example, the preset speed deviation threshold is set to a value of...). If the value is 0.1 times that of the initial gathering achievement reward, and this state lasts for no less than the second preset time (e.g., 5 time steps), a one-time positive reward will be given, namely the stable following achievement reward. This stable following achievement reward will only take effect when the conditions are met for the first time after the initial gathering achievement reward is triggered.

[0069] Here, the stable follower achievement reward function is expressed as:

[0070] in, To ensure consistent achievement rewards, To ensure stable follow-up reward coefficients.

[0071] Step 2.6: Obtain the individual reward for each agent based on the inter-agent distance constraint reward, the distance constraint reward between the agent and the virtual leader point, the speed cooperation reward, the initial aggregation achievement reward, and the stable following achievement reward.

[0072] Here, the individual reward function of the agent is expressed as:

[0073] in, For intelligent agents Individual rewards.

[0074] Step 2.7: Obtain the initial total reward based on the sum of the individual rewards of all agents.

[0075] Here, the initial total reward is expressed as:

[0076] in, This is the initial total reward.

[0077] Step 2.8, through The initial total reward is pruned to obtain a total reward within a preset range, in order to avoid training oscillations caused by extreme values.

[0078] Step 3, based on the current situation Normalized global state information at each moment, actions performed by the agent, total reward for the preset interval, and The normalized global state information at each time step is used to train the Actor-Critic network model based on the gradient of a multi-agent deep deterministic policy until the model converges, resulting in a trained Actor-Critic network model.

[0079] In this embodiment, the Actor-Critic network model includes and N One-to-one correspondence between intelligent agents N There are one Actor network and one Critic network, where: No. The input to the Actor network is the agent. The local state information is used to obtain the agent's information. The initial acceleration command is obtained and mapped to [the target value] using an activation function (such as the tanh activation function). 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command is mapped to the following logic: a = 5 × tanh(o), where o is the initial acceleration command output by the Actor network and a is the final acceleration command, ensuring that the output range conforms to the physical constraints of the UAV.

[0080] The input to the Critic network is normalized global state information (5×4=20 dimensions), and the output is a state value function used to evaluate the value of the Actor network's actions.

[0081] In a specific embodiment, such as Figure 2 As shown, step 3 includes: Step 3.1, set the current Normalized global state information at each moment, actions performed by the agent, total reward for the preset interval, and The normalized global state information at any given moment is stored in a shared global experience replay pool.

[0082] Specifically, each agent acts as an independent decision-making unit, outputting actions (final acceleration commands) based on its current Actor network state, and interacting with the environment. The interaction content includes the current... Normalized global state information at any given moment Actions performed by the agent (a) and the total reward for the preset interval. and Normalized global state information at any given time The action 'a' executed by the agent is the final acceleration command obtained through its current Actor network. The normalized global state information at each moment and the total reward for the preset interval are determined through the above steps. Normalized global state information at any given time This provides normalized global state information after each agent performs its corresponding action 'a'. The quadruple ( ,a, , Each sample is stored as a global experience replay pool to be used as training data for training the Actor-Critic network model.

[0083] Step 3.2: Randomly sample batch data from the global experience replay pool, update the parameters of the Critic network by minimizing the mean squared error loss of the Critic network, and update the parameters of the Actor network of each agent by maximizing the policy gradient of the Actor network.

[0084] Specifically, a batch of samples of batch B is randomly sampled from the global experience replay pool to obtain B quadruplets. The Actor-Critic network model is trained using these B quadruplets. The parameters of the current Critic network are updated by gradient descent to minimize the mean squared error loss, and the parameters of the Actor network are updated by gradient ascent to maximize the policy gradient. Step 3.3, each fixed number of steps proportionally τ Update the target Actor and target Critic networks until the model converges, resulting in a trained Actor-Critic network model.

[0085] Specifically, after a fixed number of steps, the target Actor and target Critic networks are updated by a ratio τ to ensure training stability.

[0086] Step 4: Deploy the trained Actor network onto the agent, obtain the final acceleration command through the trained Actor network, and control the agent's movement through the final acceleration command.

[0087] Specifically, the first A trained Actor network is deployed to the agent. The airborne controller inputs intelligent agents The agent obtains local state information in real time. The initial acceleration command is obtained and mapped to [the target value] through an activation function. 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command is used to control the intelligent agent. The movement.

[0088] like Figure 3 As shown, the system architecture provided in this embodiment consists of four modules: the UAV swarm layer, the environment interaction layer, the algorithm core layer, and the training and deployment layer. The connectivity between the modules is as follows: (1) Drone Cluster Layer: It consists of 5 isomorphic drones. Each drone is equipped with an inertial measurement unit (IMU) and a communication module. It is responsible for collecting its own status, the status of its neighbors within the perception range and the status of the virtual leader point, and receiving acceleration control commands output by the algorithm core layer.

[0089] (2) Environment Interaction Layer: Simulates a two-dimensional plane unobstructed physical environment, including a virtual leader point, responsible for updating the drone's position / velocity, mission phase determination, and feeding back the status information to the algorithm core layer.

[0090] (3) Algorithm core layer: The reinforcement learning framework based on MADDPG includes a perceptually restricted state encoder, a composite reward calculator, and an Actor-Critic network module, which is the core improvement of this invention.

[0091] (4) Training and deployment layer: responsible for sample storage, network parameter update, model saving and deployment. After training, the Actor network is distributed to each UAV to achieve distributed control.

[0092] This embodiment is simulated and verified in a Python environment, such as... Figure 4 and Figure 5 As shown, the results are as follows: Convergence performance: After 800 training rounds, the reward curve tends to be stable, eventually stabilizing in the range of -150±5 with no obvious oscillations. The convergence speed is 60% faster than without experience sharing.

[0093] Coordination performance: The drones start from an initial dispersed state and complete the aggregation in 28 seconds. During the following phase, the formation spacing is maintained at 5m±0.3m, and the average speed deviation is 0.12m / s.

[0094] Collision avoidance performance: The minimum distance between drones is 1.2-1.5m, and no collision occurs, meeting the requirements for safe operation.

[0095] Robust performance: After a brief disconnection (3s) of a single drone, it can reconnect to the cluster and resume its collaborative state within 5s without any loss of connection.

[0096] This invention constructs a composite reward function consisting of "basic constraint reward + phased achievement reward + reward pruning". The phased achievement reward (initial clustering achievement reward and stable following achievement reward) provides clear positive feedback for sparse reward scenarios, solving the training stagnation problem in the initial dispersed scenario of the agent. At the same time, reward pruning avoids training oscillations caused by extreme values.

[0097] This invention uses normalized relative position and relative velocity as the agent's observation information, and strictly limits the observation information to the agent's perception range. It uses state masking to process features that exceed the perception range, so that the state design fits the needs of scenarios where the agent's communication and perception are limited in actual applications, thereby improving the generalization and engineering adaptability of the strategy. This invention introduces an experience-sharing mechanism based on the MADDPG algorithm. All agents share policy network parameters and experience replay pools, uniformly store interaction samples and jointly optimize them, which greatly improves sample utilization, accelerates algorithm convergence speed, and ensures the high efficiency of multi-agent cluster collaborative control. In the training phase of this invention, a centralized value evaluation network (Critic) is used to optimize the strategy based on the global state. In the execution phase, the agent only relies on its own local perception information to autonomously output action decisions without the need for central node control. This ensures the accuracy of strategy optimization and improves the robustness and distributed collaborative capability of the cluster system.

[0098] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0099] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0100] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, any modifications made without departing from the inventive concept should be considered within the scope of protection of the present invention.

Claims

1. A multi-agent cluster cooperative control method based on reinforcement learning, characterized in that, include: S1. For each agent in the agent cluster, based on the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point within the perception range, and the second normalized relative distance and the second normalized relative velocity between the agent and the neighboring agents within the perception range, normalized global state information is obtained. S2. During the interaction between the agent and the environment, the individual reward of each agent is calculated based on the agent's individual reward function, and the total reward for a preset range is obtained based on the individual rewards of all agents. S3, based on the current situation The normalized global state information at each moment, the action performed by the agent, the total reward of the preset interval, and The normalized global state information at each time step is used to train the Actor-Critic network model based on the gradient of a multi-agent deep deterministic policy until the model converges, thus obtaining the trained Actor-Critic network model. S4. Deploy the trained Actor network onto the agent, obtain the final acceleration command through the trained Actor network, and control the movement of the agent through the final acceleration command.

2. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 1, characterized in that, S1 includes: For each agent in the cluster, calculate the first normalized relative distance and the first normalized relative velocity between the agent and the virtual leader point located within the perception range, respectively expressed as: in, In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative distance between them In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative velocity between them In order to be in Virtual leadership point at all times Location, In order to be in Virtual leadership point at all times speed, In order to be in Time-based intelligent agent Location, In order to be in Time-based intelligent agent speed, For maximum sensing distance, Maximum speed constraint; The second normalized relative distance and the second normalized relative velocity between the agent and its neighboring agents within the perception range are calculated and expressed as follows: in, In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative distance between them In order to be in Time-based intelligent agent Intelligent agents with neighbors The second normalized relative velocity between them In order to be in Time Neighbor Intelligent Agent Location, In order to be in Time Neighbor Intelligent Agent speed; Based on the first normalized relative distance, the first normalized relative velocity, the second normalized relative distance, and the second normalized relative velocity, the local state information of the current agent is constructed, represented as: in, In order to be in Time-based intelligent agent Local state information; According to the intelligent agent cluster The local state information of each agent is used to construct the global state information, and the global state information is normalized to the interval [-1, 1] to obtain the normalized global state information, which is represented as follows: in, This is global status information. , This represents the total number of agents in the agent cluster.

3. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 1, characterized in that, S1 further includes: Determine the relationship between the Euclidean distance between the current agent and the neighboring agents and the maximum perception distance, and determine the relationship between the Euclidean distance between the current agent and the virtual leader point and the maximum perception distance; If the Euclidean distance between the current agent and the neighboring agent is less than or equal to the maximum perception distance, it means that the neighboring agent is within the perception range of the current agent. If the Euclidean distance between the current agent and the neighboring agent is greater than the maximum perception distance, the relative state between the current agent and the neighboring agent is set to 0. If the Euclidean distance between the current agent and the virtual leader point is less than or equal to the maximum perception distance, it means that the virtual leader point is within the perception range of the current agent. If the Euclidean distance between the current agent and the virtual leader point is greater than the maximum perception distance, the relative state between the current agent and the virtual leader point is set to 0.

4. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 1, characterized in that, S2 includes: During the interaction between the agent and the environment, the distance constraint reward between agents is obtained by constructing a distance constraint reward function between agents based on the relationship between the Euclidean distance between the agents and the preset expected safe distance and maximum perception distance between agents. The distance constraint reward function between the agent and the virtual leader point is constructed based on the relationship between the Euclidean distance between the agents and the preset expected safe distance between the agents, and the distance constraint reward between the agent and the virtual leader point is obtained. Based on the L2 norm, a speed collaborative reward is obtained by constructing a speed collaborative reward function based on the speed of the agent and the speed of the virtual leader point; When all agents are in an initial dispersed state, and when the first normalized relative distance of each agent to the virtual leader point is less than a preset aggregation threshold, and this state lasts for at least a first preset duration, an initial aggregation achievement reward is provided to each agent using the initial aggregation achievement reward function. After the initial aggregation achievement reward is triggered, when the first normalized relative speed between each agent and the virtual leader point is less than a preset speed deviation threshold, and this state lasts for at least a second preset duration, a stable following achievement reward is provided to each agent using the stable following achievement reward function. The individual reward for each agent is obtained based on the agent-to-agent distance constraint reward, the agent-to-virtual-leader point distance constraint reward, the speed cooperation reward, the initial aggregation achievement reward, and the stable following achievement reward. The initial total reward is obtained by summing the individual rewards of all the agents. pass The initial total reward is pruned to obtain the total reward for the preset interval.

5. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 4, characterized in that, The distance constraint reward function between agents is expressed as follows: in, Rewards are based on distance constraints between agents. Let Euclidean distance be the distance between the agents. The preset expected safe distance between agents, For maximum sensing distance, , All are positive constant coefficients; The distance-constrained reward function between the agent and the virtual leader point is expressed as follows: in, Rewards are constrained by the distance between the agent and the virtual leader point. Positive constant coefficients; The speed-cooperative reward function is expressed as follows: in, For speed-coordinated rewards, The velocity vector of the virtual leader point. For intelligent agents The velocity vector, for Norm, It is a positive constant coefficient.

6. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 4, characterized in that, The initial aggregation achievement reward function is expressed as follows: in, As an initial achievement reward, This represents the total number of agents in the agent cluster. In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative distance between them The preset expected safe distance between agents, This is the initial aggregation reward coefficient; The stable following achievement reward function is expressed as follows: in, To ensure consistent achievement rewards, In order to be in Time-based intelligent agent With virtual leadership points The first normalized relative velocity between them To ensure stable follow-up reward coefficients.

7. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 4, characterized in that, The individual reward function of the agent is expressed as: in, For intelligent agents Individual rewards Rewards are based on distance constraints between agents. Rewards are constrained by the distance between the agent and the virtual leader point. For speed-coordinated rewards, As an initial achievement reward, To ensure consistent achievement rewards, This represents the total number of agents in the agent cluster.

8. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 1, characterized in that, The Actor-Critic network model includes and N Each of the aforementioned intelligent agents corresponds to one-to-one N There are one Actor network and one Critic network, where: No. The input to the Actor network is the agent. The local state information is used to obtain the agent's information. The initial acceleration command is obtained and mapped to [the target value] through an activation function. 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command; The Critic network takes normalized global state information as input and outputs a state value function.

9. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 8, characterized in that, S3 includes: Will the current The normalized global state information at each moment, the action performed by the agent, the total reward of the preset interval, and The normalized global state information at each moment is stored in a shared global experience replay pool; Randomly sample batch data from the global experience replay pool, update the parameters of the Critic network by minimizing the mean squared error loss of the Critic network, and update the parameters of the Actor network of each agent by maximizing the policy gradient of the Actor network. Each fixed number of steps is proportional τ Update the target Actor and target Critic networks until the model converges, resulting in a trained Actor-Critic network model.

10. The multi-agent cluster cooperative control method based on reinforcement learning according to claim 1, characterized in that, S4 includes: The first A trained Actor network is deployed to the agent. The airborne controller inputs intelligent agents The agent obtains local state information in real time. The initial acceleration command is obtained and mapped to [the target value] through an activation function. 1,1] Then linearly scaled to [ 5,5] To output intelligent agents The final acceleration command is used to control the intelligent agent. The movement.

Images