Multi-agent system adaptive target tracking formation control method based on reinforcement learning

Abstract

The invention discloses a multi-agent system adaptive target tracking formation control method based on reinforcement learning, and relates to the field of multi-agent system target tracking formation control, and the method comprises the following steps: 1, designing a scene, and making a decision in real time; 2, constructing a constraint analysis model, and constructing a target function and a constraint condition; step 3, designing an intelligent agent strategy; designing an observation space, an action space and a reward function; step 4, experiment design; step 5, intelligent agent training; a two-stage training process is adopted; 6, designing a self-adaptive target tracking formation control framework; an ATTFCF framework is provided. According to the method, target tracking and formation control problems are fused, the MPR-DDPG algorithm is adopted for solving, the problems that multi-agent cooperative target tracking and formation control is low in precision and poor in strategy safety are solved, and the method can be suitable for cooperative target tracking tasks of different numbers of agents.

Description

Technical Field

[0001] This invention relates to the field of target tracking and formation control in multi-agent systems, and particularly to an adaptive target tracking and formation control method for multi-agent systems based on reinforcement learning. Background Technology

[0002] With the continuous improvement of robots' autonomous intelligence, mobile robots, with their high mobility, convenient deployment, and low cost, are widely used in civilian fields such as emergency communication, detection and rescue, and agriculture, as well as in fields such as reconnaissance and surveillance, and target tracking. Among these applications, mobile robot target tracking has attracted increasing attention from researchers due to its important role in maintaining public safety.

[0003] Maneuvering target tracking refers to the process by which a mobile robot uses sensors to observe a moving target, perceive its environment, track the target, and maintain a certain tracking distance. When the environment and the target's motion are uncertain, performing target tracking tasks requires the robot to possess autonomous perception, decision-making, and real-time planning capabilities to achieve high-precision tracking of maneuvering targets.

[0004] Due to the uncertainty of target motion, the limited perception capabilities of airborne cameras, and constraints on control, adaptive strategies generated through online interaction between the agent and the environment are needed to dynamically respond to environmental changes and autonomously make tracking decisions based on real-time perception. However, the maneuverability of existing individual agents is limited, and their information acquisition and processing capabilities are insufficient to meet the demands for high tracking success rates. In dynamic target tracking tasks, the constantly changing motion of targets presents significant challenges to the perception and control systems of mobile robots. This often results in low target tracking and formation control accuracy, poor strategy robustness, and inadequate obstacle avoidance in collaborative target tracking tasks in complex dynamic environments, failing to guarantee that the robot can accurately and quickly respond to changes in the motion of the tracked target. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive target tracking formation control method for multi-agent systems based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an adaptive target tracking formation control method for a multi-agent system based on reinforcement learning, comprising the following steps:

[0007] Step 1: Scenario Design; Set up an environment containing N intelligent agents, 1 mobile target, and M obstacles. During movement, each intelligent agent makes real-time decisions based on communication information with other intelligent agents and its own observation of the current unknown environment.

[0008] Step 2: Constructing the Constraint Analysis Model; Transform the FCCATT problem into an optimization problem with an objective function and constraints, and construct the objective function and constraints.

[0009] Step 3: Agent strategy design; design the observation space, action space and reward function. The observation space consists of four parts. The action space is a continuous action space, including linear velocity and angular velocity. The reward function is designed for multiple optimization objectives such as cooperative target tracking, formation control and obstacle avoidance.

[0010] Step 4: Experimental Design; Design the training batch size, experience pool capacity, discount factor, policy network learning rate, maximum number of steps in each round, soft update factor, number of training samples, and model parameters for the policy and value network optimizers.

[0011] Step 5: Agent Training; A two-stage training process is adopted. In the first stage, the MPR-DDPG algorithm is used to train the obstacle avoidance and tracking capabilities of a single agent in a scene with obstacles, and the trained policy and value network parameters are saved. In the second stage, the number of agents is increased, and training continues in a multi-agent collaborative target tracking scenario. The model parameters saved in the first stage are loaded, the network parameters are updated, and the models of any two agents are swapped at the beginning of the next round.

[0012] Step 6: Design of Adaptive Target Tracking Formation Control Framework; Propose the ATTFCF framework, which consists of a hierarchical control strategy and a hybrid switching control strategy. The hierarchical control strategy and the hybrid switching strategy take effect sequentially at each time step of the agent's interaction with the environment.

[0013] Preferably, in step one, multiple intelligent agents quickly form a formation from the starting point, performing target tracking tasks while maintaining the desired formation.

[0014] Preferably, in step two, the continuous motion model of the mobile robot can be represented as:

[0015]

[0016] Where, x t and y t Represents the coordinates of the agent in a two-dimensional planar coordinate system. Represents the yaw angle, v t Let ω be the linear velocity at time t. t Let be the angular velocity at time t;

[0017] The position update process of the agent at time t can be represented as:

[0018]

[0019] Where Δt is the simulation step size, and considering the motion characteristics of the mobile robot, its linear velocity and angular velocity are reasonably assumed to be within a certain range. t ∈[v min ,v max ],ω t ∈[ω min ,ω max ].

[0020] Preferably, in step two, the objective function consists of the time required for the leader to reach the specified tracking distance of the moving target, the accumulated error in maintaining the formation, and the formation error during fixed-distance tracking, forming a multi-objective function. The optimization objective of the entire FCCATT problem can be defined as:

[0021]

[0022] The constraint conditions are defined as follows:

[0023]

[0024] in, This represents the position of the follower at time t. Let H be the leader's position at time t, t2 be the task completion time, and H be the position of the leader at time t. t It is an expected offset vector, representing the expected offset positional relationship of the follower relative to the leader, π. 1:N =[π 1 ,π 2 ,...π N ] represents the policy of all agents; N and M are the number of agents and obstacles in the scene, respectively, and r i The size of the entity radius. Represents the safe radius between the follower and the leader or obstacle. Let t be the position of agent i at time t. Let O be the location of obstacle o at time t. This represents the joint state observation of all agents at time t-Δt.

[0025] Preferably, in step three, in the FCCATT scenario, the observation space o of each agent at time t... t It consists of four parts: lidar scanning data, the agent's observation information of the target, the agent's own motion information, and the communication information between the agent and other agents; that is, the observation space is:

[0026]

[0027] in, For LiDAR scanning data, It can detect obstacles and other intelligent agents; The observation information of the agent on the target consists of the relative distance D and the relative azimuth angle φ. This refers to the motion information of the intelligent agent itself, consisting of its current position, linear velocity, and angular velocity. The communication information between intelligent agents consists of obstacle information, target information, and self-information from the observation data of other intelligent agents. The communication information received by intelligent agent i from other intelligent agents is as follows: in Each agent broadcasts its status information through a communication module while simultaneously receiving communication data from other agents.

[0028] Preferably, the action space in step three is:

[0029]

[0030] Let L and F represent the linear velocity and angular velocity of agent i at time t, respectively, and use superscripts L, F, and O to represent the leader, follower, and obstacle, respectively. Let be the observation of the follower at time t. Let be the leader's observation at time t.

[0031] Preferably, in step three, for the cooperative target selection problem in cluster formation tracking, a cooperative reward is designed to minimize the total movement distance of the formation. The cooperative tracking target selection reward among agents is designed as follows:

[0032]

[0033] Where N is the cluster size, i.e. the number of agents; The collaborative reward for goal allocation to followers is calculated as follows:

[0034]

[0035] in, This represents the distance between the current target point T1 and its nearest follower.

[0036] Preferably, in step six, the hierarchical control strategy includes an upper decision-making layer and a lower execution layer. The decision-making layer determines the desired formation tracking position of the leader and followers based on the Leader-Follower method and constructs the desired formation. At the same time, based on the hybrid switching control strategy, different control modes are provided to the leader and followers according to different strategy switching conditions. The ORCA method is used for agent collision speed correction in dangerous situations.

[0037] Preferably, in step six, the leader and followers at the execution layer make decisions based on the control mode determined by the decision-making layer, and use the outputs of the two strategies trained by the MPR-DDPG algorithm or the control quantities corrected by the ORCA method for cluster collaborative formation tracking and obstacle avoidance.

[0038] Preferably, in step six, the cluster in the ATTFCF framework is a fully connected network, and the hybrid switching control strategy determines the control mode that the agent should adopt at the current moment based on global information; when two dynamic entities avoid each other, they can obtain the other party's position and speed information, and both parties will actively take avoidance actions. All agents in the cluster are isomorphic, and agents can communicate with other agents through communication devices.

[0039] The technical effects and advantages of this invention are as follows:

[0040] (1) This invention includes the steps of scenario design, constraint analysis model construction, agent policy design, experiment design, agent training, and adaptive target tracking formation control framework design. It integrates target tracking and formation control problems and uses the MPR-DDPG algorithm for solution. At the same time, it designs a multi-agent target tracking formation control framework to solve the problems of low accuracy and poor policy security in multi-agent cooperative target tracking and formation control. The ATTFCF framework performs well in terms of multi-agent cooperative target tracking accuracy, formation control accuracy and obstacle avoidance security. Its overall performance is better than the MADDPG policy method. It improves the problems of low accuracy, poor policy robustness and poor obstacle avoidance in target tracking and formation control, and can be applied to cooperative target tracking tasks with different numbers of agents.

[0041] (2) This invention integrates multi-agent target tracking with formation control problem and designs the ATTFCF framework for multi-agent target tracking formation control. During the execution of task decision, the framework takes effect sequentially at each time step of the interaction between the agent and the environment. The adaptive target tracking formation control framework consists of a hierarchical control strategy and a hybrid switching control strategy. By combining the formation control capability of the Leader-Follower method with the obstacle avoidance tracking and cooperative target selection capability of the reinforcement learning algorithm, the adaptive cooperative formation tracking and obstacle avoidance of multiple agents in complex and dynamic scenarios is realized.

[0042] (3) The hierarchical control strategy in the adaptive target tracking formation control framework design of the present invention includes an upper decision layer and a lower execution layer. In the decision layer, the desired formation tracking position of the leader and followers is determined based on the Leader-Follower method according to the target tracking task, and the desired formation is constructed. At the same time, based on the hybrid switching control strategy, different control modes are provided to the leader and followers according to different strategy switching conditions. In the execution layer, the leader and followers make decisions according to the control mode determined by the decision layer, and use the two strategies to output or the control quantity after being corrected by the ORCA method to realize the cluster collaborative formation tracking and obstacle avoidance. The hybrid switching control strategy is used to select the appropriate control mode for each agent in scenarios with different levels of complexity. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the adaptive target tracking formation control framework of the present invention.

[0044] Figure 2 This is a schematic diagram of the ATTFCF framework switching control strategy of the present invention.

[0045] Figure 3 This is a schematic diagram of the task scenario of the present invention.

[0046] Figure 4 This is a schematic diagram of the two-stage training process of the MPR-DDPG algorithm of the present invention.

[0047] Figure 5 This is a schematic diagram illustrating the intelligent agent formation tracking effect of the present invention.

[0048] Figure 6 This is a schematic diagram illustrating the tracking distance error of the intelligent agent formation according to the present invention.

[0049] Figure 7 This is a schematic diagram illustrating the specific control of intelligent agent formation tracking according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention provides, for example Figure 1-7 The method shown is an adaptive target tracking formation control method for a multi-agent system based on reinforcement learning, comprising:

[0052] Step 1: Scenario Design; Set the environment to include N agents, 1 moving target, and M obstacles. During the movement of each agent, multiple agents quickly form a formation from the starting point. While maintaining the desired formation, they perform target tracking tasks to avoid collisions with other agents or obstacles. Based on communication information with other agents and their own observation of the current unknown environment, they make decisions in real time to achieve collaborative target tracking, maintaining formation, and obstacle avoidance.

[0053] As attached Figure 1-3 As shown, the FCCATT problem can be viewed as a partially observable sequence decision problem for a single agent performing a maneuvering target tracking task on a two-dimensional plane with obstacles. At each time step t, the agent is located at position p. t Received observations from the environment. t Calculate action a t In order to control itself without hitting obstacles k (0≤k≤M) or other agents collide with the target to track the specified target. For each agent, other agents can be regarded as dynamic obstacles with uncertainty, and the agent communicates with other agents at each time step.

[0054] Step 2: Constraint Analysis Model Construction; The FCCATT problem is transformed into an optimization problem with an objective function and constraints. The objective function and constraints are constructed, including collision avoidance constraints and agent kinematic constraints. The objective function consists of the time required for the leader to reach the specified tracking distance of the moving target, the accumulated error in maintaining formation, and the formation error during fixed-distance tracking, forming a multi-objective function. The optimization objective of the entire FCCATT problem can be defined as:

[0055]

[0056] The constraint conditions are defined as follows:

[0057]

[0058] in, This represents the position of the follower at time t. Let H be the leader's position at time t, t2 be the task completion time, and H be the position of the leader at time t. t It is an expected offset vector, representing the expected offset positional relationship of the follower relative to the leader, π. 1:N =[π 1 ,π 2 ,...π N [] represents the policy of all agents. N and M are the number of agents and obstacles in the scene, respectively, and r i The size of the entity radius. Represents the safe radius between the follower and the leader or obstacle. Let t be the position of agent i at time t. Let O be the location of obstacle o at time t. The equation represents the joint state observation of all agents at time t-Δt. The first two equations represent collision avoidance constraints, and the third equation represents the kinematic constraints of the agents.

[0059] To simplify the analysis, this design assumes that the mobile robot moves within a two-dimensional plane and is equipped with a lidar system capable of acquiring obstacle information within a certain angle and distance range. The continuous motion model of the mobile robot can be represented as follows:

[0060]

[0061] Where, x t and y t Represents the coordinates of the agent in a two-dimensional planar coordinate system. Represents the yaw angle, v t Let ω be the linear velocity at time t. t Let be the angular velocity at time t;

[0062] The position update process of the agent at time t can be represented as:

[0063]

[0064] Where Δt is the simulation step size. Considering the motion characteristics of the mobile robot, its linear velocity and angular velocity are reasonably assumed to be within a certain range. t ∈[v min ,v max ],ω t ∈[ω min ,ω max ].

[0065] Step 3: Agent strategy design; design the observation space, action space, and reward function. The observation space consists of four parts, the action space is a continuous action space, including linear velocity and angular velocity, and the reward function is designed for multiple optimization objectives such as cooperative target tracking, formation control, and obstacle avoidance.

[0066] In the FCCATT scenario, the observation space o of each agent at time t t It consists of four parts: lidar scanning data, the agent's observation information of the target, the agent's own motion information, and the communication information between the agent and other agents; that is, the observation space is:

[0067]

[0068] in, For LiDAR scanning data, It can detect obstacles and other intelligent agents; The observation information of the agent on the target consists of the relative distance D and the relative azimuth angle φ. Since the PR-DDPG algorithm uses LSTM to extract features from the target's historical trajectory, it can effectively improve the target tracking accuracy. Therefore, in the observation space design of the MPR-DDPG algorithm, LSTM is also used to model the target's historical trajectory. This refers to the motion information of the intelligent agent itself, consisting of its current position, linear velocity, and angular velocity. The communication information between intelligent agents consists of obstacle information, target information, and self-information from the observation data of other intelligent agents. The communication information received by intelligent agent i from other intelligent agents is as follows: in, Other intelligent agents follow the same pattern. Each agent broadcasts its status information through a communication module, while simultaneously receiving communication data from other agents.

[0069] The agent uses a continuous action space, which is:

[0070]

[0071] Let L and F represent the linear velocity and angular velocity of agent i at time t, respectively, and use superscripts L, F, and O to represent the leader, follower, and obstacle, respectively. Let be the observation of the follower at time t. Based on the leader's observations at time t, the control stability and target tracking accuracy of the mobile robot are improved through the design of the action space.

[0072] When constructing the state input, the communication information sequence between agents is sequentially input into the LSTM network to generate all the encoded communication information states; at the same time, LSTM is used to extract features from the target's historical motion trajectory, and then it is concatenated with obstacle observation information and the agent's own motion information, and finally input into the fully connected neural network;

[0073] The policy network consists of three fully connected layers with 128, 128, and 64 neurons respectively. The hidden layer variables of the LSTM network for extracting features from variable-length communication information and the LSTM network for extracting features from the target's historical trajectory are the encoded fixed-length feature vectors, both with a length of 32. After the policy network completes the forward computation, it will produce a 2D continuous action output. The structure of the value network is basically the same as that of the policy network, but the input of the first hidden layer of the fully connected network includes 2D agent action information.

[0074] Solutions to the FCCATT problem involve multiple optimization objectives, including cooperative target tracking, formation control, and obstacle avoidance. For the cooperative target selection problem in swarm formation tracking, a cooperative reward is designed to minimize the total movement distance of the formed formation. The cooperative target selection reward among agents is designed as follows:

[0075]

[0076] Where N is the cluster size, i.e. the number of agents; The collaborative reward for goal allocation to followers is calculated as follows:

[0077]

[0078] in, This represents the distance between the current target point T1 and its nearest follower.

[0079] Simultaneously, a penalty function is designed. When multiple agents jointly perform a target tracking task, they need to quickly form a formation while minimizing the total movement distance. At the beginning of the task, a leader is first identified and assigned a tracking target. Simultaneously, the Leader-Follower method is used to generate the desired formation based on the leader's position, i.e., the desired tracking position of the followers. However, for the question of how each follower determines its tracking target, instead of assignment, the aforementioned collaborative reward function is used. This allows multiple followers to collaboratively determine the target allocation. The purpose of this collaborative reward is to assign a target to each follower agent given a known number and position of follower tracking target points, minimizing the minimum distance from all follower agents to their tracking positions. The allocation of tracking targets requires collaboration among multiple followers to minimize the total movement distance of all followers when the formation is formed. This reward is shared among all followers, and the collaborative reward value is the same at each time step. However, the leader assigns tracking targets, and its reward function does not include a collaborative reward term.

[0080] In addition, each agent receives a reward for maintaining the formation at time t:

[0081]

[0082] Wherein, the formation error at time t is defined as c1 and c2 are constant values ​​used to determine the desired target tracking distance. Let t be the position of the follower. Let H be the position of the leader at time t. t Let t be the desired formation.

[0083] To ensure that no collisions occur when the cluster is performing tasks, the obstacle avoidance reward is designed as follows:

[0084]

[0085] in, p is the minimum distance between the follower and the leader or obstacle. o1 p represents the position of obstacle o1 L This represents the leader's position. Because the obstacle avoidance reward function is designed to account for the influence of other agents, the reward implicitly includes a penalty for assigning the same desired follower formation tracking position to multiple follower agents.

[0086] Step 4: Experimental Design; Design the training batch size, experience pool capacity, discount factor, policy network learning rate, maximum number of steps per round, soft update factor, number of training samples, and model parameters for the policy and value network optimizers; At the beginning of each round, the positions of the agent and obstacles are randomly generated, but to simulate real-world conditions, the relative distance between agents does not exceed 50m. Simultaneously, to increase the complexity of the task scenario and stimulate the agent to learn obstacle avoidance behavior, obstacles are randomly initialized near the trajectory of the moving target. The algorithm hyperparameters are shown in the table below:

[0087] Multi-agent reinforcement learning algorithm parameter configuration

[0088]

[0089] Step 5: Agent Training; A two-stage training process is adopted. In the first stage, the MPR-DDPG algorithm is used to train the obstacle avoidance and tracking capabilities of a single agent in a scene with obstacles. After the learning process is completed, the trained policy and value network parameters are saved. In the second stage, the number of agents is increased, and training continues in a multi-agent collaborative target tracking scenario. The model parameters saved in the first stage are loaded, the network parameters are updated, and at the beginning of the next round, the models of any two agents are swapped to reduce the impact of the initial position settings of the agents on the policy performance, avoid the solidification of the training mode, and improve the generalization of the policy model during the learning process.

[0090] The two-stage training process of the MPR-DDPG algorithm is as follows:

[0091]

[0092] Step Six: Adaptive Target Tracking Formation Control Framework Design; An ATTFCF framework consisting of a hierarchical control strategy and a hybrid switching control strategy is proposed. The hierarchical control strategy includes an upper decision layer and a lower execution layer. The decision layer determines the desired formation tracking positions of the leader and followers based on the Leader-Follower method and constructs the desired formation. Simultaneously, based on the hybrid switching control strategy, different control modes are provided to the leader and followers according to different strategy switching conditions. The ORCA method is used for agent collision velocity correction in dangerous situations. In the execution layer, the leader and followers make decisions according to the control modes determined by the decision layer. The outputs of the two strategies trained using the MPR-DDPG algorithm or the control quantities corrected by the ORCA method are used for cluster cooperative formation tracking and obstacle avoidance. The hierarchical control strategy and the hybrid switching strategy take effect sequentially at each time step of the agent's interaction with the environment.

[0093] In the ATTFCF framework, the cluster is a fully connected network. The hybrid switching control strategy determines the control mode that the agent should adopt at the current moment based on global information. When two dynamic entities avoid each other, they can obtain each other's position and speed information, and both parties will take the initiative to avoid each other. All agents in the cluster are isomorphic. Agents can communicate with other agents through communication devices, but the communication range is limited.

[0094] This framework models multi-agent cooperative target tracking as a combination of hierarchical control and hybrid switching control strategies. First, the decision layer in the hierarchical control strategy uses the Leader-Follower method to generate the tracking positions of each agent, constructs the desired formation, and selects different targeted control modes for the agents according to the hybrid switching control strategy. In the execution layer, each agent makes cooperative target selection or obstacle avoidance tracking decisions based on the control mode it is currently using. Simulation results show that the ATTFCF framework performs excellently in terms of multi-agent cooperative target tracking accuracy, formation control accuracy, and obstacle avoidance safety, outperforming the MADDPG strategy method overall, and is applicable to cooperative target tracking tasks with varying numbers of agents.

[0095] To further improve the accuracy of target tracking, formation control, and obstacle avoidance safety when the cluster is collaboratively performing target tracking tasks, a hybrid switching control strategy is designed. Under different complexity scenarios, an appropriate control mode is selected for each agent. At each step of the agent's interaction with the environment, the control mode of the leader and followers at the current moment is determined according to the strategy switching conditions. The scenarios can be divided into simple scenarios, obstacle avoidance scenarios, and dynamic obstacle avoidance scenarios.

[0096] 1) Leader Control Strategies

[0097] Scenario 1: Hazard level DL =0

[0098] in, This represents the minimum distance between the leader and all other followers. d represents the minimum distance between the leader and all obstacles. safe This represents a safe distance. In this scenario, the leader is considered to be in a simple, safe situation. The above conditions indicate that the leader is far from other followers and obstacles. Under these circumstances, the decision-making level sets strategy model 1 as the leader's control strategy, controlling the leader to achieve precise tracking of the target.

[0099] Scenario 2: and Hazard level D L =1

[0100] in, This represents the minimum distance from the leader to all static obstacles. At this point, the leader's distance from the static obstacle is less than the safe distance, and the distance to other followers is greater than the safe distance. This indicates that the leader is at risk of colliding with the obstacle and is in a complex situation that requires obstacle avoidance behavior. In this case, strategy model 2 is used to control the leader.

[0101] Scenario 3: or Hazard level D L =2

[0102] in, This represents the minimum distance from the leader to all dynamic obstacles. In this case, the leader's distance from a dynamic obstacle is less than the safe distance, or the leader's distance from other followers is less than the safe distance. At this time, the danger level of the leader's environment is the highest, and obstacle avoidance behavior is required for dynamic objects. A precise control strategy is adopted to prevent collisions. Considering that strategy model 2 may not perform well in handling dynamic obstacle avoidance, the ORCA method is used for correction. During correction, the speed output by strategy model 2 is used as the preferred speed input for ORCA, and the collision-free speed control quantity with the smallest deviation between the output and input speed is obtained.

[0103] 2) Follower control strategy

[0104] Scenario 1: Hazard level D L =0

[0105] in, This represents the minimum distance between the follower and all other agents. d represents the minimum distance of the follower from all obstacles. safeThis represents the minimum safe distance between the follower and other units. In this simple scenario where the follower is considered relatively safe, the primary task is to maintain formation. The follower's desired target point is calculated using the Leader-Follower method, and strategy model 1 is used to control the follower.

[0106] Scenario 2: and Hazard level D L =1

[0107] At this point, the distance between the follower and other agents is greater than the safe distance, but the minimum distance to the static obstacle is less than the safe distance, which puts it in a situation where it may collide with the static obstacle. In this case, the Leader-Follower method calculates the position of the target that the follower expects to track, and the follower makes obstacle avoidance behavior based on policy model 2 to avoid colliding with the static obstacle.

[0108] Scenario 3: or Hazard level D L =2

[0109] At this point, the minimum distance between the follower and other agents or the minimum distance to dynamic obstacles is less than the safe distance, indicating a high level of danger. The follower needs to maintain its formation while avoiding obstacles. Strategy Model 2 is used as the control mode for the current moment. However, unlike Case 2, the expected tracking target position of the current follower has changed. In Case 2, the tracking target is calculated using the Leader-Follower method. The distance to the target point is still calculated using the Leader-Follower method, but the calculation method for the tracking target's orientation has changed. All agents on the side of the perpendicular line connecting the current follower and the leader that is closer to the leader are grouped into a set. The center coordinates of the elements in the set are calculated and used as virtual tracking target points to calculate the expected target orientation. The condition for determining the elements in the set is:

[0110] cos<|p L -p F |,|p i -p F |>0, (i≠F, i=1,2,L,N-1)

[0111] The coordinates of the center point of the elements in the set are:

[0112]

[0113] The purpose of this approach is to reduce the risk of collision and improve formation safety when followers are close to dynamic obstacles by shrinking their formation towards the center of the cluster. Simultaneously, the ORCA method is used to correct the velocity output of policy model 2. The hybrid switching control strategy encompasses two policy models trained based on the MPR-DDPG algorithm.

[0114] The ATTFCF framework, which combines hierarchical control strategies and hybrid handover control strategies, follows this process:

[0115]

[0116] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adaptive target tracking formation control of a multi-agent system based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Scenario Design; Set up an environment containing N intelligent agents, 1 mobile target, and M obstacles. During movement, each intelligent agent makes real-time decisions based on communication information with other intelligent agents and its own observation of the current unknown environment. Step 2: Constructing the Constraint Analysis Model; Transform the FCCATT problem into an optimization problem with an objective function and constraints, and construct the objective function and constraints. Step 3: Agent strategy design; The design includes an observation space, an action space, and a reward function. The observation space consists of four parts, the action space is a continuous action space, including linear velocity and angular velocity, and the reward function is designed for multiple optimization objectives such as cooperative target tracking, formation control, and obstacle avoidance. Step 4: Experimental Design; Design the training batch size, experience pool capacity, discount factor, policy network learning rate, maximum number of steps in each round, soft update factor, number of training samples, and model parameters for the policy and value network optimizers. Step 5: Agent Training; A two-stage training process is adopted. In the first stage, the MPR-DDPG algorithm is used to train the obstacle avoidance and tracking capabilities of a single agent in a scene with obstacles, and the trained policy and value network parameters are saved. In the second stage, the number of agents is increased, and training continues in a multi-agent collaborative target tracking scenario. The model parameters saved in the first stage are loaded, the network parameters are updated, and the models of any two agents are swapped at the beginning of the next round. Step 6: Design of Adaptive Target Tracking Formation Control Framework; Propose the ATTFCF framework, which consists of a hierarchical control strategy and a hybrid switching control strategy. The hierarchical control strategy and the hybrid switching strategy take effect sequentially at each time step of the agent's interaction with the environment.

2. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step one, multiple agents quickly form a formation from the starting point and perform target tracking tasks while maintaining the desired formation.

3. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step two, the continuous motion model of the mobile robot can be represented as: Where, x t and y t Represents the coordinates of the agent in a two-dimensional planar coordinate system. Represents the yaw angle, v t Let ω be the linear velocity at time t. t Let be the angular velocity at time t; The position update process of the agent at time t can be represented as: Where Δt is the simulation step size, and considering the motion characteristics of the mobile robot, its linear velocity and angular velocity are reasonably assumed to be within a certain range. t ∈[v min ,v max ],ω t ∈[ω min ,ω max ].

4. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step two, the objective function consists of the time required for the leader to reach the specified tracking distance of the moving target, the accumulated error in maintaining the formation, and the formation error during fixed-distance tracking, forming a multi-objective function. The optimization objective of the entire FCCATT problem can be defined as: The constraint conditions are defined as follows: in, This represents the position of the follower at time t. Let H be the leader's position at time t, t2 be the task completion time, and H be the position of the leader at time t. t It is an expected offset vector, representing the expected offset positional relationship of the follower relative to the leader, π. 1:N =[π 1 ,π 2 ,...π N ] represents the policy of all agents; N and M are the number of agents and obstacles in the scene, respectively, and r i The size of the entity radius. Represents the safe radius between the follower and the leader or obstacle. Let t be the position of agent i at time t. Let O be the location of obstacle o at time t. This represents the joint state observation of all agents at time t-Δt.

5. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step three, in the FCCATT scenario, the observation space o of each agent at time t. t It consists of four parts: lidar scanning data, the agent's observation information of the target, the agent's own motion information, and the communication information between the agent and other agents; that is, the observation space is: in, For LiDAR scanning data, It can detect obstacles and other intelligent agents; The observation information of the agent on the target consists of the relative distance D and the relative azimuth angle φ. This refers to the motion information of the intelligent agent itself, consisting of its current position, linear velocity, and angular velocity. The communication information between intelligent agents consists of obstacle information, target information, and self-information from the observation data of other intelligent agents. The communication information received by intelligent agent i from other intelligent agents is as follows: in Each agent broadcasts its status information through a communication module while simultaneously receiving communication data from other agents.

6. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, The action space in step three: Let L and F represent the linear velocity and angular velocity of agent i at time t, respectively, and use superscripts L, F, and O to represent the leader, follower, and obstacle, respectively. Let be the observation of the follower at time t. Let be the leader's observation at time t.

7. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step three, for the cooperative target selection problem in cluster formation tracking, a cooperative reward is designed to minimize the total movement distance of the formation. The cooperative tracking target selection reward among agents is designed as follows: Where N is the cluster size, i.e. the number of agents; The collaborative reward for goal allocation to followers is calculated as follows: in, This represents the distance between the current target point T1 and its nearest follower.

8. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step six, the hierarchical control strategy includes an upper decision-making layer and a lower execution layer. The decision-making layer determines the desired formation tracking position of the leader and followers based on the Leader-Follower method and constructs the desired formation. At the same time, based on the hybrid switching control strategy, different control modes are provided to the leader and followers according to different strategy switching conditions. The ORCA method is used for agent collision speed correction in dangerous situations.

9. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step six, the leader and followers at the execution layer make decisions based on the control mode determined by the decision-making layer. The outputs of the two strategies trained by the MPR-DDPG algorithm or the control quantities corrected by the ORCA method are used for cluster collaborative formation tracking and obstacle avoidance.

10. The adaptive target tracking formation control method for a multi-agent system based on reinforcement learning according to claim 1, characterized in that, In step six, the cluster in the ATTFCF framework is a fully connected network. The hybrid switching control strategy determines the control mode that the agent should adopt at the current moment based on global information. When two dynamic entities avoid each other, they can obtain the other party's position and speed information, and both parties will take the initiative to avoid each other. All agents in the cluster are isomorphic, and agents can communicate with other agents through communication devices.

Images