A method for implementing a siege by using obstacles based on multi-agent deep reinforcement learning
By using the multi-agent deep reinforcement learning algorithm MATD3, an obstacle reward function was designed and a multi-robot encirclement strategy was optimized, which solved the problem of insufficient encirclement performance in obstacle scenarios and achieved a more efficient encirclement effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DALIAN MARITIME UNIVERSITY
- Filing Date
- 2023-11-02
- Publication Date
- 2026-05-19
AI Technical Summary
Most existing multi-robot collaborative trapping methods are designed only for obstacle-free scenarios and fail to effectively utilize obstacles to improve trapping performance.
We employ the multi-agent deep reinforcement learning algorithm MATD3 and design a reward function based on state transitions. We use obstacles as reward factors and divide the state into four states: search, entrapment, transfer, and capture, to optimize the robot's entrapment strategy.
It improved the success rate and efficiency of the capture operation, and reduced the time required for a successful capture.
Smart Images
Figure CN117400248B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-robot cooperative trapping technology, and more particularly to a trapping method based on multi-agent deep reinforcement learning using obstacles. Background Technology
[0002] Multi-robot cooperative encirclement refers to the use of multiple pursuit robots to surround and capture intruders or escapees. It has a wide range of important applications and has therefore been a research hotspot. Research methods include model-based differential game theory, optimal control, and geometric programming. Recent research on deep reinforcement learning algorithms does not rely on specific robot motion models; instead, it acquires encirclement strategies through offline training in a defined task scenario.
[0003] However, most existing methods and strategies are designed only for obstacle-free scenarios, while for obstacles in the task scenario, the robot simply performs avoidance maneuvers. Considering that the presence of obstacles can block the escapee's movement path and reduce its range of activity, the rational and effective use of obstacles can improve the capture performance. Summary of the Invention
[0004] Based on the aforementioned problem of multi-robot cooperative encirclement in obstacle-filled scenarios, this invention provides a method for encirclement using obstacles based on multi-agent deep reinforcement learning. This invention considers encirclement scenarios with static obstacles and utilizes the MATD3 multi-agent deep reinforcement learning algorithm to design the robot encirclement strategy. By incorporating obstacles as a crucial factor in the reward function design, it effectively improves the success rate and efficiency of encirclement.
[0005] The technical means employed in this invention are as follows:
[0006] A method for encirclement and trapping using obstacles based on multi-agent deep reinforcement learning includes:
[0007] S1, constructing includes N A pursuit robot , An escape robot E and a static obstacle O The chase scene;
[0008] S2. Based on the constructed pursuit and escape scenario, construct the MATD3 framework of a multi-agent deep reinforcement learning algorithm;
[0009] S3. Based on the constructed multi-agent deep reinforcement learning algorithm MATD3 framework, design a reward function for a multi-robot encirclement strategy based on state transition;
[0010] S4. Combine the designed reward function with the MATD3 algorithm to obtain a multi-robot cooperative trapping strategy based on multi-agent deep reinforcement learning MATD3.
[0011] Further, step S1 specifically includes:
[0012] S11. Select a two-dimensional finite closed area as the scene, represent obstacles with straight lines of fixed length, and assume that the obstacles only block the robot's path of travel, but do not obstruct the observation line of sight.
[0013] S12, Use and They represent obstacles. O The two endpoints, using Indicates a pursuit robot and escape robots E The intersection of the line connecting the two obstacles with the obstacle;
[0014] S13. Construct the motion equations for each robot as follows:
[0015]
[0016] in, It's a robot. i Position coordinates in a two-dimensional planar scene It is linear velocity. It is the direction angle of the robot's movement speed. It is acceleration; the maximum speed of all pursuit robots is limited to... The maximum speed of the escape robot is .
[0017] Further, in step S2, a multi-agent dual-delay deep policy determination gradient algorithm is used. The constructed multi-agent deep reinforcement learning algorithm MATD3 framework includes six networks for each agent: the current Actor network, the target Actor network, two current Critic networks, and two target Critic networks, wherein:
[0018] The Actor network employs a distributed execution approach, taking its own state information as input and outputting the decision action; the two Critic networks employ a centralized training approach, taking their own state and action information, as well as the state and action information of all other agents, as input, and outputting two Q-values that are close to the truth.
[0019] Furthermore, the learning objective of each agent is to maximize the expected reward, i.e., the Q value, and the network structure and training method in each agent adopt the settings in the Deep Deterministic Policy Gradient Algorithm 3 (TD3).
[0020] Further, step S3 specifically includes:
[0021] S31, Design an obstacle avoidance reward ,as follows:
[0022]
[0023] in, and Intersection points To the left and right ends of the obstacle and distance, A collection of all robots without obstructions; each pursuit robot Determine if it is related to the escape robot E Whether there are obstacles in between, i.e., determining the pursuit robot With escape robots E Does the line connecting the positions intersect with the obstacles? If there are obstacles, the pursuit robot cannot participate in the encirclement operation and must therefore bypass the obstacles.
[0024] S32, Designing Robot Collision Avoidance Rewards ,as follows:
[0025]
[0026] in, To set a safe distance, It is a constant. For robots i To the obstacle O The minimum distance, For robots i The minimum distance to the scene boundary. For robots i and j The distance between them; the designed robot collision avoidance reward is used to guide the robot to avoid collisions with obstacles and the boundaries of the motion scene, and to prevent robots from colliding with each other;
[0027] S33, Design Encirclement Rewards ,as follows:
[0028] In search mode, let This enables the pursuit robot to learn to use obstacles to form an encirclement around the escape robot;
[0029] In a siege, This allows the pursuit robot to continuously surround the escape robot using obstacles;
[0030] In the transition state, let This enables the pursuit robots to learn to narrow down the encirclement;
[0031] In the state of arrest, This causes the pursuit robot to move toward the escaped robot and capture it.
[0032] S34, Pursuit Robot In the t The overall reward function for each time step is the designed obstacle avoidance reward. Robot collision avoidance reward and capture rewards The sum of the three parts is expressed as:
[0033] .
[0034] Furthermore, in step S33, the designed capture reward... The game is divided into four states: search, siege, transfer, and capture, each with its own reward function. , , , Each state corresponds to an easier subtask and a corresponding reward function. When the conditions for each state are met, the corresponding reward function is applied. The conditions for each state and the corresponding reward function are designed as follows:
[0035] S331. Search Status: In the search status, the escape robot is outside the encirclement formed by the pursuing robot and obstacles, determined by the following conditions:
[0036]
[0037] in, S Let represent the area of the largest convex region enclosed by the index points. The reward function for the search state is designed as follows:
[0038]
[0039] in, Indicates the first i From a pursuit robot to an escape robot E distance, This item is used to punish the pursuit robot for moving away from the escape robot;
[0040] S332, Encirclement State: In the encirclement state, the escape robot is located within a polygonal area formed by the pursuing robot and obstacles. The robots are controlled in a counter-clockwise order from 1 to... N Number them when season ;when season Based on the relative positions of each robot and the obstacles, it is determined whether the robot remains trapped, and a corresponding reward function is applied, which is divided into the following four cases:
[0041] Scenario 1: For those who meet the conditions robots ,robot The left and right sides are not connected to any obstacles, satisfying the following condition:
[0042]
[0043] in, The maximum speed ratio, K The reward function is designed to represent the maximum number of time steps required to achieve a siege.
[0044]
[0045] Scenario 2: For those who meet the conditions robots ,robot The right side is connected to an obstacle, while the left side is other pursuit robots, satisfying the following conditions:
[0046]
[0047] in, Representative robot The center of the circle of Apollonius formed by the escapee and the escapee A i To the obstacle The minimum distance, the center of the circle A i The coordinates are ,in and respectively, the pursuit robot and escape robots E Position coordinates; The radius of the Apollonius circle is defined as... This condition indicates that the robot To form an encirclement around the robot on its left and defend the gap between it and the obstacle on its right, the corresponding reward function is designed as follows:
[0048]
[0049] Scenario 3: For those who meet the conditions robots ,robot The right side connects to other pursuit robots while the left side is an obstacle, satisfying the following conditions:
[0050]
[0051] That is, robots To form an encirclement around the robot on its right and defend the gap between it and the obstacle on its left, the corresponding reward function is designed as follows:
[0052]
[0053] Scenario 4: For those who meet the conditions robots Its position is between the obstacle and the escaped robot, so it is impossible to use the obstacle to surround and capture the escaped robot. Therefore, the following reward function is set:
[0054]
[0055] in, The reward is the radian value of the angle formed by the points indicated by their subscripts; this reward is intended to improve the robot's performance. The location can be transformed into one of the three situations mentioned above, so as to make full use of obstacles for encirclement and capture.
[0056] S333, Transition State: The transition state is the state that shrinks from the siege state to the capture state, and satisfies the following conditions:
[0057]
[0058] in, For a time step, This represents the maximum distance at which the pursuing robot can initiate a capture action against the escaped robot. In this state, the pursuing robot continuously narrows the encirclement, trapping the escapee near the obstacle until the subsequent capture state is reached. During this period, adjacent pursuing robots continuously adjust the distance between them to make them as equal as possible, preventing the escapee from escaping between two adjacent pursuing robots with a large gap. The reward function corresponding to the transition state is designed as follows:
[0059]
[0060] in, It allows the pursuit robots to narrow down the encirclement. It forces the escapee to get closer to the obstacle. This is to ensure that the distance between adjacent robots is equal, defined as ,in ;
[0061] S334. Capture State: In the capture state, regardless of the action chosen by the escape robot, as long as the pursuing robot moves towards the escape robot at maximum speed, it can be captured. When in the capture state, each robot must meet the following conditions:
[0062]
[0063] The corresponding reward function is designed as follows:
[0064]
[0065] This function encourages the pursuit robot to approach the escaped robot in order to capture it.
[0066] Compared with the prior art, the present invention has the following advantages:
[0067] The present invention provides a method for encirclement and capture based on multi-agent deep reinforcement learning and using obstacles. It addresses the problem of collaborative encirclement and capture of robot swarms in scenarios with obstacles. The method uses a multi-agent deep reinforcement learning algorithm based on MATD3 to design an encirclement and capture strategy. The encirclement and capture process is divided into four states: search, encirclement, transfer and capture. The method makes full use of obstacles to construct the reward function for each state, thereby improving the success rate of encirclement and capture and reducing the time required for successful encirclement and capture.
[0068] Based on the above reasons, this invention can be widely applied in fields such as multi-robot collaborative capture. Attached Figure Description
[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0070] Figure 1 This is a flowchart of the method of the present invention.
[0071] Figure 2 This invention provides a scenario of pursuit and escape involving obstacles, as provided in an embodiment of the invention.
[0072] Figure 3 This is a schematic diagram of a trapped state provided in an embodiment of the present invention.
[0073] Figure 4 This is a schematic diagram of another entrapment state provided for an embodiment of the present invention.
[0074] Figure 5 The graph shows the change in capture success rate with the number of pursuit robots, as provided in the embodiments of the present invention.
[0075] Figure 6 The graph shows the change in time taken for successful capture as a function of the number of pursuit robots, as provided in this embodiment of the invention.
[0076] Figure 7 The time taken for each state without considering obstacles is provided for embodiments of the present invention.
[0077] Figure 8 The time taken for each state when considering obstacles is provided for embodiments of the present invention. Detailed Implementation
[0078] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0079] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0080] like Figure 1 As shown, this invention provides a method for encirclement using obstacles based on multi-agent deep reinforcement learning, comprising:
[0081] S1, constructing includes N A pursuit robot , An escape robot E and a static obstacle O The chase scene;
[0082] S2. Based on the constructed pursuit and escape scenario, construct the MATD3 framework of a multi-agent deep reinforcement learning algorithm;
[0083] S3. Based on the constructed multi-agent deep reinforcement learning algorithm MATD3 framework, design a reward function for a multi-robot encirclement strategy based on state transition;
[0084] S4. Combine the designed reward function with the MATD3 algorithm to obtain a multi-robot cooperative trapping strategy based on multi-agent deep reinforcement learning MATD3.
[0085] In a specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:
[0086] S11, such as Figure 2 As shown, the selected scene is a two-dimensional finite closed area, and obstacles are represented by straight lines of fixed length. It is assumed that the obstacles only block the robot's path of travel, but do not obstruct the observation line of sight.
[0087] S12, Use and They represent obstacles. O The two endpoints, using Indicates a pursuit robot and escape robots E The intersection of the line connecting the two obstacles with the obstacle;
[0088] S13. Construct the motion equations for each robot as follows:
[0089]
[0090] in, It's a robot. i Position coordinates in a two-dimensional planar scene It is linear velocity. It is the direction angle of the robot's movement speed. It is acceleration; the maximum speed of all pursuit robots is limited to... The maximum speed of the escape robot is .
[0091] In a specific implementation, as a preferred embodiment of the present invention, in step S2, a multi-agent dual-delay deep policy determination gradient algorithm is used. Each agent in the constructed multi-agent deep reinforcement learning algorithm MATD3 framework includes six networks: the current Actor network, the target Actor network, two current Critic networks, and two target Critic networks, wherein:
[0092] The Actor network employs a distributed execution approach, taking its own state information as input and outputting the decision action; the two Critic networks employ a centralized training approach, taking their own state and action information, as well as the state and action information of all other agents, as input, and outputting two Q-values that are close to the truth.
[0093] The learning objective of each agent is to maximize the expected reward, i.e., the Q value. The network structure and training method in each agent adopt the settings in the Deep Deterministic Policy Gradient Algorithm 3 (TD3).
[0094] In a specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:
[0095] S31, Design an obstacle avoidance reward ,as follows:
[0096]
[0097] in, and Intersection points To the left and right ends of the obstacle and distance, A collection of all robots without obstructions; each pursuit robot Determine if it is related to the escape robot E Whether there are obstacles in between, i.e., determining the pursuit robot With escape robots E Does the line connecting the positions intersect with the obstacles? If there are obstacles, the pursuit robot cannot participate in the encirclement operation and must therefore bypass the obstacles.
[0098] S32, Designing Robot Collision Avoidance Rewards ,as follows:
[0099]
[0100] in, To set a safe distance, It is a constant. For robots i To the obstacle O The minimum distance, For robots i The minimum distance to the scene boundary. For robots i and j The distance between them; the designed robot collision avoidance reward is used to guide the robot to avoid collisions with obstacles and the boundaries of the motion scene, and to prevent robots from colliding with each other;
[0101] S33, Design Encirclement Rewards ,as follows:
[0102] In search mode, let This enables the pursuit robot to learn to use obstacles to form an encirclement around the escape robot;
[0103] In a siege, This allows the pursuit robot to continuously surround the escape robot using obstacles;
[0104] In the transition state, let This enables the pursuit robots to learn to narrow down the encirclement;
[0105] In the state of arrest, This causes the pursuit robot to move toward the escaped robot and capture it.
[0106] S34, Pursuit Robot In the t The overall reward function for each time step is the designed obstacle avoidance reward. Robot collision avoidance reward and capture rewards The sum of the three parts is expressed as:
[0107] .
[0108] In a specific implementation, as a preferred embodiment of the present invention, in step S33, the designed capture reward... The game is divided into four states: search, siege, transfer, and capture, each with its own reward function. , , , Each state corresponds to an easier subtask and a corresponding reward function. When the conditions for each state are met, the corresponding reward function is applied. The conditions for each state and the corresponding reward function are designed as follows:
[0109] S331. Search Status: In the search status, the escape robot is outside the encirclement formed by the pursuing robot and obstacles, determined by the following conditions:
[0110]
[0111] in, S Let represent the area of the largest convex region enclosed by the index points. The reward function for the search state is designed as follows:
[0112]
[0113] in, Indicates the first i From a pursuit robot to an escape robot E distance, This item is used to punish the pursuit robot for moving away from the escape robot;
[0114] S332, Encirclement State: In the encirclement state, the escaping robot is located within a polygonal area formed by the pursuing robot and obstacles, such as... Figure 3 As shown. The robot is arranged in a counter-clockwise order from 1 to... N Number them when season ;when season Based on the relative positions of each robot and the obstacles, it is determined whether the robot remains trapped, and a corresponding reward function is applied, which is divided into the following four cases:
[0115] Scenario 1: For those who meet the conditions robots (i.e., robot) The left and right sides are not connected to obstacles, for example. Figure 3 Robots in ), satisfying the following conditions:
[0116]
[0117] in, The maximum speed ratio, K The reward function is designed to represent the maximum number of time steps required to achieve a siege.
[0118]
[0119] Scenario 2: For those who meet the conditions robots (i.e., robot) The right side connects to obstacles, while the left side contains other pursuit robots, such as... Figure 3 Robots in ), satisfying the following conditions:
[0120]
[0121] in, Representative robot The center of the circle of Apollonius formed by the escapee and the escapee A i To the obstacle minimum distance (e.g.) Figure 3 (As shown), center of circle A i The coordinates are ,in and respectively, the pursuit robot and escape robots E Position coordinates; The radius of the Apollonius circle is defined as... This condition indicates that the robot To form an encirclement around the robot on its left and defend the gap between it and the obstacle on its right, the corresponding reward function is designed as follows:
[0122]
[0123] Scenario 3: For those who meet the conditions robots (i.e., robot) The right side connects to other pursuit robots, while the left side is an obstacle, such as... Figure 3 Robots in ), satisfying the following conditions:
[0124]
[0125] That is, robots To form an encirclement around the robot on its right and defend the gap between it and the obstacle on its left, the corresponding reward function is designed as follows:
[0126]
[0127] Scenario 4: For those who meet the conditions robots (i.e., robot) Both the left and right sides are connected to obstacles, such as Figure 4 Robots in Its position is between the obstacle and the escaped robot, so it is impossible to use the obstacle to capture the escaped robot. Therefore, the following reward function is set:
[0128]
[0129] in, The reward is the radian value of the angle formed by the points indicated by their subscripts; this reward is intended to improve the robot's performance. The location can be transformed into one of the three situations mentioned above, so as to make full use of obstacles for encirclement and capture.
[0130] S333, Transition State: The transition state is the state that shrinks from the siege state to the capture state, and satisfies the following conditions:
[0131]
[0132] in, For a time step, This represents the maximum distance at which the pursuing robot can initiate a capture action against the escaped robot. In this state, the pursuing robot continuously narrows the encirclement, trapping the escapee near the obstacle until the subsequent capture state is reached. During this period, adjacent pursuing robots continuously adjust the distance between them to make them as equal as possible, preventing the escapee from escaping between two adjacent pursuing robots with a large gap. The reward function corresponding to the transition state is designed as follows:
[0133]
[0134] in, It allows the pursuit robots to narrow down the encirclement. It forces the escapee to get closer to the obstacle. This is to ensure that the distance between adjacent robots is equal, defined as ,in ;
[0135] S334. Capture State: In the capture state, regardless of the action chosen by the escape robot, as long as the pursuing robot moves towards the escape robot at maximum speed, it can be captured. When in the capture state, each robot must meet the following conditions:
[0136]
[0137] The corresponding reward function is designed as follows:
[0138]
[0139] This function encourages the pursuit robot to approach the escaped robot in order to capture it.
[0140] Example
[0141] A multi-to-one pursuit and escape environment with linear obstacles was constructed, with specific parameters shown in Table 1. The simulation environment was based on the PyCharm Community 2020.3 and Anaconda3 platforms, using the Baidu PaddlePaddle 2.2 deep learning framework, with Adam Optimizer as the training optimizer. The computer platform consisted of an AMD Ryzen 7 5800H CPU, 16GB of RAM, and an NVIDIA RTX 3050 Ti GPU with 4GB of VRAM. The MATD3 algorithm from this invention, combined with the designed reward function, was used to train the pursuit robot. The escape robot was also trained using the MATD3 algorithm, with its reward function designed to avoid being captured and avoid collisions with the environment. The reward for being captured was -10, and the reward for successfully escaping was 1. The reward for avoiding collisions was set the same as that for the pursuit robot.
[0142] Table 1 Environmental Parameters
[0143]
[0144] To demonstrate the advantages of the method of this invention, the comparison algorithm used only considers collision avoidance rewards. The same treatment applies to obstacles, but the reward for avoiding obstacles is removed. And in designing the capture reward The presence of obstacles is not considered. When the number of pursuit robots is 3-9, each robot is trained for 20,000 rounds. Each trained model is then tested 1,000 times. The capture success rate and the time taken for successful capture are statistically analyzed. The results are as follows: Figure 5 and Figure 6 As shown in the diagram, the obstacle-based reward system can improve the success rate of encirclement and reduce the time required for successful capture, especially when the number of pursuing robots is small (3-6). Therefore, the pursuers effectively utilize obstacles to encircle and capture the escapee. Furthermore, taking the example of 3 pursuing robots encircling 1 escape robot, the time used for each of the four states—search, encirclement, transfer, and capture—is statistically analyzed. The results are presented in the form of a box plot, as shown below. Figure 7 and Figure 8 As shown in the figure, comparing the two figures reveals that the algorithm designed in this invention significantly reduces the time required for the trapped state, while the time required for other states remains largely unchanged, thereby improving the overall trapping efficiency.
[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for encirclement and trapping using obstacles based on multi-agent deep reinforcement learning, characterized in that, include: S1, constructing includes N A pursuit robot , An escape robot E and a static obstacle O The chase scene; S2. Based on the constructed pursuit and escape scenario, construct the MATD3 framework of a multi-agent deep reinforcement learning algorithm; S3. Based on the constructed multi-agent deep reinforcement learning algorithm MATD3 framework, a reward function for a multi-robot encirclement strategy based on state transitions is designed, specifically including: S31, Design an obstacle avoidance reward ,as follows: in, and Intersection points To the left and right ends of the obstacle and distance, A collection of all robots without obstructions; each pursuit robot Determine if it is related to the escape robot E Whether there are obstacles in between, i.e., determining the pursuit robot With escape robots E Does the line connecting the positions intersect with the obstacles? If there are obstacles, the pursuit robot cannot participate in the encirclement operation and must therefore bypass the obstacles. S32, Designing Robot Collision Avoidance Rewards ,as follows: in, To set a safe distance, It is a constant. For robots i To the obstacle O The minimum distance, For robots i The minimum distance to the scene boundary. For robots i and j The distance between them; the designed robot collision avoidance reward is used to guide the robot to avoid collisions with obstacles and the boundaries of the motion scene, and to prevent robots from colliding with each other; S33, Design Encirclement Rewards ,as follows: In search mode, let This enables the pursuit robot to learn to use obstacles to form an encirclement around the escape robot; In a siege, This allows the pursuit robot to continuously surround the escape robot using obstacles; In the transition state, let This enables the pursuit robots to learn to narrow down the encirclement; In the state of arrest, This causes the pursuit robot to move toward the escaped robot and capture it. S34, Pursuit Robot In the t The overall reward function for each time step is the designed obstacle avoidance reward. Robot collision avoidance reward and capture rewards The sum of the three parts is expressed as: S4. Combine the designed reward function with the MATD3 algorithm to obtain a multi-robot cooperative trapping strategy based on multi-agent deep reinforcement learning MATD3.
2. The obstacle-based encirclement method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, Step S1 specifically includes: S11. Select a two-dimensional finite closed area as the scene, represent obstacles with straight lines of fixed length, and assume that the obstacles only block the robot's path of travel, but do not obstruct the observation line of sight. S12, Use and They represent obstacles. O The two endpoints, using Indicates a pursuit robot and escape robots E The intersection of the line connecting the two obstacles with the obstacle; S13. Construct the motion equations for each robot as follows: in, It's a robot. i Position coordinates in a two-dimensional planar scene It is linear velocity. It is the direction angle of the robot's movement speed. It is acceleration; the maximum speed of all pursuit robots is limited to... The maximum speed of the escape robot is .
3. The obstacle-based encirclement method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S2, a multi-agent, dual-delay deep policy determination gradient algorithm is used. The MATD3 framework, a multi-agent deep reinforcement learning algorithm, constructs a network in which each agent comprises six networks: the current Actor network, the target Actor network, two current Critic networks, and two target Critic networks. The Actor network employs a distributed execution approach, taking its own state information as input and outputting the decision action; the two Critic networks employ a centralized training approach, taking their own state and action information, as well as the state and action information of all other agents, as input, and outputting two Q-values that are close to the truth.
4. The obstacle-based encirclement method based on multi-agent deep reinforcement learning according to claim 3, characterized in that, The learning objective of each agent is to maximize the expected reward, i.e., the Q value. The network structure and training method in each agent adopt the settings in the Deep Deterministic Policy Gradient Algorithm 3 (TD3).
5. The obstacle-based encirclement method based on multi-agent deep reinforcement learning according to claim 1, characterized in that, In step S33, the designed capture reward The game is divided into four states: search, siege, transfer, and capture, each with its own reward function. , , , Each state corresponds to an easier subtask and a corresponding reward function. When the conditions for each state are met, the corresponding reward function is applied. The conditions for each state and the corresponding reward function are designed as follows: S331. Search Status: In the search status, the escape robot is outside the encirclement formed by the pursuing robot and obstacles, determined by the following conditions: in, S Let represent the area of the largest convex region enclosed by the index points. The reward function for the search state is designed as follows: in, Indicates the first i From a pursuit robot to an escape robot E distance, This item is used to punish the pursuit robot for moving away from the escape robot; S332, Encirclement State: In the encirclement state, the escape robot is located within a polygonal area formed by the pursuing robot and obstacles. The robots are pursued in a counter-clockwise order from 1 to... N Number them when season ;when season Based on the relative positions of each robot and the obstacles, it is determined whether the robot remains trapped, and a corresponding reward function is applied, which is divided into the following four cases: Scenario 1: For those who meet the conditions robots ,robot The left and right sides are not connected to any obstacles, satisfying the following condition: in, The maximum speed ratio, K The reward function is designed to represent the maximum number of time steps required to achieve a siege. Scenario 2: For those who meet the conditions robots ,robot The right side is connected to an obstacle, while the left side is other pursuit robots, satisfying the following conditions: in, Representative robot The center of the circle of Apollonius formed by the escapee and the escapee A i To the obstacle The minimum distance, the center of the circle A i The coordinates are ,in and respectively, the pursuit robot and escape robots E Position coordinates; The radius of the Apollonius circle is defined as... This condition indicates that the robot To form an encirclement around the robot on its left and defend the gap between it and the obstacle on its right, the corresponding reward function is designed as follows: Scenario 3: For those who meet the conditions robots ,robot The right side connects to other pursuit robots while the left side is an obstacle, satisfying the following conditions: That is, robots To form an encirclement around the robot on its right and defend the gap between it and the obstacle on its left, the corresponding reward function is designed as follows: Scenario 4: For those who meet the conditions robots Its position is between the obstacle and the escaped robot, so it is impossible to use the obstacle to surround and capture the escaped robot. Therefore, the following reward function is set: in, The reward is the radian value of the angle formed by the points indicated by their subscripts; this reward is intended to improve the robot's performance. The location is transformed into the three situations mentioned above, so as to make full use of obstacles for encirclement and capture; S333, Transition State: The transition state is the state that shrinks from the siege state to the capture state, and satisfies the following conditions: in, For a time step, This represents the maximum distance at which the pursuing robot can initiate a capture action against the escaped robot. In this state, the pursuing robot continuously narrows the encirclement, trapping the escapee near the obstacle until the subsequent capture state is reached. During this period, adjacent pursuing robots continuously adjust the distance between them to make them as equal as possible, preventing the escapee from escaping between two adjacent pursuing robots with a large gap. The reward function corresponding to the transition state is designed as follows: in, It allows the pursuit robots to narrow down the encirclement. It forces the escapee to get closer to the obstacle. This is to ensure that the distance between adjacent robots is equal, defined as ,in ; S334. Capture State: In the capture state, regardless of the action chosen by the escape robot, as long as the pursuing robot moves towards the escape robot at maximum speed, it can be captured. When in the capture state, each robot must meet the following conditions: The corresponding reward function is designed as follows: This function encourages the pursuit robot to approach the escaped robot in order to capture it.