A multi-robot cooperative hunting method based on maximum entropy reinforcement learning

Abstract

The application provides a kind of multi-robot cooperative hunting method based on maximum entropy reinforcement learning.The method of the application comprises: establishing a pursuit and escape confrontation scene;based on the established pursuit and escape confrontation scene and SAC multi-agent reinforcement learning algorithm, design MASAC algorithm suitable for multi-robot; based on the multi-robot hunting strategy of course learning, design reward function; combine the designed reward function with MASAC algorithm to obtain the multi-robot cooperative hunting strategy algorithm based on MASAC. The application adopts the multi-agent reinforcement learning algorithm based on maximum entropy to realize the cooperative hunting of multiple mobile robots on a single target in a two-dimensional scene; the hunting stage is divided into four stages of search, encirclement, transfer and capture, and the idea of course learning is used to design reward function and conversion condition for each stage; the effectiveness of the deep reinforcement learning method in the multi-pursuit robot hunting of the fast-moving single-escape robot is verified, and the efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-robot cooperative path planning technology, and more particularly to a multi-robot cooperative encirclement method based on maximum entropy reinforcement learning. Background Technology

[0002] Multi-robot cooperative encirclement strategies have significant application value in scenarios such as military confrontation, autonomous search and rescue, and have always been a research hotspot. Currently, most research on multi-robot encirclement problems starts from classical control theory, designing encirclement strategies by manually setting or optimizing solutions based on the robot's mathematical model. This approach ignores the difficulty of establishing accurate mathematical models for robots in real-world scenarios, thus having certain limitations. In recent years, deep reinforcement learning methods in the field of artificial intelligence combine the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning. They can directly learn control strategies from high-dimensional raw data, exhibiting strong versatility and thus receiving extensive research in multi-robot cooperative encirclement problems. Among them, the maximum entropy-based reinforcement learning SAC method has stronger exploration capabilities and faster training speed compared to the deterministic policy gradient (DDPG) method, showing advantages in handling complex tasks. However, existing deep reinforcement learning methods in multi-robot encirclement research lack detailed design of encirclement rewards, resulting in slow algorithm convergence speeds and ultimately, a relatively low success rate in encirclement. Summary of the Invention

[0003] To address the aforementioned technical issues, a multi-robot cooperative encirclement algorithm based on deep reinforcement learning is provided. A multi-agent reinforcement learning algorithm based on maximum entropy reinforcement learning (SAC) is designed, along with a reward function for the multi-robot encirclement strategy based on a curriculum learning mechanism. This enables multiple pursuing robots to effectively encircle and capture escaped robots, improving the algorithm's convergence speed and success rate.

[0004] The technical means employed in this invention are as follows:

[0005] A multi-robot cooperative encirclement method based on maximum entropy reinforcement learning includes:

[0006] Establish a scenario for pursuit and combat;

[0007] Based on the established pursuit and escape adversarial scenario and the SAC multi-agent reinforcement learning algorithm, a MASAC algorithm suitable for multiple robots is designed.

[0008] Design a reward function for a multi-robot encirclement strategy based on curriculum learning;

[0009] By combining the designed reward function with the MASAC algorithm, a multi-robot cooperative trapping strategy algorithm based on MASAC is obtained.

[0010] Furthermore, the established pursuit and escape scenario includes N pursuing robots P and a single escape robot E.

[0011] Furthermore, based on the established pursuit-escape adversarial scenario and the SAC multi-agent reinforcement learning algorithm, a MASAC algorithm suitable for multiple robots is designed, including:

[0012] The reinforcement learning SAC algorithm is extended to the MASAC algorithm, which is applicable to multiple robots, using a framework of centralized training and distributed execution. Specifically, this includes:

[0013] The MASAC experience pool is designed as follows:

[0014]

[0015] in, Let represent the set of observations of all robots at time t. Let represent the set of all robot actions at time t. Let represent the reward obtained by all robots after performing their respective actions at time t. This represents the set of observations from all robots at time t+1;

[0016] Using the basic Actor-Critic network framework, we design one Actor network and two Critic networks for each robot. The Actor network is used to learn the motion strategy, that is, to determine the next motion direction and acceleration based on its current position and velocity. The Critic network is used to evaluate the quality of the learned strategy, that is, to evaluate the quality of the motion strategy adopted in the current state based on the current state (including position and velocity) and the strategy adopted by the robot.

[0017] In SAC, entropy regularization is introduced to maximize the expectation of the objective function, whose value function is:

[0018]

[0019] Where γ is the discount factor, α is the temperature parameter, and H(π(·|s) t The entropy of policy π is given by (). In MASAC, assuming that the policies of robots are independent of each other, the entropy of the joint policy is:

[0020]

[0021] During the policy evaluation phase, the Q-value function is updated based on the Bellman optimality equation, and its learning objective function is:

[0022]

[0023] Where θ is the Critic network parameter, θ′ is the target Critic network parameter, φ is the Actor network parameter, and D is the empirical data, i.e., the data in the empirical replay pool;

[0024] Based on the objective function, the network parameters of Critic are updated using stochastic gradient descent.

[0025]

[0026] in,

[0027] During the policy improvement phase, stochastic gradient ascent is used to update the parameters of the Actor network, and the learning objective function is:

[0028]

[0029] Based on the objective function defined above, the gradient of the objective function is:

[0030]

[0031] The temperature parameter is updated as follows:

[0032]

[0033] Finally, the moving average method is used to update the target network parameters to ensure the stability of the algorithm. The update method is as follows:

[0034] θ i ′ ,n =τθ i,n +(1-τ)θ i ′ ,n n = 1, 2.

[0035] Furthermore, the input to the Actor network of each robot is its own state information (including position and velocity), and the output is the strategy adopted based on the current state. The input to the Critic network of each robot is the state information of all robots (including position and velocity) and the actions performed, and the output is the state-action value function, i.e., the Q value. Both the Actor network and the Critic network are composed of three fully connected layers with 64 hidden neurons. The activation function of the first two layers of the neural network is the Rectified Linear Unit function, and the last layer of the neural network does not use an activation function.

[0036] Furthermore, the multi-robot encirclement strategy based on curriculum learning includes a reward function, specifically comprising:

[0037] Design the overall reward function:

[0038] The overall reward function for each hunting robot i in the t-th time step is expressed as:

[0039]

[0040] in, As a reward for the capture, Rewards for robots that avoid collisions with each other. The scene boundary avoidance reward is used to guide the robot to avoid getting too close to the boundary of the moving scene;

[0041] Design a capture reward function The design process is as follows:

[0042] Using the concepts of course learning, the cooperative capture process of multiple robots is divided into four states: search, encirclement, transfer, and capture. Each state corresponds to an easier subtask, and these subtasks are completed sequentially until the final capture task is achieved. The four subtasks correspond to four reward functions, namely... First let equal This teaches the pursuit robots how to form a formation to surround the escaped robots; then it enables them to... equal The pursuing robots surround the escaped robot to meet the siege conditions; then... equal The pursuit robots will learn to narrow down the encirclement; finally... equal The pursuing robot moves toward the escaped robot until the capture mission is complete.

[0043] Design a reward function for robots to avoid collisions. The function is defined as follows:

[0044]

[0045] Where, δ safety For a safe distance, ζ is a negative constant;

[0046] Design a scene boundary collision avoidance reward function. The function is defined as follows:

[0047]

[0048] Furthermore, the design of the capture reward function The specific conditions that each state should satisfy and the corresponding reward function are defined as follows:

[0049] Search status:

[0050] The state of a search is when the escape robot is not surrounded by pursuing robots, and the following formula applies:

[0051]

[0052] Where S represents the area of ​​the region enclosed by the subscript point;

[0053] The reward function corresponding to the search state is as follows:

[0054]

[0055] Where, d i,E This represents the distance from the i-th pursuing robot to the escaped robot. The regularization term is used to penalize escaped robots during the search phase, as the pursuing robots may expand their enclosed area to keep them away from the escaped robots.

[0056] Siege status:

[0057] When under siege, the following formula applies:

[0058]

[0059] Where, d i,i+1 This represents the distance between the i-th pursuing robot and its adjacent (i+1)-th pursuing robot; in and represents the maximum speed of the pursuing robot and the escape robot, respectively; the above formula means that N pursuing robots can surround the escapee for at least K time steps; during the siege, the pursuing robots can further adopt strategies so that after a maximum of K steps, they enter the fourth state (capture state).

[0060] The reward function for the siege state is as follows:

[0061]

[0062] The reward is only positive when the conditions of the encirclement are met, and the larger the reward, the more time the hunting robot has to shrink the encirclement.

[0063] Transition status:

[0064] The transition state is used to describe the transition from the trapped state to the captured state, and satisfies the following formula:

[0065]

[0066] Where Δt is a time step, this subtask aims to adjust the distance between any two pursuing robots to make them as equidistant as possible, while narrowing the encirclement until the capture state is achieved. The corresponding reward function is as follows:

[0067]

[0068] Among them, -logd i,E The purpose is to narrow down the encirclement. in Item H is to make d i,i+1 Equidistant;

[0069] Capture status:

[0070] The capture state is the final state in which all pursuing robots successfully surround and capture the escaped robot. Regardless of the escaped robot's chosen action, as long as the pursuing robots move towards the escaped robot when the capture state is satisfied, the pursuing robots can capture the escaped robot. The capture state satisfies the following formula:

[0071]

[0072] The corresponding capture state reward function is as follows:

[0073]

[0074] The reward means that the chasing robot will receive more rewards if it gets close to the escaped robot.

[0075] Furthermore, the combination of the designed reward function with the MASAC algorithm yields a multi-robot cooperative trapping strategy algorithm based on MASAC, specifically including:

[0076] Initialize two Critic networks and the corresponding Critic network parameters θ i,1 θ i,2 Actor Network and the corresponding Actor network parameters φ i Initialize the two target Critic network parameters θ′ i,1 ←θ i,1 , θ′ i,2 ←θ i,2 Initialize the experience replay pool D; Initialize the robot's state;

[0077] For each robot i, select an action based on the exploration noise. Receive rewards

[0078] Sample M samples are randomly sampled from the experience pool D.

[0079] Update Q-value function;

[0080] Update the Critic network parameters.

[0081] Update the Actor network parameters.

[0082] Update the temperature coefficient.

[0083] The target network parameters, θ′, are updated using the moving average method. i,n =τθ i,n +(1-τ)θ′ i,n n = 1, 2.

[0084] Compared with the prior art, the present invention has the following advantages:

[0085] 1. The multi-robot cooperative trapping method based on maximum entropy reinforcement learning provided by this invention extends the maximum entropy reinforcement learning algorithm using a centralized training and distributed execution framework, thus making it applicable to multi-robot cooperative trapping task scenarios.

[0086] 2. The multi-robot cooperative trapping method based on maximum entropy reinforcement learning provided by this invention divides the trapping phase into four stages: search, trapping, transfer, and capture, and designs a corresponding reward function for each stage, thereby improving the convergence rate of the learning algorithm and the success rate of the overall trapping strategy.

[0087] 3. The multi-robot cooperative trapping method based on maximum entropy reinforcement learning provided by this invention designs a mutual collision avoidance constraint reward for robots, which can effectively meet the collision avoidance requirements in multi-robot cooperative trapping tasks.

[0088] Based on the above reasons, this invention can be widely applied in fields such as multi-robot collaborative path planning. Attached Figure Description

[0089] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a flowchart of the method of the present invention.

[0091] Figure 2This is a schematic diagram of the motion of the pursuit robot P and the escape robot E provided in an embodiment of the present invention.

[0092] Figure 3 This invention provides a centralized training and distributed execution framework for embodiments of the invention.

[0093] Figure 4 This provides a multi-to-one pursuit and escape confrontation environment for embodiments of the present invention;

[0094] Figure 5 This is a comparison chart of capture success rates provided in an embodiment of the present invention.

[0095] Figure 6 This is a comparison chart of capture success rates provided in an embodiment of the present invention.

[0096] Figure 7 A comparison chart of successful capture times provided for embodiments of the present invention. Detailed Implementation

[0097] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0098] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0099] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0100] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0101] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0102] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0103] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0104] like Figure 1As shown, this invention provides a multi-robot cooperative encirclement method based on maximum entropy reinforcement learning, comprising:

[0105] S1. Establish a pursuit and escape confrontation scenario;

[0106] S2. Based on the established pursuit and escape adversarial scenario and the SAC multi-agent reinforcement learning algorithm, design the MASAC algorithm suitable for multiple robots.

[0107] S3. Design a reward function for a multi-robot encirclement strategy based on course learning;

[0108] S4. Combine the designed reward function with the MASAC algorithm to obtain a multi-robot cooperative trapping strategy algorithm based on MASAC.

[0109] In a specific implementation, as a preferred embodiment of the present invention, the pursuit and escape confrontation scenario established in step S1 is as follows: Figure 2 As shown, it contains N pursuing robots P and a single escape robot E.

[0110] In a specific implementation, as a preferred embodiment of the present invention, in step S2, based on the established pursuit-escape adversarial scenario and the SAC multi-agent reinforcement learning algorithm, a MASAC algorithm suitable for multiple robots is designed, including:

[0111] Utilize Figure 3 The centralized training and distributed execution framework shown extends the reinforcement learning SAC algorithm to the MASAC algorithm, which is applicable to multiple robots, specifically including:

[0112] S21. Design the MASAC experience pool as follows:

[0113]

[0114] in, Let represent the set of observations of all robots at time t. Let represent the set of all robot actions at time t. Let represent the reward obtained by all robots after performing their respective actions at time t. This represents the set of observations from all robots at time t+1;

[0115] S22. Using the basic Actor-Critic network framework, design one Actor network and two Critic networks for each robot. The Actor network is used to learn the motion strategy, that is, to determine the next motion direction and acceleration based on its current position and velocity. The Critic network is used to evaluate the quality of the learned strategy, that is, to evaluate the quality of the motion strategy adopted in the current state (including position and velocity) and the strategy adopted by the robot. In order to solve the problem of Q-value overestimation, two sets of target Critic networks are used to represent different Q-values. By selecting the smallest one as the updated target Q-value, the continuous overestimation is suppressed, and this information is fed back to the Actor network. Then, the Actor network and the Critic network are trained together. In this embodiment, the input to the Actor network of each robot is its own state information (including position and velocity), and the output is the strategy adopted based on the current state. The input to the Critic network of each robot is the state information of all robots (including position and velocity) and the actions performed, and the output is the state-action value function, i.e., the Q value. Both the Actor network and the Critic network are composed of three fully connected layers with 64 hidden neurons. The activation function of the first two neural networks is the Rectified Linear Unit (ReLU) function, and the last neural network does not use an activation function.

[0116] S23. In SAC, entropy regularization is introduced to maximize the expectation of the objective function, and its value function is:

[0117]

[0118] Where γ is the discount factor, α is the temperature parameter, and H(π(·|s) t The entropy of policy π is given by (). In MASAC, assuming that the policies of robots are independent of each other, the entropy of the joint policy is:

[0119]

[0120] S24. During the policy evaluation phase, the Q-value function is updated based on the Bellman optimal equation, and its learning objective function is:

[0121]

[0122] Where θ is the Critic network parameter, θ′ is the target Critic network parameter, φ is the Actor network parameter, and D is the empirical data, i.e., the data in the empirical replay pool;

[0123] S25. Based on the objective function, update the Critic network parameters using stochastic gradient descent:

[0124]

[0125] in,

[0126] S26. In the policy improvement phase, the stochastic gradient ascent method is used to update the parameters of the Actor network. The learning objective function is:

[0127]

[0128] S27. Based on the objective function defined above, the gradient of the objective function is:

[0129]

[0130] S28. Temperature parameter update method is as follows:

[0131]

[0132] S29. Finally, the moving average method is used to update the target network parameters to ensure the stability of the algorithm. The update method is as follows:

[0133] θ i ′ ,n =τθ i,n +(1-τ)θ i ′ ,n n = 1, 2.

[0134] In a specific implementation, as a preferred embodiment of the present invention, step S3, which involves designing a reward function based on a multi-robot encirclement strategy learned from the course, specifically includes:

[0135] S31. Design the overall reward function:

[0136] The overall reward function for each hunting robot i in the t-th time step is expressed as:

[0137]

[0138] in, As a reward for the capture, Rewards for robots that avoid collisions with each other. The scene boundary avoidance reward is used to guide the robot to avoid getting too close to the boundary of the moving scene;

[0139] S32. Design the capture reward function The design process is as follows:

[0140] Using the concepts of course learning, the cooperative capture process of multiple robots is divided into four states: search, encirclement, transfer, and capture. Each state corresponds to an easier subtask, and these subtasks are completed sequentially until the final capture task is achieved. The four subtasks correspond to four reward functions, namely... First let equal This teaches the pursuit robots how to form a formation to surround the escaped robots; then it enables them to... equal The pursuing robots surround the escaped robot to meet the siege conditions; then... equal The pursuit robots will learn to narrow down the encirclement; finally... equal The pursuing robot moves towards the escaped robot until the capture mission is complete. In this embodiment, the specific conditions that each state should satisfy and the corresponding reward function are defined as follows:

[0141] (1) Search status:

[0142] The state of a search is when the escape robot is not surrounded by pursuing robots, and the following formula applies:

[0143]

[0144] Where S represents the area of ​​the region enclosed by the subscript point;

[0145] The reward function corresponding to the search state is as follows:

[0146]

[0147] Where, d i,E This represents the distance from the i-th pursuing robot to the escaped robot. The regularization term is used to penalize escaped robots during the search phase, as the pursuing robots may expand their enclosed area to keep them away from the escaped robots.

[0148] (2) Besieged state:

[0149] When under siege, the following formula applies:

[0150]

[0151] Where, d i,i+1 This represents the distance between the i-th pursuing robot and its adjacent (i+1)-th pursuing robot; in and represents the maximum speed of the pursuing robot and the escape robot, respectively; the above formula means that N pursuing robots can surround the escapee for at least K time steps; during the siege, the pursuing robots can further adopt strategies so that after a maximum of K steps, they enter the fourth state (capture state).

[0152] The reward function for the siege state is as follows:

[0153]

[0154] The reward is only positive when the conditions of the encirclement are met, and the larger the reward, the more time the hunting robot has to shrink the encirclement.

[0155] (3) Transition state:

[0156] The transition state is used to describe the transition from the trapped state to the captured state, and satisfies the following formula:

[0157]

[0158] Where Δt is a time step, this subtask aims to adjust the distance between any two pursuing robots to make them as equidistant as possible, while narrowing the encirclement until the capture state is achieved. The corresponding reward function is as follows:

[0159]

[0160] Among them, -logd i,E The purpose is to narrow down the encirclement. in Item H is to make d i,i+1 Equidistant;

[0161] (4) Capture state:

[0162] The capture state is the final state in which all pursuing robots successfully surround and capture the escaped robot. Regardless of the escaped robot's chosen action, as long as the pursuing robots move towards the escaped robot when the capture state is satisfied, the pursuing robots can capture the escaped robot. The capture state satisfies the following formula:

[0163]

[0164] The corresponding capture state reward function is as follows:

[0165]

[0166] The reward means that the chasing robot will receive more rewards if it gets close to the escaped robot.

[0167] S33, Design a reward function for robot collision avoidance. The function is defined as follows:

[0168]

[0169] Where, δ safety For a safe distance, ζ is a negative constant;

[0170] S34. Design the scene boundary collision avoidance reward function The function is defined as follows:

[0171]

[0172] In a specific implementation, as a preferred embodiment of the present invention, step S4 combines the designed reward function with the MASAC algorithm to obtain a multi-robot cooperative trapping strategy algorithm based on MASAC, specifically including:

[0173] S41. Initialize two Critic networks. and the corresponding Critic network parameters θ i,1 θ i,2 Actor Network and the corresponding Actor network parameters φ i Initialize the parameters θ′ of the two target Critic networks. i,1 ←θ i,1 , θ′ i,2 ←θ i,2 Initialize the experience replay pool D; Initialize the robot's state;

[0174] S42. Select an action for each robot i based on the exploration noise. Receive rewards

[0175] S43, Sample M samples are randomly sampled from the experience pool D.

[0176] S44, Q-value update function;

[0177] S45. Update the Critic network parameters.

[0178] S46. Update the Actor network parameters.

[0179] S47. Update the temperature coefficient.

[0180] S48. Update the target network parameters, θ′, using the moving average method. i,n =τθ i,n +(1-τ)θ′i,n n = 1, 2.

[0181] Example

[0182] Setting up a multi-to-one encirclement environment, such as Figure 4 As shown, the red team is designated as the pursuing robot, and the blue team as the escaping robot. The scene environment is a circular bounded area with a radius of 10m. The simulation environment is based on Python, utilizing PyCharm Community 2020.3 and the Anaconda3 platform. The deep learning framework used is Baidu PaddlePaddle 2.2, and the training optimizer used in the experiment is Adam Optimizer. The computer configuration is Windows 10 operating system, Intel i5-9400F CPU @ 2.90GHz, and 8GB of memory.

[0183] Specifically, the following explanation is provided regarding the setup for the multi-hunting robot / single-escape robot pursuit experiment: all pursuing robots are isomorphic, meaning their parameters and performance are identical; both are positioned within a two-dimensional continuous bounded circle with a radius of 10m. The escape robot's escape strategy is also developed through training, and it can only obtain rewards by preventing itself from being captured and avoiding collisions. The rewards for being captured are shown in Table 3, and the rewards for avoiding collisions are consistent with those for the pursuing robots. The training parameters for the experiment are shown in Table 1, and the environmental parameters for the experiment are shown in Table 2.

[0184] Table 1 Training Hyperparameters

[0185]

[0186] Table 2. Experimental parameters for a multi-to-one encirclement and suppression environment.

[0187]

[0188] Experiment 1: To verify the effectiveness and superiority of the designed strategy, a comparison was made between the reward based on course learning and the method based on the original reward. The reward design based on the original reward is shown in Table 3. Both the pursuit robot and the escape robot were trained using the MASAC algorithm for 20,000 rounds, with a test every 200 rounds. A pursuit robot failed to capture the robot and the escape robot successfully escaped when the capture time exceeded 200 seconds. The number of successful captures was counted to calculate the capture success rate. A comparison of the capture success rates under the two strategies is shown below. Figure 5 As shown, the reward system based on course learning significantly improves the training speed and the final capture success rate.

[0189] Table 3 Reward Design Based on Original Rewards

[0190]

[0191] Experiment 2: The designed MASAC algorithm is compared with existing MADDPG and IL-DDPG algorithms. Each algorithm uses a reward function based on course learning. The success rate of encirclement and capture and the time spent on successful capture are statistically analyzed as follows: Figure 6 and Figure 7 As shown. The comparison yields the following conclusions: IL-DDPG, due to the robot's independent learning, achieves a low capture success rate and takes the longest to successfully capture, exhibiting poor convergence stability. In contrast, the MASAC algorithm, which uses a centralized training and distributed execution framework, converges faster and achieves a higher capture success rate than IL-DDPG. Furthermore, the MASAC algorithm converges even faster than the MADDPG algorithm, thus demonstrating the advantages of the MASAC algorithm.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-robot cooperative trapping method based on maximum entropy reinforcement learning, characterized in that, include: Establish a scenario for pursuit and combat; Based on the established pursuit-escape adversarial scenario and the SAC multi-agent reinforcement learning algorithm, a MASAC algorithm suitable for multiple robots is designed, including: The reinforcement learning SAC algorithm is extended to the MASAC algorithm, which is applicable to multiple robots, using a framework of centralized training and distributed execution. Specifically, this includes: The MASAC experience pool is designed as follows: in, express t The set of observations from all robots at any given time. express t The collection of all robot actions at any given moment. express t The rewards that all robots receive after performing their respective actions at any given time. express t The set of observations of all robots at time +1; Using the basic Actor-Critic network framework, we design one Actor network and two Critic networks for each robot. The Actor network is used to learn the motion strategy, that is, to determine the next motion direction and acceleration based on its current position and velocity. The Critic network is used to evaluate the quality of the learned strategy, that is, to evaluate the quality of the current motion strategy based on the current state including position and velocity and the strategy adopted by the robot. In SAC, entropy regularization is introduced to maximize the expectation of the objective function, whose value function is: in, As a discount factor, For temperature parameters, For strategy The entropy of the joint policy; in MASAC, assuming that the policies of the robots are independent of each other, the entropy of the joint policy is: During the policy evaluation phase, updates are based on the Bellman optimal equation. Q The value function, whose learning objective function is: in, These are the parameters of the Critic network. These are the parameters of the target Critic network. These are the parameters of the Actor network. D It is empirical data, that is, data in the experience replay pool; Based on the objective function, the network parameters of Critic are updated using stochastic gradient descent. in, ; During the policy improvement phase, stochastic gradient ascent is used to update the parameters of the Actor network, and the learning objective function is: Based on the objective function defined above, the gradient of the objective function is: The temperature parameter is updated as follows: Finally, the moving average method is used to update the target network parameters to ensure the stability of the algorithm. The update method is as follows: ； A multi-robot encirclement strategy based on curriculum learning is proposed, and a reward function is designed, including: Design the overall reward function: Each pursuit robot i In the t The overall reward function over a time step is expressed as: in, As a reward for the capture, Rewards for robots that avoid collisions with each other. The scene boundary avoidance reward is used to guide the robot to avoid getting too close to the boundary of the moving scene; Design a capture reward function The design process is as follows: Using the concepts of course learning, the cooperative capture process of multiple robots is divided into four states: search, encirclement, transfer, and capture. Each state corresponds to an easier subtask, and these subtasks are completed sequentially until the final capture task is achieved. The four subtasks correspond to four reward functions, namely... , , , First, let equal This enabled the pursuit robots to learn how to form a formation to surround the escaped robot; then it allowed them to... equal This causes the pursuing robots to surround the escaped robot, thus fulfilling the siege condition; then... equal The pursuit robots will learn to narrow down the encirclement; finally, equal The pursuing robot moves toward the escaped robot until the capture mission is complete; Design a reward function for robots to avoid collisions. The function is defined as follows: in, For a safe distance, It is a negative constant; Design a scene boundary collision avoidance reward function. The function is defined as follows: By combining the designed reward function with the MASAC algorithm, a multi-robot cooperative trapping strategy algorithm based on MASAC is obtained.

2. The multi-robot cooperative encirclement method based on maximum entropy reinforcement learning according to claim 1, characterized in that, The established pursuit and escape confrontation scenario includes N A pursuit robot P and a single escape robot E .

3. The multi-robot cooperative encirclement method based on maximum entropy reinforcement learning according to claim 1, characterized in that, The input to the Actor network of each robot is its own state information, including position and velocity, and the output is the strategy adopted based on the current state. The input to the Critic network of each robot is the state information of all robots and the actions performed, and the output is the state-action value function, i.e., the Q value. Both the Actor network and the Critic network consist of three fully connected layers with 64 hidden neurons. The activation function of the first two layers of the neural network is the Rectified LinearUnit function, and the last layer of the neural network does not use an activation function.

4. The multi-robot cooperative encirclement method based on maximum entropy reinforcement learning according to claim 1, characterized in that, The design of the capture reward function The specific conditions that each state should satisfy and the corresponding reward function are defined as follows: Search status: The state of a search is when the escape robot is not surrounded by pursuing robots, and the following formula applies: in, Indicates the area of ​​the region enclosed by the subscript point; The reward function corresponding to the search state is as follows: in, Indicates the first i The distance between the pursuing robot and the escaped robot. For regularization, in order to contain the escaped robot during the search phase, the pursuing robot may take action to expand its enclosed area, which will keep them away from the escaped robot. The regularization term is to punish this situation. Siege status: When under siege, the following formula applies: in, Indicates the first i The first pursuit robot and the adjacent first i +1 distance to a pursuit robot; ,in and These represent the maximum speeds of the pursuing robot and the escape robot, respectively; the above formula indicates... N One pursuit robot can surround at least one escapee. K One time step; during the siege, the hunting robot can further employ strategies to maximize... K After the first step, it enters the fourth state, namely the capture state; The reward function for the siege state is as follows: The reward is only positive when the conditions of the encirclement are met, and the larger the reward, the more time the hunting robot has to shrink the encirclement. Transition status: The transition state is used to describe the transition from the trapped state to the captured state, and satisfies the following formula: in, For a given time step, this subtask aims to adjust the distance between any two pursuing robots to make them equidistant, while simultaneously narrowing the encirclement until a capture state is achieved. The corresponding reward function is as follows: in, The purpose is to narrow down the encirclement. ,in , The item is to make Equidistant; Capture status: The capture state is the final state in which all pursuing robots successfully surround and capture the escaped robot. Regardless of the escaped robot's chosen action, as long as the pursuing robots move towards the escaped robot when the capture state is satisfied, the pursuing robots can capture the escaped robot. The capture state satisfies the following formula: The corresponding capture state reward function is as follows: The reward means that the chasing robot will receive more rewards if it gets close to the escaped robot.

5. The multi-robot cooperative encirclement method based on maximum entropy reinforcement learning according to claim 1, characterized in that, The process of combining the designed reward function with the MASAC algorithm to obtain a multi-robot cooperative trapping strategy algorithm based on MASAC specifically includes: Initialize two Critic networks , and the corresponding Critic network parameters , Actor Network and the corresponding Actor network parameters Initialize the parameters of the two target Critic networks. , Initialize the experience replay pool D Initialize the robot's state; For each robot i Select action based on noise exploration Receive rewards ; Sample Stored in the experience pool D In the middle, random sampling is performed from the experience pool. M Sample ; Update Q-value function; Update the Critic network parameters. ; Update the Actor network parameters. ; Update the temperature coefficient. ; The target network parameters are updated using the moving average method. .

Images