Improved matd3 multi-robot cooperative hunting method based on bi-gru structure

Abstract

The application provides an improved MATD3 multi-robot cooperative hunting method based on a BiGRU structure, which comprises the following steps: establishing a pursuit and escape confrontation scene; based on the established pursuit and escape confrontation scene and a reinforcement learning TD3 algorithm, designing a MATD3 algorithm suitable for multi-robots; based on the BiGRU structure, improving the designed MATD3 algorithm suitable for multi-robots; based on a multi-robot hunting strategy of individual and team rewards, designing a reward function; combining the designed reward function with the improved MATD3 algorithm to obtain a multi-robot cooperative hunting strategy algorithm based on MATD3-BiGRU. The application adopts a multi-agent reinforcement learning algorithm based on MATD3 to solve a multi-robot cooperative hunting problem in a two-dimensional scene, and the reward function in the algorithm adopts a weighted form of individual capture rewards and team cooperation rewards; the multi-agent reinforcement learning algorithm based on the improved MATD3 of the BiGRU structure is used to improve the problem that the hunting performance decreases when the number of pursuit and capture robots changes.

Description

Technical Field

[0001] This invention relates to the field of multi-robot cooperative trapping technology, and more particularly to an improved MATD3 multi-robot cooperative trapping method based on a BiGRU structure. Background Technology

[0002] Multi-robot cooperative encirclement strategies have significant application value in scenarios such as military confrontation, autonomous search and rescue, and have always been a research hotspot. Research methods have evolved from early model-based differential game theory and optimal control methods to more recent deep reinforcement learning methods that do not rely on specific models. These methods can be trained offline for a given task scenario and then applied to real-world situations. In multi-robot cooperative encirclement tasks, when a robot malfunctions and cannot continue participating in the encirclement mission, or when a new robot joins during the encirclement process, changing the number of robots, the trained strategies may not be as effective in adapting to the new environment. Therefore, deep reinforcement learning algorithms need to remain robust to environmental changes and be able to adapt to unfamiliar (but similar) environments. Most existing methods involve retraining the robots in the new environment, which is time-consuming and computationally wasteful, and such situations are unacceptable in real-world scenarios. Summary of the Invention

[0003] To address the technical problem of discrepancies between the actual number of robots used in the environment and the number used in training in multi-robot cooperative encirclement tasks, this invention provides an improved MATD3 multi-robot cooperative encirclement method based on a BiGRU structure. This invention achieves effective encirclement of a single escaping robot by multiple pursuing robots and improves the algorithm's adaptability to changes in the number of robots.

[0004] The technical means employed in this invention are as follows:

[0005] An improved MATD3 multi-robot cooperative trapping method based on a BiGRU structure includes:

[0006] Establish a scenario for pursuit and combat;

[0007] Based on the established pursuit and escape adversarial scenario and the reinforcement learning TD3 algorithm, a MATD3 algorithm suitable for multiple robots is designed.

[0008] Based on the BiGRU structure, the MATD3 algorithm suitable for multiple robots is improved.

[0009] Design a reward function based on a multi-robot encirclement strategy that combines individual and team rewards.

[0010] By combining the designed reward function with the improved MATD3 algorithm, a multi-robot cooperative trapping strategy algorithm based on MATD3-BiGRU is obtained.

[0011] Furthermore, the establishment of the pursuit and escape confrontation scenario includes:

[0012] Establish a pursuit-escape scenario involving N pursuing robots and a single escape robot, and express the motion equations of each robot as follows:

[0013]

[0014] Among them, v i ψ represents the speed of the pursuing or escaping robot. i The angle 'a' represents the velocity direction of the pursuing or escaping robot. i This indicates the acceleration of the pursuing or escaping robot.

[0015] Furthermore, based on the established pursuit-escape adversarial scenario and the reinforcement learning TD3 algorithm, a MATD3 algorithm suitable for multiple robots is designed, including:

[0016] The reinforcement learning TD3 algorithm is extended to the MATD3 algorithm, which is applicable to multiple robots, using a centralized training and distributed execution framework. Each agent in the MATD3 algorithm consists of six networks: the current Actor network, the target Actor network, two current Critic networks, and two target Critic networks. The Actor network adopts a distributed execution approach, taking its own state information as input and outputting the decision action. The two Critic networks adopt a centralized training approach, taking not only their own state and action information but also the state and action information of other agents as input, and outputting two Q values ​​close to the true value. When calculating the target Q value, in order to alleviate the overestimation problem, a smaller target Q value is selected to guide the parameter update of the Actor network.

[0017] In a multi-robot cooperative pursuit task, the learning objective of each robot is to maximize its expected reward, i.e., its Q-value; for the pursuit robot, its learning objective function is:

[0018]

[0019] Where φ is the Actor network parameter, s is the robot's current state, μ is the deterministic policy corresponding to the policy network output, θ is the Critic network parameter, N is the number of pursuing robots, and D is the empirical data, i.e. the data in the experience replay pool.

[0020] At each time t, the loss function of the Critic network is:

[0021]

[0022]

[0023] Where θ′ is the parameter of the target Critic network and φ′ is the parameter of the target Actor network. The target Critic network and the target Actor network have the same structure as the Critic network and the Actor network, but the parameter update frequency is different.

[0024] The Actor network is trained by maximizing the objective function, using stochastic gradient ascent to update its parameters. Based on the objective function defined above, its gradient is:

[0025]

[0026] The Critic network is trained by minimizing its loss function, and its parameters are updated using stochastic gradient descent. The gradient of the Critic network's loss function is calculated based on its formula:

[0027]

[0028]

[0029] Finally, the moving average method is used to update the target network parameters to ensure the stability of the algorithm. The update method is as follows:

[0030]

[0031] Furthermore, the improvement of the designed MATD3 algorithm for multi-robot applications based on the BiGRU structure includes:

[0032] The structure of each Critic network is introduced into a bidirectional gated loop unit. The bidirectional connection characteristics of the bidirectional gated loop unit are used to connect robots in the same team, so that robots can exchange information and share real-time information.

[0033] By utilizing the parameter sharing characteristic of bidirectional gated loop units, robots in the same team can share parameters. The number of robots is independent of the number of parameters. When the number of robots changes, the network structure can also automatically expand. Therefore, it can be applied to situations where the number of robots changes.

[0034] Furthermore, the multi-robot encirclement strategy based on individual and team rewards includes a reward function designed as follows:

[0035] The overall reward function of robot i at time step t consists of three parts, represented as follows:

[0036]

[0037] in, As a reward for the capture, Rewards for robots that avoid collisions with each other. Rewards for avoiding collisions at scene boundaries;

[0038] Design a capture reward function The design process is as follows:

[0039] The aim is to encourage each pursuit robot to capture an escaped robot and to encourage pursuit robots to cooperate as a team to form a good encirclement formation, as defined below:

[0040]

[0041] Where d i,e and d j,e Let d represent the distances from pursuing robots i and j to the escaped robot, respectively. cap Indicates the capture distance; if the target is captured by robot i (i.e., d...). i,e ≤d cap If so, then robot i will receive a reward. And the other robots will receive set up This encourages the pursuit robots to cooperate with each other; during the movement of a target that is not captured, each pursuit robot receives a negative reward -ω. q q-ω d d i,e It is determined by the individual's reward (the distance d from the target). i,e It consists of a weighted linear combination of the formation score and the group reward (formation score q), where the formation score q is defined as:

[0042]

[0043] In the above formula, n represents the number of robots, and the pursuit robot closest to the target is defined as robot 0. This represents the unit vector between robot i and the target direction; this formation score encourages robots to spread out around the target, i.e., approach the target from different directions to form an encirclement posture; weight ω q and ω d The choice should ensure that the robot's reward is primarily determined by the formation score as it approaches the target, thereby encouraging the chasing robots to cooperate as a team and form a good encirclement formation.

[0044] Design a reward function for robots to avoid collisions. The function is defined as follows:

[0045]

[0046] Where, δ safety For a safe distance, ζ is a negative constant, d i,o Let d be the distance between robot i and obstacle o. i,j Let i be the distance between robots i and j;

[0047] Design a scene boundary collision avoidance reward function. Used to guide robots to avoid collisions with the boundaries of the moving scene, defined as follows:

[0048]

[0049] Where, δ safety For a safe distance, ξ is a negative constant, d i,b Let be the distance between robot i and the boundary.

[0050] Furthermore, the combination of the designed reward function with the improved MATD3 algorithm yields a multi-robot cooperative encirclement strategy algorithm based on MATD3-BiGRU, including:

[0051] Initialize two Critic networks and the corresponding Critic network parameters θ i,1 θ i,2 Actor network μ θi and the corresponding Actor network parameters φ i Two target Critic network parameters θ i ′ ,1 ←θ i,1 θ i ′ ,2 ←θ i,2 and Actor network parameters φ i ′←φ i The experience replay pool D and the robot's state;

[0052] For each robot i, select an action based on the exploration noise. Receive rewards

[0053] Sample M samples are randomly sampled from the experience pool D.

[0054] calculate

[0055] Update the Critic network parameters.

[0056] Update the Actor network parameters using policy gradients.

[0057] The target network parameters are updated using the moving average method.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] 1. The improved MATD3 multi-robot cooperative encirclement method based on BiGRU structure provided by this invention addresses the cooperative encirclement problem of robot clusters in two-dimensional scenes. It extends the TD3 algorithm to the case of multiple robots by using a centralized training and distributed execution framework, thus making it suitable for multi-robot encirclement applications.

[0060] 2. The improved MATD3 multi-robot cooperative encirclement method based on BiGRU structure provided by this invention improves the problem of algorithm performance degradation when the number of robots changes in the pursuit problem by introducing an improved MATD3 deep reinforcement learning algorithm based on BiGRU structure.

[0061] Based on the above reasons, this invention can be widely applied in fields such as multi-robot collaborative capture. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of the method of the present invention.

[0064] Figure 2 This is a schematic diagram of the motion of the pursuit robot P and the escape robot E provided in an embodiment of the present invention.

[0065] Figure 3 The MATD3 algorithm framework diagram is provided for embodiments of the present invention.

[0066] Figure 4 The diagram shows the framework of the MATD3 algorithm based on the BiGRU structure provided in this embodiment of the invention.

[0067] Figure 5 This is a schematic diagram of a multi-to-one pursuit and escape confrontation environment provided in an embodiment of the present invention;

[0068] Figure 6The graph showing the sum of reward values ​​for the pursuit robot in Experiment 1, provided as an embodiment of the present invention.

[0069] Figure 7 The graph showing the sum of reward values ​​for the pursuit robot in Experiment 2, provided as an embodiment of the present invention. Detailed Implementation

[0070] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0073] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0074] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0075] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0076] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0077] like Figure 1 As shown, this invention provides an improved MATD3 multi-robot cooperative trapping method based on a BiGRU structure, comprising:

[0078] S1. Establish a pursuit and escape confrontation scenario;

[0079] S2. Based on the established pursuit and escape adversarial scenario and the reinforcement learning TD3 algorithm, design the MATD3 algorithm suitable for multiple robots;

[0080] S3. Based on the BiGRU structure, the MATD3 algorithm suitable for multiple robots is improved.

[0081] S4. Design a reward function for a multi-robot encirclement strategy based on individual and team rewards;

[0082] S5. The designed reward function is combined with the improved MATD3 algorithm to obtain a multi-robot cooperative trapping strategy algorithm based on MATD3-BiGRU.

[0083] In a specific implementation, as a preferred embodiment of the present invention, step S1, establishing a pursuit and escape confrontation scenario, includes:

[0084] Establish a pursuit and escape scenario involving N pursuing robots and a single escape robot, such as... Figure 2 As shown, the motion equations for each robot are expressed as follows:

[0085]

[0086] Among them, v i ψ represents the speed of the pursuing or escaping robot. i The angle 'a' represents the velocity direction of the pursuing or escaping robot. i This indicates the acceleration of the pursuing or escaping robot.

[0087] In a specific implementation, as a preferred embodiment of the present invention, in step S2, based on the established pursuit-escape adversarial scenario and the reinforcement learning TD3 algorithm, a MATD3 algorithm suitable for multiple robots is designed, including:

[0088] This paper extends the reinforcement learning TD3 algorithm to the MATD3 algorithm, applicable to multiple robots, using a centralized training and distributed execution framework. The MATD3 algorithm has the following characteristics: Each agent not only has a current Actor network and a corresponding target Actor network, but also two current Critic networks and two corresponding target Critic networks. Overestimation is mitigated by selecting a smaller Q-value as the Q-value for the next state; a delayed update method is used to update the Actor network, i.e., the Actor network is updated only after the Critic network has been updated several times, solving the coupling problem between the value function and the policy; noise is added to the output of the target Actor network to smooth the target policy. Adding noise to the target action is equivalent to a regularization method to alleviate the overfitting problem, thus making the predicted output Q-value of the target Critic network as accurate as possible. Figure 3The diagram shows the framework of the MATD3 algorithm. Each agent in the diagram includes six networks: the current Actor network, the target Actor network, two current Critic networks, and two target Critic networks. The Actor network adopts a distributed execution approach, taking its own state information as input and outputting the decision action. The two Critic networks adopt a centralized training approach, taking not only their own state and action information but also the state and action information of other agents as input, and outputting two Q values ​​close to the true value. When calculating the target Q value, in order to alleviate the overestimation problem, a smaller target Q value is chosen to guide the parameter update of the Actor network.

[0089] In a multi-robot cooperative pursuit task, the learning objective of each robot is to maximize its expected reward, i.e., its Q-value; for the pursuit robot, its learning objective function is:

[0090]

[0091] Where φ is the Actor network parameter, s is the robot's current state, μ is the deterministic policy corresponding to the policy network output, θ is the Critic network parameter, N is the number of pursuing robots, and D is the empirical data, i.e. the data in the experience replay pool.

[0092] At each time t, the loss function of the Critic network is:

[0093]

[0094]

[0095] Where θ′ is the parameter of the target Critic network and φ′ is the parameter of the target Actor network. The target Critic network and the target Actor network have the same structure as the Critic network and the Actor network, but the parameter update frequency is different.

[0096] The Actor network is trained by maximizing the objective function, using stochastic gradient ascent to update its parameters. Based on the objective function defined above, its gradient is:

[0097]

[0098] The Critic network is trained by minimizing its loss function, and its parameters are updated using stochastic gradient descent. The gradient of the Critic network's loss function is calculated based on its formula:

[0099]

[0100]

[0101] Finally, the moving average method is used to update the target network parameters to ensure the stability of the algorithm. The update method is as follows:

[0102]

[0103] In a specific implementation, as a preferred embodiment of the present invention, step S3 involves improving the designed MATD3 algorithm suitable for multiple robots based on the BiGRU structure, including:

[0104] To enhance the algorithm's generalization ability, a Bidirectional Gated Recurrent Unit (BiGRU) is introduced into the structure of each Critic network. The bidirectional connectivity of BiGRU connects robots within the same team, enabling them to exchange and share real-time information. Furthermore, leveraging BiGRU's parameter-sharing feature, robots within the same team can share parameters; the number of robots is independent of the number of parameters. The network structure automatically expands when the number of robots changes, thus making it applicable to scenarios with varying robot counts. The MATD3 algorithm framework based on the BiGRU structure is as follows: Figure 4 As shown.

[0105] In a specific implementation, as a preferred embodiment of the present invention, step S4 involves designing a reward function based on a multi-robot encirclement strategy that includes individual and team rewards, comprising:

[0106] The overall reward function of robot i at time step t consists of three parts, represented as follows:

[0107]

[0108] in, As a reward for the capture, Rewards for robots that avoid collisions with each other. Rewards for avoiding collisions at scene boundaries;

[0109] Design a capture reward function The design process is as follows:

[0110] The aim is to encourage each pursuit robot to capture an escaped robot and to encourage pursuit robots to cooperate as a team to form a good encirclement formation, as defined below:

[0111]

[0112] Where d i,e and d j,eLet d represent the distances from pursuing robots i and j to the escaped robot, respectively. cap Indicates the capture distance; if the target is captured by robot i (i.e., d...). i,e ≤d cap If so, then robot i will receive a reward. And the other robots will receive set up This encourages the pursuit robots to cooperate with each other; during the movement of a target that is not captured, each pursuit robot receives a negative reward -ω. q q-ω d d i,e It is determined by the individual's reward (the distance d from the target). i,e It consists of a weighted linear combination of the formation score and the group reward (formation score q), where the formation score q is defined as:

[0113]

[0114] In the above formula, n represents the number of robots, and the pursuit robot closest to the target is defined as robot 0. This represents the unit vector between robot i and the target direction; this formation score encourages robots to spread out around the target, i.e., approach the target from different directions to form an encirclement posture; weight ω q and ω d The choice should ensure that the robot's reward is primarily determined by the formation score as it approaches the target, thereby encouraging the chasing robots to cooperate as a team and form a good encirclement formation.

[0115] Design a reward function for robots to avoid collisions. The function is defined as follows:

[0116]

[0117] Where, δ safety For a safe distance, ζ is a negative constant, d i,o Let d be the distance between robot i and obstacle o. i,j Let i be the distance between robots i and j;

[0118] Design a scene boundary collision avoidance reward function. Used to guide robots to avoid collisions with the boundaries of the moving scene, defined as follows:

[0119]

[0120] Where, δ safety For a safe distance, ξ is a negative constant, d i,b Let be the distance between robot i and the boundary.

[0121] In a specific implementation, as a preferred embodiment of the present invention, step S5 combines the designed reward function with the improved MATD3 algorithm to obtain a multi-robot cooperative encirclement strategy algorithm based on MATD3-BiGRU, including:

[0122] Initialize two Critic networks and the corresponding Critic network parameters θ i,1 θ i,2 Actor Network and the corresponding Actor network parameters φ i Two target Critic network parameters θ′ i,1 ←θ i,1 , θ′ i,2 ←θ i,2 and Actor network parameters φ i ′←φ i The experience replay pool D and the robot's state;

[0123] For each robot i, select an action based on the exploration noise. Receive rewards

[0124] Sample M samples are randomly sampled from the experience pool D.

[0125] calculate

[0126] Update the Critic network parameters.

[0127] Update the Actor network parameters using policy gradients.

[0128] The target network parameters are updated using the moving average method.

[0129] Example

[0130] Based on the multi-to-one pursuit and confrontation scenario, construct a multi-to-one fugitive pursuit environment such as... Figure 5As shown, the black circles represent pursuing robots, the white circles represent escaping robots, and the black rectangles in the scene represent obstacles. The simulation environment is based on Python, utilizing PyCharm Community 2020.3 and the Anaconda3 platform. The deep learning framework used is Baidu PaddlePaddle 2.2, and the training optimizer used in the experiment is AdamOptimizer. The computer configuration is Windows 10 operating system, Intel i5-9400F CPU @ 2.90GHz, and 8GB of memory.

[0131] Specifically, the following description is provided for the multi-hunting robot single-escape robot pursuit experiment setup: all pursuing robots are isomorphic, meaning their parameters and performance are identical; both are positioned within a two-dimensional continuous bounded circle with a radius of 10m. The escape robot's escape strategy is also developed through training, and it can only obtain rewards by preventing itself from being captured and avoiding collisions. The rewards for being captured are shown in Table 3, and the rewards for avoiding collisions are consistent with those for the pursuing robots. The training parameters for the experiment are shown in Table 1, and the environmental parameters for the experiment are shown in Table 2.

[0132] Table 1 Training Hyperparameters

[0133]

[0134] Table 2. Experimental parameters for a multi-to-one confrontational environment.

[0135]

[0136] Table 3 Reward Design for Escaped Robots

[0137]

[0138] Both the pursuing and escaping robots were trained using the MATD3 algorithm for a total of 20,000 rounds. Experiment 1 involved three pursuing robots surrounding an escaping robot. After the algorithm converged (at round 10,000), three new pursuing robots were added. A test was conducted every 100 rounds, and the sum of the rewards obtained by all pursuing robots was calculated. Figure 6 As shown, the overall reward decreases due to the increased likelihood of collisions with more robots, but the MATD3 algorithm using BiGRU experiences a smaller decrease in the final reward value compared to the original MATD3 algorithm, and the reward value curve recovers to stability faster.

[0139] In Experiment 2, six pursuit robots were initially used to capture one escaped robot. After the algorithm converged (at round 10,000), three pursuit robots withdrew, leaving three robots to continue the capture mission. A test was conducted every 100 rounds, and the sum of the rewards obtained by all pursuit robots was calculated as follows: Figure 7 As shown, the MATD3 algorithm using BiGRU converges faster and yields a higher final reward value.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An improved MATD3 multi-robot cooperative hunting method based on a BiGRU structure, characterized in that, The application relates to a multi-robot cooperative hunting strategy based on MATD3-BiGRU. A pursuit and evasion confrontation scene is established; Based on the established pursuit and evasion confrontation scene and a reinforcement learning TD3 algorithm, a MATD3 algorithm suitable for multiple robots is designed; Based on a BiGRU structure, the designed MATD3 algorithm suitable for multiple robots is improved, including: The structure of each Critic network is introduced into a bidirectional gated recurrent unit, the bidirectional connection characteristics of the bidirectional gated recurrent unit are utilized to connect the robots of the same team with each other, information is exchanged between the robots, and real-time information is shared; The parameter sharing characteristics of the bidirectional gated recurrent unit are utilized to realize parameter sharing between the robots of the same team, the number of robots is independent of the number of parameters, and when the number of robots changes, the network structure is automatically expanded, so that the algorithm is applied to the case where the number of robots changes; A multi-robot hunting strategy based on individual and team rewards is designed, and a reward function is designed, including: robot i In the t The overall reward function within a time step consists of three parts, represented as follows: wherein, is a corral reward, is a robot mutual collision avoidance reward, is a scene boundary collision avoidance reward; Designing a corral reward function The design process is as follows: The goal is to encourage each of the chasing robots to capture the escaping robot and to encourage the chasing robots to cooperate in order to form a good encircling formation, defined as follows: in and They represent pursuit robots. i and j The distance to the escaped robot, Indicates the capture distance; if the target is captured by the robot i Capture, i.e. So, the robot i You will receive a reward Other robots j Will receive ,set up This encourages the pursuit robots to cooperate with each other; each robot receives a negative reward during the movement of a target that is not captured. Negative rewards are given by individuals, i.e., the chasing robots. i Distance to the escape robot And group rewards, i.e., formation scores It consists of a weighted linear combination, where the formation score q Defined as: In the above formula, n denotes the number of robots, the pursuit robot closest to the target is defined as robot 0, denotes the robot i and the unit vector of the target direction; this formation score encourages the robots to spread around the target, i.e. to approach the target from different directions to form a siege situation; the weight and The selection of and should ensure that the reward of the robots in the process of approaching the target is mainly determined by the formation score, so as to encourage the pursuit robots to cooperate to form a good siege formation; Designing a collision avoidance reward function for robots The function is defined as follows: wherein, is a safety distance, is a negative constant, is a robot i to an obstacle o between the robot, is a robot i and j between the robot; Designing a scenario boundary collision avoidance reward function To guide the robot to avoid collision with the boundary of the motion scenario, the following definition is given: wherein, is a safety distance, is a negative constant, is a robot i distance between the robot and the border; The designed reward function is combined with the improved MATD3 algorithm to obtain a multi-robot cooperative hunting strategy algorithm based on MATD3-BiGRU.

2. The improved MATD3 multi-robot cooperative hunting method based on the BiGRU structure according to claim 1, wherein The pursuit and evasion confrontation scene is established, including: A pursuit-evasion game scenario is established including N The motion equation of each robot is represented as: wherein, denotes the speed magnitude of the pursuing robot or the escaping robot, denotes the speed direction angle of the pursuing robot or the escaping robot, denotes the acceleration of the pursuing robot or the escaping robot.

3. The improved MATD3 multi-robot cooperative hunting method based on the BiGRU structure according to claim 1, wherein, Based on the established pursuit and evasion confrontation scene and a reinforcement learning TD3 algorithm, a MATD3 algorithm suitable for multiple robots is designed, including: The reinforcement learning TD3 algorithm is expanded to the MATD3 algorithm suitable for multiple robots by using a centralized training and decentralized execution framework, each agent in the MATD3 algorithm includes six networks, namely a current Actor network, a target Actor network, two current Critic networks and two target Critic networks; the Actor network adopts a decentralized execution mode, inputs state information of itself, and outputs a decision action; the two Critic networks adopt a centralized training mode, input state and action information of other agents in addition to state and action information of themselves, and output two Q values close to true values; in the calculation of a target Q value, a smaller target Q value is selected to guide the parameter update of the Actor network in order to alleviate the overestimation problem; In the multi-robot cooperative hunting task, the learning goal of each robot is to maximize the expected reward, that is, a Q value; for a pursuit robot, a target function to be learned is: wherein, are Actor network parameters, is the current state of the robot, is the deterministic policy corresponding to the policy network output, are Critic network parameters, N is the number of pursuer robots, D is experience data, i.e. data in the experience replay pool; At each time instant t The loss function of Critic network is: wherein, are parameters of the target Critic network, are parameters of the target Actor network, the target Critic network and the target Actor network have the same structure as the Critic network and the Actor network, and have different parameter update frequencies; The Actor network is trained by maximizing the target function, and a stochastic gradient ascent method is adopted to update the parameters of the Actor network; according to the above definition of the target function, the gradient of the target function is: The Critic network is trained by minimizing the loss function of the Critic network, and a stochastic gradient descent method is adopted to update the network parameters of the Critic; according to the loss function formula of the Critic network, the gradient of the loss function of the Critic network is calculated as: Finally, the method of moving average is used to update the parameters of the target network, which is used to ensure the stability of the algorithm, and the updating mode is as follows: 。 4. The improved MATD3 multi-robot cooperative hunting method based on the BiGRU structure according to claim 1, wherein, The designed reward function is combined with the improved MATD3 algorithm to obtain a multi-robot cooperative hunting strategy algorithm based on MATD3-BiGRU. Initialize 2 Critic networks , and corresponding Critic network parameters , , Actor network and corresponding Actor network parameters , 2 target Critic network parameters , and Actor network parameters , experience replay pool D and state of the robot; Select action for each robot i according to exploration noise , obtain reward ; sample stored in the experience pool D, randomly sampling M samples from the experience pool ; Computing ; updating the Critic network parameters, ; updating the actor network parameters by policy gradient, ; updating the target network parameters by a sliding average method, .

Images