Deep reinforcement learning multi-agent cooperative hunting method with time feature extraction mechanism and hunting system thereof

Abstract

The present application belongs to the field of multi-agent cooperative hunting, and provides a deep reinforcement learning multi-agent cooperative hunting method with time feature extraction mechanism and a hunting system thereof. Step 1: initializing environment information, determining the agent escape strategy, determining the observation space of the actor network and the central critic network, constructing the reward function, and constructing the hunting task completion condition; Step 2: constructing the MADDPG architecture, which includes a central critic network with a time feature extraction mechanism and an actor network; Step 3: the agent interacts with the environment information, collects data, and preheats; Step 4: updating the neural network parameters; Step 5: repeating step 4 until the number of training rounds is completed, and finally obtaining the trained actor network to realize multi-agent cooperative hunting. For the hunting task, accurately extracting the time feature is crucial, which not only can significantly improve the success rate of hunting, but also is the key to promoting the intelligent level of the multi-agent system.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperative trapping technology, specifically involving a deep reinforcement learning multi-agent cooperative trapping method and its trapping system with a time feature extraction mechanism. Background Technology

[0002] Multi-agent encirclement, as a typical complex system problem, involves multiple intelligent agents (such as drones, robots, or software agents) cooperating to track and capture moving targets. Due to its high level of intelligence, flexible and varied operational strategies, excellent collaborative capabilities, and low operating costs, multi-agent encirclement has significant application prospects in civilian tasks such as urban security, border patrol, and monitoring of major events, as well as in military tasks such as reconnaissance, surveillance, and strike operations.

[0003] The core of the multi-agent cooperative encirclement problem lies in the pursuit-escape game, a field that has become a hot topic in robotics research in recent years, covering various types of vehicles including drones, ground robots, and underwater vehicles. In nature, cooperative pursuit is a common phenomenon, aiming to enable predators to capture prey in the shortest possible time. Through strategic cooperation, predators can effectively surround prey or use environmental obstacles to restrict its escape path. Currently, mainstream methods for solving this problem include numerical solutions based on environmental information, bio-inspired strategies, force-based interaction models, control and optimization techniques, and learning-based methods. Among these, learning-based methods are particularly noteworthy, as they seek solutions through data analysis rather than direct analysis, marking a significant advancement in learning and adaptation from experience.

[0004] Reinforcement learning, a branch of machine learning related to control and planning, has combined with advances in deep learning, utilizing neural networks as function approximators to process high-dimensional data, thus evolving into deep reinforcement learning. This approach not only provides a powerful tool for solving complex multi-agent cooperative encirclement tasks but also opens up new research and application prospects. Currently, encirclement algorithms based on deep reinforcement learning mainly focus on exploring how to extract spatial features of multi-agent systems or predict the dynamic state of escaping targets, often neglecting the importance of the temporal features inherent in the training process. However, for encirclement tasks, accurately extracting temporal features is crucial; it not only significantly improves the success rate of encirclement but is also key to advancing the intelligence level of multi-agent systems. Therefore, developing a reinforcement learning algorithm focused on temporal feature extraction is extremely important in multi-agent encirclement tasks. Such an algorithm will be able to capture the details of task evolution more meticulously, thereby guiding agents to cooperate more effectively and improving the overall system efficiency and intelligence. Summary of the Invention

[0005] This invention provides a deep reinforcement learning multi-agent cooperative trapping method with a time feature extraction mechanism. For trapping tasks, accurate extraction of time features is crucial. It can not only significantly improve the success rate of trapping, but also is the key to advancing the intelligence level of multi-agent systems.

[0006] This invention provides a deep reinforcement learning multi-agent cooperative trapping system with a temporal feature extraction mechanism, which is used to realize a deep reinforcement learning multi-agent cooperative trapping method with a temporal feature extraction mechanism.

[0007] This invention is achieved through the following technical solution:

[0008] A deep reinforcement learning multi-agent cooperative trapping method with a temporal feature extraction mechanism, the multi-agent cooperative trapping method specifically includes the following steps:

[0009] Step 1: Initialize environmental information, determine the agent's escape strategy, determine the observation space of the actor network and the central commentator network, construct the reward function, and construct the conditions for completing the encirclement task;

[0010] Step 2: Construct the MADDPG architecture, which includes a central commentator network with a time feature extraction mechanism and an actor network;

[0011] Step 3: The intelligent agent interacts with the environment, collects data, and performs preheating;

[0012] Step 4: Update the neural network parameters;

[0013] Step 5: Repeat step 4 until the required number of training rounds are completed, and finally obtain the trained actor network to achieve multi-agent cooperative encirclement.

[0014] Furthermore, the observation space in step 1 specifically refers to,

[0015] The observation space of the actor network and the central commentator network is constructed as follows:

[0016] The observation space of the i-th actor network is,

[0017]

[0018] in Let be the absolute position and absolute velocity of the i-th capture agent in the inertial coordinate system. This represents the relative position of the i-th hunting agent with other hunting agents, hunting targets, or obstacles.

[0019] The central commentator network observation space is

[0020] Its dimension is Qcritic ,

[0021] in and These represent the absolute coordinates and absolute velocity in the inertial coordinate system, respectively. Let represent the action of the i-th capturing agent; in order to extract temporal features, the central commentator needs to use t during training. W The observations at each time step are stacked to form a tensor, so the data taken at time t will be stored as...

[0022]

[0023] Furthermore, the reward function constructed in step 1 is specifically as follows:

[0024] The reward function for the capture mission is as follows:

[0025]

[0026] Where k1, k2, k3, and k4 represent the weights of different rewards, and P H It represents the set of position coordinates of all intelligent agents; The distance between the center of the multi-agent system and the fleeing agent; This represents the terminal reward; when the multi-agent system completes the encirclement task, it will receive a substantial reward. The capture reward is given when the distance between a single agent and the fleeing agent is less than a threshold d. cap When the target is captured, it is considered successfully captured.

[0027] Furthermore, step 2, which involves constructing the MADDPG architecture, specifically involves first determining the meaning of the parameters; secondly, constructing the actor network and the central commentator network respectively; and finally, determining the training hyperparameters and constructing a data cache.

[0028] Furthermore, the meaning of the parameters is specifically defined as follows: the actor network and the target actor network for the i-th agent are defined as follows. and Where θ and θ′ are the learnable parameters of the actor network and the target actor network, respectively;

[0029] Define the actor network and the target actor network as follows: and in and These are the learnable parameters of the central commentator network and the target central commentator network, respectively.

[0030] The construction of the actor network specifically involves a multilayer perceptron consisting of two fully connected layers and a normalized layer group activated by the ReLU function. The actor network can be represented as follows, where θ represents the actor network parameters.

[0031]

[0032] The construction of the central commentator network specifically comprises a bidirectional gated recurrent unit network, a residual self-attention mechanism, and an MLP; the commentator network is used to evaluate the state. The effect of performing action a For ease of expression, the central commentator network will be defined in the following form:

[0033]

[0034] In this context, Q represents the central commentator network's assessment of state-action pairs. The assessment Representatives of the Central Commentators Network, Parameters of the network of commentators at the agency center.

[0035] Furthermore, step 3 includes the following steps:

[0036] Step 3.1: Transform the environmental information into the form required by the actor network, and obtain the actor network output and the actions of the fleeing agent, thereby constructing A. t After interacting with the environment, the agent obtains a new state S. t+1 and task end identifier D t , will (S t A t ,R t ,S t+1 D t Store in ReplayBuffer;

[0037] Step 3.2: Repeat step 3.1 until the amount of data stored in the ReplayBuffer is greater than the number of preheating samples W. p .

[0038] Furthermore, step 4 includes the following steps:

[0039] Step 4.1: If the current training step idx equals the round length L ep If so, the environment will be reset;

[0040] Step 4.2: If the remainder between the current training step number idx and the update interval U is 0, then update the network; otherwise, the agent generates a data tuple (S,A,R,S′,D) based on the current interaction between the agent network and the environment, and stores it in the ReplayBuffer.

[0041] A deep reinforcement learning multi-agent cooperative trapping system with a temporal feature extraction mechanism, wherein the trapping system uses the deep reinforcement learning multi-agent cooperative trapping method described above, and the trapping system includes...

[0042] Construction module: Initialize environmental information, determine the agent's escape strategy, determine the observation space of the actor network and the central commentator network, construct the reward function, and construct the conditions for completing the encirclement task;

[0043] The MADDPG architecture was constructed, which included a central commentator network with a temporal feature extraction mechanism and an actor network.

[0044] The information interaction and update module allows the intelligent agent to interact with the environment, collect data, and perform preheating; update the neural network parameters; and repeatedly update the neural network parameters until the training rounds are completed, ultimately obtaining a trained actor network to achieve multi-agent collaborative capture.

[0045] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described above.

[0046] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0047] The beneficial effects of this invention are:

[0048] This invention accelerates the execution efficiency of encirclement and capture missions, improves the success rate of encirclement and capture, and enhances the intelligence level of multi-agent systems. Attached Figure Description

[0049] Figure 1 The method flowchart of the present invention.

[0050] Figure 2 This is a structural diagram of the deep reinforcement learning algorithm of the present invention.

[0051] Figure 3 This is a simulation result diagram of the present invention. Detailed Implementation

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0053] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0054] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] The following is in conjunction with the appendix to this application specification. Figure 1-3 The technical solutions in the embodiments of this application are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0056] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0057] This invention is based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) framework, which combines a central commentator network equipped with a temporal feature extraction mechanism to guide the actor network for effective training.

[0058] like Figure 1 As shown, this invention is a deep reinforcement learning multi-agent cooperative trapping method with a temporal feature extraction mechanism. The method includes the following steps:

[0059] Step 1: Initialize the environment and generate N H N intelligent agents are used for the capture operation.T N escaped intelligent agents o One obstacle and environmental boundary B;

[0060] Step 1.1 Escape strategy of the fleeing agent;

[0061] Step 1.1.1 Obtain information about the pursuers and obstacle information The superscripts H, T, and O represent the pursuers, the target, and the obstacle, respectively. Represents coordinates, N H N represents the number of pursuers. O Represents the number of obstacles. and Code the threshold values ​​of the escape agent's actions on the pursuer and obstacles respectively, and define the set of actions of the escape agent as H∪O, where the elements will change the escape direction of the escape agent;

[0062] Step 1.1.2 The agent calculates the escape direction based on the surrounding agent and obstacles according to the following rules:

[0063] dir = normal(f(H∪O,p) T )+rand(-0.1,0.1))

[0064] Where normal() represents the normalization function, rand(a,b) represents generating random numbers between [a,b] according to a uniform distribution, and f(H∪O,p) T The definition of ) is as follows:

[0065]

[0066] Where |H∪O| represents the elements in the set H∪O, that is, the number of pursuers and obstacles within the scope of the target T, and 0 represents the zero vector. The definitions and meanings of f1, f2, and f3 are as follows:

[0067]

[0068] That is, when the number of elements in the scope is 1, it will run away from the position away from that element;

[0069]

[0070] Where B represents the boundary point of the encirclement area, d B d represents the threshold distance from the boundary point. s p represents the threshold value representing the distance between two pursuers. mid =(p i +p j f2(H∪O, p) / 2 represents the coordinates of the midpoint between the two pursuers.T This means that if there are only two elements in the scope of the fleeing agent T, and if the fleeing agent T is far from the boundary point and the elements in the scope are close to each other, the fleeing agent will escape in a direction away from the elements in the scope; otherwise, the fleeing agent T will escape towards the midpoint of the elements in the scope.

[0071]

[0072] Among them, inside(H∪O,p T ) is a Boolean function representing whether the fleeing agent is enclosed by a polygon formed by the elements in its scope, p mid,max It represents the midpoint of the longest side in the polygon formed by the elements in H∪O.

[0073] Step 1.2: Observation space of actor network and central commentator network, reward function, and end flag of the encirclement task;

[0074] Step 1.21: The observation space of the actor network is

[0075] in Let be the absolute position and absolute velocity of the i-th capture agent in the inertial coordinate system. This represents the relative position of the i-th hunting agent with other hunting agents, hunting targets, or obstacles.

[0076] Step 1.2.2: The observation space of traditional commentator networks is

[0077] Its dimension is Q critic ,in and These represent the absolute coordinates and absolute velocity in the inertial coordinate system, respectively. This represents the action of the i-th capture agent. To extract temporal features, the central commentator in this invention needs to use t during training. W The observations at each time step are stacked to form a tensor, so the data taken at time t will be stored as...

[0078] Step 1.3: The reward function for the capture mission is as follows:

[0079]

[0080] To encourage cooperation among multiple agents, the reward function evaluates the performance of all agents rather than just a single agent. Here, k1, k2, k3, and k4 represent the weights of different rewards, and P... HIt represents the set of position coordinates of all intelligent agents. The distance between the center of the multi-agent system and the fleeing agent; This represents the terminal reward; when the multi-agent system completes the encirclement task, it will receive a substantial reward. The capture reward is given when the distance between a single agent and the fleeing agent is less than a threshold d. cap When the i-th agent successfully captures the target, it is considered that the i-th agent has successfully captured the target.

[0081] Step 1.4: End condition of the encirclement task: If there are three elements in the action set H∪O of the escape agent that satisfy (1) the distance between them and the escape agent is less than a specified threshold, and (2) the escape agent is within the polygon enclosed by these three elements, then the encirclement task ends.

[0082] Step 2: Construct the MADDPG architecture, which includes a central commentator network with a temporal feature extraction mechanism and an actor network. To improve network stability, this invention enables the network in the MADDPG architecture and uses soft updates to update the parameters of the target network after each update. For ease of description, the actor network and the target actor network for the i-th agent are defined as follows: and Where θ and θ′ are the learnable parameters of the actor network and the target actor network, respectively; the actor network and the target actor network are defined as follows: and in and These are the learnable parameters of the central commentator network and the target central commentator network, respectively.

[0083] Step 2.1: Construct the Actor Network. The actor network consists of a multi-layer perceptron (MLP), which includes two fully connected layers and a normalized layer group activated by the ReLU function. The actor network can be represented as follows, where θ represents the actor network parameters.

[0084]

[0085] Step 2.2: Construct the central commentator network. The central commentator network consists of a bidirectional gated recurrent unit (Bi-GRU) network, a residual self-attention mechanism, and an MLP. The commentator network is used to evaluate the state... The effect of performing action a For ease of expression, the central commentator network will be defined in the following form:

[0086]

[0087] In this context, Q represents the central commentator network's assessment of state-action pairs. The assessment Representatives of the Central Commentators Network, The parameters of the central commentator network are given below, and the structure of the central commentator network is as follows:

[0088] Step 2.2.1: The training data, stacked through time windows, is processed by Bi-GRU to obtain the temporal features of the data within the time windows. The output can be represented as follows:

[0089] h t =Bi-GRU(X)

[0090] in, express and A t Merge along the last dimension into a new quantity. It contains information hidden forward and backward in the time dimension, where H represents the dimension of the hidden layer in Bi-GRU.

[0091] Step 2.2.2: Time feature h t The feature vector e is obtained by using a dot product self-attention mechanism. The specific process is as follows:

[0092] First, for each input h t Three different representations are obtained through three different linear transformations: query q i , key k i Sum of values ​​v i Specifically, for each element h t

[0093] q i =W q h t ,k i =W k h t ,v i =W v h t

[0094] Among them W q W k and W v It is a learnable weight matrix; secondly, scaled dot product attention is used to compute q for each query. i

[0095] With all keys k j The similarity between them is used to obtain the attention score e. ij The calculation formula is as follows:

[0096]

[0097] Next, apply the softmax function to all scores e for each i. ij Normalization is performed to obtain the attention weights:

[0098]

[0099] Finally, apply the obtained attention weights to all values ​​v i Perform a weighted average to obtain the final attention output:

[0100]

[0101] Step 2.2.3: Tensors output by the self-attention mechanism O is obtained by taking the mean along the dimension containing the time window. mean Then, O mean Input an embedding layer and transform its feature dimensions into Q. critic :

[0102]

[0103] Step 4.2.4: Output O from the embedding layer e and The sums form the residual structure, which is then input into the MLP to obtain the output of the commentator network. Its mathematical expression is shown below:

[0104]

[0105] Step 2.3: Determine the training hyperparameters and construct the data buffer:

[0106] Step 2.3.1: Determine the training hyperparameters. In this invention, the neural network training parameters include: the number of training rounds N. ep Round length L ep Update interval U, number of preheated samples W p Actor's online learning rate lr actor Commentator's online learning rate (lr) critic Actor network optimizer, commentator network optimizer, training sample size N BS Discount factor γ, network update weight τ, data cache capacity l max Number of preheated samples l Y .

[0107] Step 2.3.2: Generate the sampling data buffer, ReplayBuffer. ReplayBuffer stores data tuples (S, A, R, S′, D) collected before and after the multi-agent system interacts with the environment at each sampling time. Here, S represents the environment state at the sampling time, A represents the actions performed by the capturing and escaping agents based on the current state S, R represents the reward for each agent and escaping agent performing action A in state S, S′ represents the environment state at the next sampling time, and D represents an identifier indicating whether the capture task has ended. When the data length in ReplayBuffer exceeds the set capacity, the stored data tuples will be deleted according to the first-in, first-out principle.

[0108] Step 3: The intelligent agent interacts with the environment, collects data, and warms up;

[0109] Step 3.1: Transform the environmental information into the form required by the actor network in Step 3.2, and obtain the actor network output and the actions of the fleeing agent, thereby constructing A. t After interacting with the environment, the agent obtains a new state S. t+1 and task end identifier D t , will (S t A t ,R t ,S t+1 D t Store it in the ReplayBuffer.

[0110] Step 3.2: Repeat step 6.2 until the amount of data stored in the ReplayBuffer is greater than the number of preheating samples W. p .

[0111] Step 4: Update the neural network parameters.

[0112] Step 4.1: If the current training step idx equals the round length L ep If so, the environment will be reset.

[0113] Step 4.2: If the remainder between the current training step number idx and the update interval U is 0, then update the network; otherwise, the agent generates a data tuple (S,A,R,S′,D) based on the current interaction between the actor network and the environment, and stores it in the ReplayBuffer. The update process of the actor network and the center commentator network is as follows:

[0114] Step 4.2.1: Randomly sample N from the ReplayBuffer BS A training sample can be represented as (S) e A e ,R e ,S′e D e This training sample data is then transformed into the observation state of the central commentator network as specified in step 3. Observation status of actor network

[0115] Step 4.2.2: Using the Bellman update formula, calculate the target state of the central commentator network. The mathematical expression is as follows:

[0116]

[0117] Among them, A′ e By Target Actor Network The input data of the target actor network is generated through... The construct represents the action to be taken by all the trapping agents in the next state based on the state information.

[0118] Step 4.2.3: Update the central commentator network using the minimum mean squared error as the loss function. The mathematical expression is as follows:

[0119]

[0120] Step 4.2.4: Update the policy gradient of the actor network for the i-th agent, the mathematical expression of which is shown below:

[0121]

[0122] Step 4.2.5: Perform a soft update on the target network parameters:

[0123] θ′ i =τθ i +(1-τ)θ′ i

[0124]

[0125] Step 4.2.6: Store the generated data tuple (S,A,R,S′,D) into the ReplayBuffer;

[0126] Step 4.2.7: Repeat steps 4.2.2 to 4.2.6 until all the actors' networks of the capture agents have been updated.

[0127] Step 5: Repeat step 4 until the training rounds N are completed. ep Ultimately, this resulted in a network of trained actors.

[0128] Specifically, to verify the effectiveness of this invention, a deep reinforcement learning multi-agent cooperative encirclement method with a temporal feature extraction mechanism proposed in this invention will be compared with the classic MADDPG algorithm in the simple_tag scenario of the open-source multiagent-particle-envs simulation environment provided by OpenAI. The experimental environment parameters are shown in Table 1, and the algorithm-related parameters are shown in Table 2.

[0129] Table 1 Simulation Scene Parameters

[0130]

[0131] Table 2 Algorithm Hyperparameters

[0132]

[0133]

[0134] according to Figure 3 Data analysis shows that, under the training environment and algorithm hyperparameters defined in Tables 1 and 2, this invention outperforms the traditional MADDPG algorithm in terms of convergence speed, capture success rate, and task completion efficiency. During 5000 training rounds, this invention achieved convergence around the 3000th round, while the standard MADDPG did not show a significant convergence trend throughout the entire training process. Furthermore, after convergence, the capture success rate of this invention consistently exceeded 0.8, with an average of approximately 75 steps completed per round, demonstrating a significant performance improvement compared to MADDPG. This outstanding performance is attributed to the temporal feature extraction mechanism integrated into the central commentator network, which effectively improves the algorithm's learning efficiency, capture success rate, and overall task execution efficiency.

[0135] Finally, it should be noted that the above embodiments are only used to describe the technical solutions of the present invention and not to limit the technical methods. The present invention can be extended to other modifications, variations, applications and embodiments, and therefore all such modifications, variations, applications and embodiments are considered to be within the spirit and teachings of the present invention.

[0136] Example 2

[0137] A deep reinforcement learning multi-agent cooperative trapping system with a temporal feature extraction mechanism, wherein the trapping system uses the deep reinforcement learning multi-agent cooperative trapping method described above, and the trapping system includes...

[0138] Construction module: Initialize environmental information, determine the agent's escape strategy, determine the observation space of the actor network and the central commentator network, construct the reward function, and construct the conditions for completing the encirclement task;

[0139] The MADDPG architecture was constructed, which included a central commentator network with a temporal feature extraction mechanism and an actor network.

[0140] The information interaction and update module allows the intelligent agent to interact with the environment, collect data, and perform preheating; update the neural network parameters; and repeatedly update the neural network parameters until the training rounds are completed, ultimately obtaining a trained actor network to achieve multi-agent collaborative capture.

[0141] Furthermore, the observation space of the construction module is specifically as follows:

[0142] The observation space of the actor network and the central commentator network is constructed as follows:

[0143] The observation space of the i-th actor network is,

[0144]

[0145] in Let be the absolute position and absolute velocity of the i-th capture agent in the inertial coordinate system. This represents the relative position of the i-th hunting agent with other hunting agents, hunting targets, or obstacles.

[0146] The central commentator network observation space is

[0147] Its dimension is Q critic ,

[0148] in and These represent the absolute coordinates and absolute velocity in the inertial coordinate system, respectively. This represents the action of the i-th capture agent. To extract temporal features, the central commentator needs to use t during training. W The observations at each time step are stacked to form a tensor, so the data taken at time t will be stored as...

[0149]

[0150] Furthermore, the reward function of the construction module is specifically as follows:

[0151] The reward function for the capture mission is as follows:

[0152]

[0153] To encourage cooperation among multiple agents, the reward function evaluates the performance of all agents rather than just a single agent. Here, k1, k2, k3, and k4 represent the weights of different rewards, and P... HIt represents the set of position coordinates of all intelligent agents; The distance between the center of the multi-agent system and the fleeing agent; This represents the terminal reward; when the multi-agent system completes the encirclement task, it will receive a substantial reward. The capture reward is given when the distance between a single agent and the fleeing agent is less than a threshold d. cap When the target is captured, it is considered successfully captured.

[0154] Furthermore, the MADDPG architecture constructed in the construction module specifically involves first determining the meaning of the parameters; secondly, constructing the actor network and the central commentator network respectively; and finally, determining the training hyperparameters and constructing a data cache.

[0155] Furthermore, to improve network stability, this invention enables a target network in the MADDPG architecture and updates the target network's parameters using a soft update after each update. For ease of description, the parameter meanings are defined as follows: the actor network for the i-th agent and the target actor network are defined as follows: and Where θ and θ′ are the learnable parameters of the actor network and the target actor network, respectively;

[0156] Define the actor network and the target actor network as follows: and in and These are the learnable parameters of the central commentator network and the target central commentator network, respectively.

[0157] The construction of the actor network specifically involves a multi-layer perceptron (MLP) consisting of two fully connected layers and a normalized layer group activated by the ReLU function. The actor network can be represented as follows, where θ represents the actor network parameters.

[0158]

[0159] The construction of the central commentator network specifically comprises a bidirectional gated recurrent unit (Bi-GRU) network, a residual self-attention mechanism, and an MLP; the commentator network is used to evaluate the state... The effect of performing action a For ease of expression, the central commentator network will be defined in the following form:

[0160]

[0161] In this context, Q represents the central commentator network's assessment of state-action pairs. The assessment Representatives of the Central Commentators Network, Parameters of the network of commentators at the agency center.

[0162] The specific structure of the central commentator network is as follows:

[0163] The training data, stacked through time windows, is processed by Bi-GRU to obtain the temporal features of the data within the time windows. The output can be represented as follows:

[0164] h t =Bi-GRU(X)

[0165] in, express and A t Merge along the last dimension into a new quantity.

[0166] The information contained in the forward and backward directions in the time dimension, where H represents the dimension of the hidden layer in Bi-GRU.

[0167] The time feature h t The feature vector e is obtained by using a dot product self-attention mechanism. The specific process is as follows:

[0168] First, for each input h t Three different representations are obtained through three different linear transformations: query q i , key k i Sum of values ​​v i ;

[0169] Specifically, for each element h t

[0170] q i =W q h t ,k i =W k h t ,v i =W v h t

[0171] Among them W q W k and W v It is a learnable weight matrix; secondly, scaled dot product attention is used to compute q for each query. i With all keys k j The similarity between them is used to obtain the attention score e. ijThe calculation formula is as follows:

[0172]

[0173] Next, apply the softmax function to all scores e for each i. ij Normalization is performed to obtain the attention weights:

[0174]

[0175] Finally, apply the obtained attention weights to all values ​​v i Perform a weighted average to obtain the final attention output:

[0176]

[0177] Tensors output by the self-attention mechanism O is obtained by taking the mean along the dimension containing the time window. mean ;

[0178] Then, O mean Input an embedding layer and transform its feature dimensions into Q. critic :

[0179]

[0180] Output O to the embedding layer e and The sums form the residual structure, which is then input into the MLP to obtain the output of the commentator network. Its mathematical expression is shown below:

[0181]

[0182] Furthermore, the specific function of the information interaction and update module is to:

[0183] The environmental information is transformed into the form required by the actor network, and the output of the actor network and the actions of the fleeing agent are obtained, thus constituting A. t After interacting with the environment, the agent obtains a new state S. t+1 and task end identifier D t , will (S t A t ,R t ,S t+1 D t Store in ReplayBuffer;

[0184] Repeat the process until the amount of data stored in the ReplayBuffer exceeds the number of preheating samples W. p .

[0185] Furthermore, the specific function of the information interaction and update module is to:

[0186] If the current training step number idx is equal to the round length L ep If so, the environment will be reset;

[0187] If the remainder between the current training step idx and the update interval U is 0, then the network is updated; otherwise, the agent generates a data tuple (S,A,R,S′,D) based on the current interaction between the network and the environment, and stores it in the ReplayBuffer.

[0188] Example 3

[0189] This invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and processor are connected via a bus. Specifically, the processor implements any of the steps in Embodiment 1 by running the computer program stored in the memory.

[0190] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0191] Memory may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory may also include non-volatile random access memory.

[0192] It should be understood that if the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0193] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0194] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0195] It should be noted that the methods and detailed examples provided in the above embodiments can be incorporated into the apparatus and devices provided in the embodiments, and can be referred to each other, without further elaboration.

[0196] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0197] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning multi-agent cooperative trapping method with a temporal feature extraction mechanism, characterized in that, The multi-agent cooperative trapping method specifically includes the following steps: Step 1: Initialize environmental information, determine the agent's escape strategy, determine the observation space of the actor network and the central commentator network, construct the reward function, and construct the conditions for completing the encirclement task; Step 2: Construct the MADDPG architecture, which includes a central commentator network with a time feature extraction mechanism and an actor network; Step 3: The intelligent agent interacts with the environment, collects data, and performs preheating; Step 4: Update the neural network parameters; Step 5: Repeat step 4 until the required number of training rounds are completed, and finally obtain the trained actor network to achieve multi-agent cooperative encirclement; Step 2, constructing the MADDPG architecture, specifically involves first determining the meaning of the parameters; secondly, constructing the actor network and the central commentator network respectively; and finally, determining the training hyperparameters and constructing a data cache. Specifically, the meaning of the parameters is defined as follows: The first... The actor network and the target actor network of each agent are as follows: and ,in and These are the learnable parameters of the actor network and the target actor network, respectively. Define the central commentator network and the target central commentator network as follows: and ,in and These are the learnable parameters of the central commentator network and the target central commentator network, respectively. The actor network is constructed as follows: it consists of a multilayer perceptron, which includes two fully connected layers and a group of normalized layers activated by the ReLU function. The actor network can be represented as follows, where... Representative actor network parameters: ； The central commentator network is constructed by consisting of a bidirectional gated recurrent unit network, a residual self-attention mechanism, and an MLP. The commentator network is used to evaluate the state... Execute action Effect For ease of expression, the central commentator network will be defined in the following form: in, Representatives from the central commentator network commented on the state-action pair. The assessment Representatives of the Central Commentators Network, Parameters of the network of commentators at the agency center; The multiple agents can be one or more of the following: unmanned aerial vehicles, ground robots, and underwater vehicles.

2. The deep reinforcement learning multi-agent cooperative trapping method according to claim 1, characterized in that, The observation space in step 1 is specifically as follows: The observation space of the actor network and the central commentator network is constructed as follows: No. The observation space of an actor network is, ， in For the first The absolute position and absolute velocity of each hunting agent in the inertial coordinate system. Then it represents the first The relative position of an individual hunting agent with other hunting agents, hunting targets, or obstacles; The central commentator network observation space is: Its dimensions are , in and These represent the absolute coordinates and absolute velocity in the inertial coordinate system, respectively; in order to extract time features, the central commentator needs to use these parameters during training. The observations at each time step are stacked to form a tensor, so at the th time step... The data used at each moment will be stored as: 。 3. The deep reinforcement learning multi-agent cooperative trapping method according to claim 2, characterized in that, The reward function constructed in step 1 is specifically as follows: The reward function for the capture mission is as follows: in, These represent the weight of different rewards. The set representing the position coordinates of the intelligent agent; The distance between the center of the multi-agent system and the fleeing agent; This represents the terminal reward; when the multi-agent system completes the encirclement task, it will receive a substantial reward. The capture reward is given when the distance between a single agent and the fleeing agent is less than a threshold. When the target is captured, it is considered successfully captured.

4. The deep reinforcement learning multi-agent cooperative trapping method according to claim 1, characterized in that, Step 3 Includes the following steps, Step 3.1: Transform the environmental information into the form required by the actor network, and obtain the actor network output and the actions of the fleeing agent, thereby constituting... After interacting with the environment, the agent obtains a new state. and task end identifier ,Will Store in ReplayBuffer; Step 3.2: Repeat step 3.1 until the amount of data stored in the ReplayBuffer is greater than the number of preheating samples. .

5. The deep reinforcement learning multi-agent cooperative trapping method according to claim 4, characterized in that, Step 4 includes the following steps: Step 4.1: If the current training steps Equal to the length of the round If so, the environment will be reset; Step 4.2: If the current training steps With update interval If the remainder between the two is 0, the network is updated; otherwise, the agent generates data tuples based on the interaction between the current actor network and the environment. And store it in the ReplayBuffer.

6. A deep reinforcement learning multi-agent cooperative trapping system with a temporal feature extraction mechanism, characterized in that, The trapping system uses the deep reinforcement learning multi-agent cooperative trapping method as described in any one of claims 1-5, the trapping system comprising: Construction module: Initialize environmental information, determine the agent's escape strategy, determine the observation space of the actor network and the central commentator network, construct the reward function, and construct the conditions for completing the encirclement task; The MADDPG architecture was constructed, which included a central commentator network with a temporal feature extraction mechanism and an actor network. The information interaction and update module allows the intelligent agent to interact with the environment, collect data, and perform preheating; update the neural network parameters; and repeatedly update the neural network parameters until the training rounds are completed, ultimately obtaining a trained actor network to achieve multi-agent collaborative capture.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images