Multi-agent cooperative decision-making method based on improved MADDPG

Abstract

The application discloses a kind of multi-agent collaborative decision-making methods based on improved MADDPG, belong to multi-agent multi-task allocation technical field, in this method including that reward function is designed, makes agent form hunting formation;A kind of layered multi-agent collaborative framework is designed;Finally, the priority experience replay mechanism is added to MADDPG algorithm and the decay factor is added under OU noise noise to optimize exploration mechanism, while the experience pool in the priority experience replay mechanism is divided into ordinary experience pool and high-reward round experience pool, and the experience sampling number is introduced into experience importance weight calculation formula.The method can effectively solve the problem of multi-agent multi-task allocation.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent multi-task allocation technology, and more specifically relates to a multi-agent cooperative decision-making method based on an improved MADDPG. Background Technology

[0002] In the future combat scenarios envisioned by current new combat concepts, traditional multi-mission platforms are broken down into numerous small combat units. These small combat units typically possess greater flexibility, enabling them to quickly adjust their assigned tasks according to changes in the battlefield environment to achieve better overall combat effectiveness. In these future combat scenarios, the traditional centralized command and control model suffers from problems such as excessively long command chains and overly complex decision-making processes, resulting in difficulties in meeting requirements for decision-making timeliness and quality.

[0003] In recent years, new command and control models such as edge command and control have emerged, giving edge nodes, i.e., various combat entities, a certain degree of autonomous decision-making capability. Due to the complex and ever-changing nature of the battlefield environment, and the trend towards miniaturization and intelligence of combat entities, distributed decision-making will play an increasingly important role in future battlefield decision-making. However, the increasing number of combat targets leads to a dramatic increase in the dimensionality of the combat situation space. Traditional methods often suffer from long response times and low accuracy when controlling large-scale systems, failing to meet practical needs. Furthermore, multi-agent collaboration faces convergence problems, hindering the achievement of better multi-agent collaborative effects and the effective resolution of multi-agent multi-task allocation issues. Summary of the Invention

[0004] In view of the above-mentioned technical problems, the present invention provides a multi-agent cooperative decision-making method based on the improved MADDPG (Multi-Agent DeepDeterministic Policy Gradient). The purpose is to solve the problem of convergence difficulty in multi-agent cooperation when the number of agents is large, and to effectively solve the problem of multi-agent multi-task allocation.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a multi-agent cooperative decision-making method based on an improved MADDPG, the method comprising the following steps:

[0007] S1: Establish a relative coordinate system with the target position as the origin, determine the relative coordinates of the agent in the relative coordinate system, and set constraints to ensure that the agent does not collide.

[0008] S2: Construct a Markov model for multi-agent cooperative encirclement of multiple targets;

[0009] S3: Design the total reward function for the agent, enabling the agent to locate the target and form an encirclement formation;

[0010] S4: Formulate multi-agent collaborative decision-making to achieve efficient allocation of multiple tasks among multiple agents, enabling multiple agents to collaboratively capture multiple targets.

[0011] Furthermore, in S1, the agent in the relative coordinate system exist Relative coordinates of time for:

[0012]

[0013] in, The x-axis represents the relative distance. The vertical axis represents the relative distance. For the agent's coordinates, The target coordinates; superscript Represents the transpose of a matrix;

[0014] The constraint formula that ensures the agent does not collide is:

[0015]

[0016] in, For intelligent agents and intelligent agents exist Relative distance at any given moment; For intelligent agents and intelligent agents The sum of radii.

[0017] Furthermore, in S2, the action space of all agents is set to be two-dimensional; the action of each agent is... and The velocities in the two directions are respectively and intelligent agent The action is represented as In having In a scenario with multiple agents performing a task, the joint action of all agents is represented as follows: ;

[0018] intelligent agent The observation information includes:

[0019] 1) Intelligent agent Action information and location information;

[0020] 2) Intelligent Agent Relative distance to other intelligent agents;

[0021] 3) Intelligent Agent The angle between the agent and its neighboring agents.

[0022] Furthermore, in S3, the agent's total reward function is:

[0023]

[0024] in, Represents intelligent agents exist The total reward function at time step 1 For intelligent agents exist The distance reward function at time step. For intelligent agents exist The formation reward function at any given moment. For intelligent agents exist The collision avoidance reward function at any given time. For intelligent agents exist The external reward function at time step; , , and They represent , , and The weighting coefficients.

[0025] Furthermore, in S4, multi-agent cooperative decision-making based on the improved MADDPG is formulated, specifically including:

[0026] (1) Constructing a hierarchical multi-agent collaborative framework:

[0027] The overall architecture is divided into three layers. The top layer is the central service node, which is responsible for multi-agent task allocation and the centralized training process of the Leader agent. The middle layer consists of Leader agents in each task area. The Leader agents participate in the task of coordinating the capture of the target and are also responsible for the centralized training process of the Slav agents in their respective task areas. The bottom layer consists of Slav agents, which are responsible for the task of coordinating the capture of the target and transmitting action information and observation information to their respective Leader agents.

[0028] (2) Add a priority experience replay mechanism to the MADDPG algorithm:

[0029] The core of the priority experience replay mechanism is to determine the standard for measuring the importance of each experience. The indicator for measuring the importance of an experience is the time series difference error, and the formula for calculating the time series difference error is:

[0030]

[0031] in, Indicates timing difference error. Represents intelligent agents Rewards earned from performing actions; Indicates the discount factor; Indicate target value, Indicates the current value; Indicates that agent i is used for computation. Neural networks of values; Represents intelligent agents The state of the action being performed; Represents intelligent agents The state observed after the action is performed; Represents intelligent agents The current action; Represents intelligent agents The next action; This represents the agent's action strategy at the current moment; This represents the agent's next action strategy;

[0032] To ensure that the priority of experiences and the probability of them being sampled are monotonic, while guaranteeing that the probability of all experiences being sampled is non-zero, the probability of an experience being sampled is calculated using the following formula:

[0033]

[0034] in, Indicates the first The probability that an experience point is sampled. Indicates sample priority. Indicates limited scheduling capacity, when A value of 0 indicates uniform sampling; Indicates the first The probability that a piece of experience will be sampled;

[0035] And calculate the importance sampling weights. The calculation formula is:

[0036]

[0037] in, This represents the sample size in the experience pool. It's a hyperparameter. Used for normalization, it is the sample with the largest weight among all samples;

[0038] (3) Construct a priority experience replay buffer pool:

[0039] Construct a priority experience replay buffer pool that includes a normal experience pool and a high-reward round experience pool. During the experience pooling phase, the normal experience pool stores the experience generated in all rounds and calculates the important new sampling weight of the experience based on the temporal difference error; the high-reward round experience pool stores the experience that appears in new rounds that are higher than the current known highest round cumulative reward and updates the known highest round cumulative reward.

[0040] And calculate the probability of the empirical sampling frequency weight decreasing, and incorporate the empirical sampling frequency into the calculation of the importance sampling weight;

[0041] (4) Construct an optimization algorithm under OU noise:

[0042] OU noise is added to the MADDPG algorithm, and an attenuation factor that gradually decreases over time is added to control the intensity of the OU noise. The formula for calculating the attenuation factor is:

[0043]

[0044] in, Indicates the attenuation factor. This represents the total number of training rounds. This indicates that the current number is the [number]. One round of training, This is the offset.

[0045] Secondly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described multi-agent cooperative decision-making method based on the improved MADDPG.

[0046] Compared with the prior art, the present invention provides a multi-agent cooperative decision-making method based on improved MADDPG, which has at least the following beneficial technical effects:

[0047] This invention, through the use of multi-agent multi-task allocation technology and an improved MADDPG algorithm, achieves better multi-agent cooperative results in multi-agent cooperative encirclement scenarios, effectively solving the multi-agent multi-task allocation problem. In multi-agent autonomous cooperative encirclement of multiple targets, this invention improves performance in terms of convergence speed and average round reward.

[0048] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0049] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0052] Figure 1 This is a schematic diagram of a multi-agent encirclement and capture multi-target environment provided in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall process of a multi-agent collaborative decision-making method based on an improved MADDPG, provided for an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram of the multi-agent collaborative decision-making design process provided in an embodiment of the present invention.

[0055] Figure 4 This is a schematic diagram of a hierarchical multi-agent collaborative framework provided in an embodiment of the present invention.

[0056] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0058] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0059] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0060] This invention provides a multi-agent cooperative decision-making method based on an improved MADDPG, which mainly includes:

[0061] (1) Design a reward function so that the agent can quickly locate the target and adjust its behavior according to the position of other agents to form a capture area in coordination, and ensure that it avoids collisions with other agents during flight;

[0062] (2) To address the problem of convergence difficulty due to the large number of agents and the excessive dimensionality of observation information, the steps include:

[0063] a) A hierarchical multi-agent cooperative framework is adopted. In the scenario of multi-agent cooperatively capturing multiple targets, there is a central service node (Center Node). Tasks are divided among agents according to a task allocation algorithm, with agents in different task regions capturing different targets. Within each task region, a Leader agent is determined based on the agents' computational capabilities, and the other agents are Slaves. Agents in the same task region share the same target, and their actions and positional information are more valuable for observation. The Leader agent, due to its relatively stronger computational capabilities, is responsible for the centralized training process of the Slaves in its task region.

[0064] b) The proposed priority experience replay mechanism can improve the utilization rate of experience to a certain extent and accelerate the convergence of the algorithm; the concept of round control is introduced to improve the priority experience replay mechanism so that the agent can learn the optimal strategy faster.

[0065] c) Two priority experience replay buffer pools were designed, namely the Single Pool and the Episodic Pool.

[0066] d) Define a formula for calculating the probability of decreasing the weight of experience sampling frequency. This formula overcomes the overfitting problem that may occur when prioritizing experience replay in cases where the importance of experience samples is highly polarized. It introduces the number of times experience is sampled into the calculation of importance sampling weight to avoid the situation where a single experience is repeatedly learned while other experiences are not.

[0067] (3) OU noise is added to the MADDPG algorithm to ensure that the agent explores within a certain range without causing drastic changes in the agent's strategy. An attenuation factor that gradually decreases over time is added to the OU noise to control the intensity of the OU noise.

[0068] The specific embodiments and working principles of the present invention will be described in detail below with reference to the accompanying drawings:

[0069] Figure 1 For a multi-agent hunting environment with multiple targets, specifically, it consists of N agents and M targets (e.g., 8 agents (drones) and 2 targets). After task allocation, the N agents are divided into M groups of agents. Agents in different groups are responsible for approaching and surrounding the M targets, and need to avoid collisions between agents.

[0070] For multi-agent, multi-target encirclement environments, and to address the challenge of implementing centralized decision-making models in complex future battlefield environments due to communication limitations, see [reference needed]. Figure 2 As shown, this embodiment of the invention provides a multi-agent cooperative decision-making method based on an improved MADDPG, specifically including the following steps:

[0071] Step 1, establish a relative coordinate system with the target location as the origin:

[0072] Specifically, in this relative coordinate system, the intelligent agent exist The relative coordinates at each moment are:

[0073]

[0074] in, The x-axis represents the relative distance. The vertical axis represents the relative distance. For the agent's coordinates, The target coordinates are shown.

[0075] intelligent agent and The positive direction of the axis is Angle of time The calculation formula is:

[0076]

[0077] intelligent agent With the goal exist The relative distance at any given time is The calculation formula is:

[0078]

[0079] The circular encirclement problem can be solved by intelligent agents. and intelligent agents Description of the included angle, included angle The calculation formula is:

[0080]

[0081] in, , Representing intelligent agents respectively Planar angle and intelligent agent The plane angle;

[0082] The circular encirclement problem is described as follows: When the angles between each agent and its neighboring agents satisfy the following conditions:

[0083]

[0084] And the distance to the target The radius of the circular enclosure At the same time, the following formula satisfies the constraint that the agent does not collide: At that time, multiple agents conducted a comprehensive manhunt for the target, among which... For intelligent agents and intelligent agents The sum of radii.

[0085] Step 2, construct a Markov model for the multi-agent cooperative encirclement of multiple objectives:

[0086] This invention studies the multi-agent trapping problem in a two-dimensional plane; therefore, the action space of all agents is two-dimensional. Each agent's action is... and Velocity in two directions, intelligent agent The action is represented as In a scenario with N agents performing tasks, the joint action of all agents is represented as:

[0087]

[0088] In the training process of a multi-agent cooperative encirclement and capture of multiple targets, the observation information of the agent regarding the environment should not only include its own action information. The actions and related positional information of other agents and targets are also crucial to the completion of the encirclement and capture task. Therefore, in this embodiment, the agent... The observation information consists of the following three parts:

[0089] 1) Intelligent agent Action information and location information, i.e. and .

[0090] 2) Settings Represents intelligent agents With intelligent agents Or the relative distance to other targets, the agent The observation information includes intelligent agents The relative distance to other intelligent agents, i.e.:

[0091]

[0092] in, Represents intelligent agents The relative distance to other intelligent agents.

[0093] 3) Observational information related to the encirclement mission, i.e., the intelligent agent and intelligent agents The included angle .

[0094] As the number of agents in an environment increases, the dimensionality of the agents' observation information grows linearly. This information is then pieced together to form the agent's data. The observation information is represented as:

[0095]

[0096] in, Represents intelligent agents Observational information.

[0097] Step 3, construct the agent's total reward function:

[0098] The design of the reward function during the training phase is crucial, determining whether the agent can locate the target, adjust its behavior based on the positions of other agents to collaboratively form a capture zone, and avoid collisions with other agents during flight. The specific design is as follows:

[0099] 1) Distance reward function :

[0100]

[0101] The distance reward function represents the difference between agents and targets performing the same task. exist Average relative distance at time The closer to the encirclement radius The greater the reward, the higher the final reward. convergence to equal.

[0102] 2) Formation reward function :

[0103]

[0104] This indicates that the closer the relative angle between adjacent agents is to the bisecting circle, the better. portion The larger the target, the greater the reward, eventually forming a circular formation to surround and capture it. Represents intelligent agents and intelligent agents The angle between them; This represents the angle after a circle is divided into N equal parts;

[0105] 3) Collision avoidance reward function :

[0106]

[0107] This indicates the distance between two agents. Less than or equal to the sum of the radii of the two agents If a collision occurs, the agent is penalized to ensure a safe distance between agents during training.

[0108] 4) External reward function :

[0109]

[0110] Indicates when performing the same task A higher average reward for each agent indicates better collaborative performance. Each agent should be given a certain external reward, namely, a team reward to evaluate the collaborative effect.

[0111] 5) Total Reward Function :

[0112] intelligent agent exist The total reward function at time step is the distance reward function. Formation reward function Collision avoidance reward function The sum of the weighted coefficients is expressed by the formula:

[0113]

[0114] in, , , and They represent , , and The weighting coefficients.

[0115] Step 4, formulate multi-agent collaborative decision-making:

[0116] To formulate an efficient and robust multi-agent cooperative strategy, this embodiment adopts a multi-agent cooperative decision-making approach based on an improved MADDPG. (See [link to relevant documentation]). Figure 3 As shown, the specific steps include:

[0117] Step 401, Construct a hierarchical multi-agent collaborative framework:

[0118] Specifically, Figure 4 The diagram illustrates a hierarchical multi-agent collaborative framework. A hierarchical multi-agent deep reinforcement learning framework is proposed, where the number of agents performing the same task is (the diagram is missing in the original text). The observation information of an agent only includes its distance to other agents performing the same task, that is:

[0119]

[0120] The overall architecture consists of three layers. The top layer is the Central Service Node, responsible for multi-agent task allocation and the centralized training process of the Leader agents. The Central Service Node determines whether to perform centralized training for the Leader agents based on the distance between different targets of different Leader agents. The Leader agents incorporate distance information from other Leader agents into their environmental observations. This design is based on the premise that if the targets are close together, the similarity between the two capture tasks is high. Since the Leader agents are not trained centrally within their own task areas, but rather centrally within the Central Service Node, the convergence speed of the training can be effectively improved. The middle layer consists of Leader agents in each task area. These Leader agents also participate in the collaborative target capture task and, leveraging their higher computing power, are responsible for the centralized training process of the Slaver agents in their respective task areas. The bottom layer consists of Slaver agents, which are responsible for coordinating the capture of the target and transmitting action and observation information to their respective Leader agents. Through centralized training, the agents are trained based on all action and observation information of the agents in the same task, thereby enhancing the coordination effect between agents.

[0121] Step 402, Construct a priority experience replay mechanism:

[0122] Specifically, a priority experience replay mechanism is proposed. The core of this mechanism is determining the standard for measuring the importance of each experience. The more we learn from an experience, the more important it is. Therefore, a key indicator for measuring the importance of an experience is the temporal difference error (TD-error), calculated as follows:

[0123]

[0124] in, This represents the timing difference error (TD-error). Represents intelligent agents Rewards earned from performing actions; Indicates the discount factor; Indicate target value, Indicates the current value; Represents intelligent agents Used for calculation Neural networks of values; Represents intelligent agents The state of the action being performed; Represents intelligent agents The state observed after the action is performed; Represents intelligent agents The current action; Represents intelligent agents The next action; This represents the agent's action strategy at the current moment; This represents the agent's next action strategy;

[0125] TD-error indicates the current Values ​​and Objectives The difference between these values, i.e., how surprising an experience is (i.e., unexpected), necessitates assigning greater weight to experiences with larger TD-error values, increasing their probability of being sampled. To ensure that the priority of experiences and their sampling probability are monotonic, while guaranteeing that the probability of all experiences being sampled is non-zero, the formula for calculating the probability of an experience being sampled is defined as follows:

[0126]

[0127] in, Indicates the first The probability that an experience point is sampled. Indicates sample priority. Indicates limited scheduling capacity, when A value of 0 indicates uniform sampling; Indicates the first The probability that an experience point will be sampled. Sample priority. Directly based on Confirmed, the calculation formula is:

[0128]

[0129] in, Random numbers are used to prevent probabilities from being zero. Using priority empirical replay will change the sample distribution; therefore, the importance sampling weight calculation formula is designed as follows:

[0130]

[0131] in, This represents the sample size in the experience pool. These are hyperparameters that determine the degree to which prioritizing empirical replay affects convergence. Time-priority empirical replay does not affect the convergence result. The sample used for normalization is the one with the largest weight among all samples. Therefore, the loss function of the MADDPG algorithm, which prioritizes empirical replay, is used. for:

[0132]

[0133] in, This indicates the number of agents participating in the capture operation. For the target of the sample value, Indicates the weight.

[0134] Step 403, construct the priority experience replay buffer pool:

[0135] Specifically, this includes a Single Pool and an Episodic Pool. To improve experience utilization efficiency, a round-control approach is adopted to improve the priority experience replay mechanism. During the experience entry phase, the Single Pool stores the experience generated in all rounds and calculates the significance of new sampling weights for experience based on TD-error. The Episodic Pool stores the experience that appears in new rounds with a higher cumulative reward than the current known highest round, and updates the known highest round cumulative reward. During the experience replay phase, a random number in the range [0,1] is generated. When the random number is in the range [0,0.6], a batch of samples B is selected from the Single Pool to update the network parameters; when the random number is in the range [0.6,1], a batch of samples B is selected from the Episodic Pool to update the network parameters.

[0136] To overcome the overfitting problem that may arise from prioritizing experience replay when the importance of experience samples is severely polarized, the number of times an experience is sampled is incorporated into the calculation of the importance sampling weight. This avoids the situation where a single experience is repeatedly learned while other experiences are not. The probability formula for calculating the decrease in the weight of experience sampling frequency is defined as follows:

[0137]

[0138] in, Indicates the first The number of times an experience is sampled in this round (episodes). This represents the weighted descent scheduling factor, when A value of 0 indicates that the weights decrease uniformly. Indicates the first round in this round The probability of an experience sample being included in the number of experience samples is calculated using the following formula:

[0139]

[0140] in, This represents the probability that the empirical sampling frequency weight decreases.

[0141] Step 404, construct the optimization algorithm under OU noise:

[0142] Specifically, since the agent's cognitive ability regarding the environment is limited in the early stages of training, it is necessary to improve its exploration ability. Therefore, an attenuation factor that gradually decreases over time is added to the OU noise to control its intensity. As training progresses, the value of the attenuation factor gradually decreases, and the noise intensity also decreases accordingly. This gives the agent a strong exploration ability in the early stages of training. As training gradually reaches the middle and late stages, the agent's cognitive ability regarding the environment matures, gradually reducing the interference of noise and focusing more on learning stable strategies, causing the training to converge. This can avoid overexploration or overexploitation during training, improve the performance of the algorithm, and make the impact of OU noise on training smaller and smaller.

[0143] Therefore, this invention proposes an improved exponentially decreasing decay factor. To solve the above-mentioned problems, The calculation formula is designed as follows:

[0144]

[0145] in, This represents the total number of training rounds. This indicates that the current number is the [number]. Training for each episode Let be the offset. The optimized OU noise is expressed as:

[0146]

[0147] Where U represents the OU noise value, N ou This represents a Gaussian random number.

[0148] The MADDPG algorithm adds an attenuation factor to the OU noise to gradually reduce the noise intensity during training, so that the agent's exploration ability gradually decreases as training progresses. The MADDPG algorithm is more stable and efficient during training and eventually converges to a stable policy.

[0149] The present invention uses the above-mentioned multi-agent collaborative decision-making, which can converge quickly and ensure high accuracy, making it easy to meet the requirements of multi-UAV mission decision-making in combat and realize mission decision-making for multiple UAVs facing multiple targets in dynamic combat environments.

[0150] As described in the above embodiments, those skilled in the art will understand that the present invention provides a multi-agent cooperative decision-making method based on an improved MADDPG, including designing a reward function to enable agents to form a containment formation; designing a hierarchical multi-agent cooperative framework; and finally, adding a priority experience replay mechanism to the MADDPG algorithm and adding a decay factor under OUnoise noise to optimize the exploration mechanism. Furthermore, the experience pool in the priority experience replay mechanism is divided into a normal experience pool and a high-reward round experience pool, and the number of times experience is sampled is incorporated into the experience importance weight calculation formula. This method can effectively solve the multi-agent multi-task allocation problem.

[0151] Additionally, refer to Figure 5 As shown, this embodiment of the invention also provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10. The processor executes the computer program to implement a multi-agent cooperative decision-making method based on an improved MADDPG according to the above method embodiment.

[0152] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0153] The memory 11 may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (RAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, and any suitable combination thereof.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, electronic devices, or computer program products, etc. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0157] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-agent cooperative decision-making method based on an improved MADDPG, characterized in that, The method includes the following steps: S1: Establish a relative coordinate system with the target position as the origin, determine the relative coordinates of the agent in the relative coordinate system, and set constraints to ensure that the agent does not collide. S2: Construct a Markov model for multi-agent cooperative encirclement of multiple targets; S3: Design the total reward function for the agent, enabling the agent to locate the target and form an encirclement formation; S4: Formulate multi-agent collaborative decision-making to achieve efficient allocation of multiple tasks among multiple agents, enabling multiple agents to collaboratively capture multiple targets; In S4, a multi-agent cooperative decision-making system based on the improved MADDPG is formulated, specifically including: (1) Constructing a hierarchical multi-agent collaborative framework: The overall architecture is divided into three layers. The top layer is the central service node, which is responsible for multi-agent task allocation and the centralized training process of the Leader agent. The middle layer consists of Leader agents in each task area. The Leader agents participate in the task of coordinating the capture of the target and are also responsible for the centralized training process of the Slav agents in their respective task areas. The bottom layer consists of Slav agents, which are responsible for the task of coordinating the capture of the target and transmitting action information and observation information to their respective Leader agents. (2) Add a priority experience replay mechanism to the MADDPG algorithm: The core of the priority experience replay mechanism is to determine the standard for measuring the importance of each experience. The indicator for measuring the importance of an experience is the time series difference error, and the formula for calculating the time series difference error is: in, Indicates timing difference error. Represents intelligent agents Rewards earned from performing actions; Indicates the discount factor; Indicate target value, Indicates the current value; Represents intelligent agents Used for calculation Neural networks of values; Represents intelligent agents The state of the action being performed; Represents intelligent agents The state observed after the action is performed; Represents intelligent agents The current action; Represents intelligent agents The next action; This represents the agent's action strategy at the current moment; This represents the agent's next action strategy; To ensure that the priority of experiences and the probability of them being sampled are monotonic, while guaranteeing that the probability of all experiences being sampled is non-zero, the probability of an experience being sampled is calculated using the following formula: in, Indicates the first The probability that an experience point is sampled. Indicates sample priority. Indicates limited scheduling capacity, when A value of 0 indicates uniform sampling; Indicates the first The probability that a piece of experience will be sampled; And calculate the importance sampling weights. The calculation formula is: in, This represents the sample size in the experience pool. It's a hyperparameter. Used for normalization, it is the sample with the largest weight among all samples; (3) Construct a priority experience replay buffer pool: Construct a priority experience replay buffer pool that includes a normal experience pool and a high-reward round experience pool. During the experience pooling phase, the normal experience pool stores the experience generated in all rounds and calculates the important new sampling weight of the experience based on the temporal difference error; the high-reward round experience pool stores the experience that appears in new rounds that are higher than the current known highest round cumulative reward and updates the known highest round cumulative reward. And calculate the probability of the empirical sampling frequency weight decreasing, and incorporate the empirical sampling frequency into the calculation of the importance sampling weight; (4) Construct an optimization algorithm under OU noise: OU noise is added to the MADDPG algorithm, and an attenuation factor that gradually decreases over time is added to control the intensity of the OU noise. The formula for calculating the attenuation factor is: in, Indicates the attenuation factor. This represents the total number of training rounds. This indicates that the current number is the [number]. One round of training, This is the offset.

2. The multi-agent cooperative decision-making method based on improved MADDPG according to claim 1, characterized in that, In S1, the agent is in a relative coordinate system. exist Relative coordinates of time for: in, The x-axis represents the relative distance. The vertical axis represents the relative distance. For the agent's coordinates, The target coordinates; superscript Represents the transpose of a matrix; The constraint formula that ensures the agent does not collide is: in, For intelligent agents and intelligent agents exist Relative distance at any given moment; For intelligent agents and intelligent agents The sum of radii.

3. The multi-agent cooperative decision-making method based on improved MADDPG according to claim 1, characterized in that, In S2, the action space of all agents is set to two dimensions; the action of each agent is... and The velocities in the two directions are respectively and intelligent agent The action is represented as In having In a scenario with multiple agents performing a task, the joint action of all agents is represented as follows: ; intelligent agent The observation information includes: 1) Intelligent agent Action information and location information; 2) Intelligent Agent Relative distance to other intelligent agents; 3) Intelligent Agent The angle between the agent and its neighboring agents.

4. The multi-agent cooperative decision-making method based on improved MADDPG according to claim 1, characterized in that, In S3, the agent's total reward function is: in, Represents intelligent agents exist The total reward function at time step 1 For intelligent agents exist The distance reward function at time step. For intelligent agents exist The formation reward function at any given moment. For intelligent agents exist The collision avoidance reward function at any given time. For intelligent agents exist The external reward function at time step; , , and They represent , , and The weighting coefficients.

5. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement a multi-agent cooperative decision-making method based on an improved MADDPG as described in any one of claims 1-4.

Images