A Multi-Agent Landmark Coverage Method and System Based on Deep Reinforcement Learning

Abstract

This disclosure relates to the field of intelligent agent control technology, and proposes a multi-agent landmark coverage method and system based on deep reinforcement learning. To avoid the phenomenon of slow convergence speed in the later stage of agent training, the deep reinforcement learning method is combined with the traditional artificial potential field method. The artificial potential field information is added to the original state space to increase the dimensionality of the state space, which solves the problem of poor decision-making effect of the agent due to partial observation of the environment. Secondly, a reward function is designed by combining the number of steps and relative distance to solve the problem of sparse rewards obtained by the agent in the early stage of exploration, so that the agent can effectively avoid obstacles, and at the same time effectively improve the learning efficiency and training efficiency of the agent, thus improving the efficiency of landmark coverage.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent agent control technology, specifically to a multi-agent landmark coverage method and system based on deep reinforcement learning. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of computer technology and artificial intelligence, multi-agent systems are being applied more and more widely, such as in cooperative navigation and group control. Multi-agent systems can solve complex tasks through cooperation among individual agents. Compared with single-agent systems, their main advantages are: first, better stability—the failure of one agent will not cause the entire system to collapse; second, higher efficiency—decomposing complex tasks into multiple sub-tasks completed by individual agents improves task efficiency; and third, stronger compatibility—the heterogeneous functional distribution of agents in the same working environment expands the perception range and spatiotemporal distribution range, improving compatibility.

[0004] Multi-agent landmark coverage is a typical task in multi-agent systems. It involves a group of agents with perceptual and computational capabilities covering landmarks in an environment along a collision-free path, based on task allocation. Landmark coverage is one of the most common problems in the multi-agent domain, widely applied in logistics, formation flying, and game AI. Research on multi-agent landmark coverage methods is a crucial area of ​​research in multi-agent systems. Integrating deep reinforcement learning into multi-agent systems has given rise to multi-agent deep reinforcement learning, where agents interact with the environment, receive feedback, and adjust their strategies to achieve self-learning.

[0005] The inventors discovered in their research that current multi-agent deep reinforcement learning still has some problems. First, in the exploration stage of a new environment for multi-agents, the reward function is usually sparse, resulting in low learning efficiency. Second, during the interaction between the agent and the environment, the agent is affected by factors such as other agents and movement speed, and single-agent algorithms will have difficulty converging when applied to multi-agent tasks. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a multi-agent landmark coverage method and system based on deep reinforcement learning. First, artificial potential field information is added to the original state space to increase its dimensionality, thus solving the problem of poor decision-making performance caused by partial environmental observations. Second, a reward function is designed by combining the number of steps and relative distance to address the problem of sparse rewards obtained by the agent in the early stages of exploration, while effectively improving training efficiency.

[0007] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0008] One or more embodiments provide a multi-agent landmark coverage method based on deep reinforcement learning, comprising the following steps:

[0009] For multi-agent systems, a multi-agent reinforcement learning framework model based on a deep deterministic policy gradient algorithm is established.

[0010] Adding artificial potential field information to the original state space increases the dimensionality of the state space. Combining the potential field of each state of the agent with the original state space updates the state of the agent itself.

[0011] The reward function is improved by combining the number of steps with the relative distance, and the agent is rewarded or punished according to the distance and number of steps in each state.

[0012] Based on the upgraded state space and the improved reward function, the multi-agent reinforcement learning framework model of the deep deterministic policy gradient algorithm is trained to obtain the updated model parameters.

[0013] The task allocation information for landmark coverage and the state information of the agents are obtained and input into the trained multi-agent reinforcement learning framework model to obtain the optimal motion planning path of the multi-agents.

[0014] One or more embodiments provide a multi-agent landmark coverage system based on deep reinforcement learning, including:

[0015] Model building module: configured to build a multi-agent reinforcement learning framework model based on a deep deterministic policy gradient algorithm for multi-agent systems;

[0016] State space construction module: It is configured to add artificial potential field information to the original state space to increase the dimensionality of the state space, and combine the potential field of each state of the agent with the original state space to update the state of the agent itself.

[0017] Reward setting module: It is configured to combine steps and relative distance to improve the reward function, and to reward or punish the agent in each state based on distance and steps.

[0018] Model training module: configured to train the multi-agent reinforcement learning framework model of deep deterministic policy gradient algorithm to obtain updated model parameters based on the upgraded state space and the improved reward function;

[0019] Path generation module: It is configured to obtain task allocation information for landmark coverage and agent state information, input them into the trained multi-agent reinforcement learning framework model, and obtain the optimal motion planning path for the multi-agent.

[0020] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps described in the above method.

[0021] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps described in the above method.

[0022] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0023] In this disclosure, to avoid the slow convergence speed of the agent in the later stages of training, a deep reinforcement learning method is combined with a traditional artificial potential field method. The artificial potential field information is added to the original state space to increase the dimensionality of the state space, which solves the problem of poor decision-making performance caused by partial observation of the environment. Secondly, a reward function is designed to combine the number of steps and the relative distance to solve the problem of sparse rewards obtained by the agent in the early stage of exploration, which enables the agent to effectively avoid obstacles, while effectively improving the learning efficiency of the agent, improving the training efficiency, and improving the efficiency of landmark coverage.

[0024] The advantages of this disclosure, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0025] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute a limitation thereof.

[0026] Figure 1 This is a multi-agent reinforcement learning framework model based on a deep deterministic policy gradient algorithm, which is the embodiment of this disclosure 1.

[0027] Figure 2 This is a side view of the structural diagram of a single intelligent agent subjected to forces according to Embodiment 1 of this disclosure;

[0028] Figure 3 This is a schematic diagram of multi-agent reinforcement learning in Embodiment 1 of this disclosure;

[0029] Figure 4 This is a detailed diagram of the network structure of the DDPG framework model in Embodiment 1 of this disclosure;

[0030] Figure 5(a) A simple environment built for a land cover task in a multi-particle environment, as shown in the simulation example of Embodiment 1 of this disclosure;

[0031] Figure 5(b) shows the task completion status of a simple environment built for a land cover task in a multi-particle environment of the simulation example of Embodiment 1 of this disclosure after training.

[0032] Figure 6(a) Reward value curve obtained by training the MADDPG algorithm in a simple environment without an artificial potential field in the simulation example of Embodiment 1 of this disclosure;

[0033] Figure 6(b) shows the reward value curve obtained by training the APF-MADDPG algorithm in a simple environment with an artificial potential field added to the simulation example according to Embodiment 1 of this disclosure.

[0034] Figure 7(a) shows a complex environment built for a land cover task in a multi-particle environment, as an example of the simulation of Embodiment 1 of this disclosure;

[0035] Figure 7(b) shows the task completion status of a complex environment built for a land cover task in a multi-particle environment, as a simulation example of Embodiment 1 of this disclosure, after training.

[0036] Figure 8(a) Reward value curve obtained by training the MADDPG algorithm in a complex environment without adding an artificial potential field in the simulation example of Embodiment 1 of this disclosure;

[0037] Figure 8(b) shows the reward curve obtained by training the APF-MADDPG algorithm in a complex environment with an artificial potential field added to the simulation example according to Embodiment 1 of this disclosure. Detailed Implementation

[0038] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0040] It should be noted that the terminology used herein is for descriptive purposes only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0041] Example 1

[0042] In one or more of the technical solutions disclosed in the embodiments, such as Figure 1 As shown in Figure 8, a multi-agent landmark coverage method based on deep reinforcement learning includes the following steps:

[0043] Step 1: For multi-agent systems, establish a multi-agent reinforcement learning framework model based on the deep deterministic policy gradient algorithm;

[0044] Step 2: Add artificial potential field information to the original state space to increase the dimensionality of the state space, and combine the potential field of each state of the agent with the original state space to update the state of the agent itself.

[0045] Step 3: Improve the reward function by combining the number of steps with the relative distance, and reward or punish the agent in each state based on the distance and number of steps.

[0046] Step 4: Based on the upgraded state space and the improved reward function, train the multi-agent reinforcement learning framework model of the deep deterministic policy gradient algorithm to obtain the updated model parameters.

[0047] Step 5: Obtain the task allocation information for landmark coverage and the state information of the agents, input them into the trained multi-agent reinforcement learning framework model, and obtain the optimal motion planning path for the multi-agents.

[0048] In this embodiment, to avoid the slow convergence speed of the agent in the later stages of training, deep reinforcement learning methods are combined with traditional artificial potential field methods. The artificial potential field information is added to the original state space to increase the dimensionality of the state space, which solves the problem of poor decision-making performance caused by partial observation of the environment. Secondly, a reward function is designed to combine the number of steps and relative distance to solve the problem of sparse rewards obtained by the agent in the early stage of exploration, so that the agent can effectively avoid obstacles and improve the learning efficiency and training efficiency of the agent.

[0049] The multi-agent reinforcement learning framework based on the deep deterministic policy gradient algorithm is abbreviated as MADDPG: Multi-Agent Deep Deterministic Policy Gradient. It builds a DDPG model for each agent based on DDPG.

[0050] The MADDPG algorithm is a novel centralized learning-distributed execution framework. The training process for each agent is similar to that of a single DDPG algorithm, but the difference lies in the fact that the critic network includes not only the current agent's state information but also the state information of other agents, such as... Figure 3 As shown, π i For agent i's own strategy, o i For agent i to receive the observation value, a i Q represents the action taken by agent i under the current policy. i The reward value is the feedback from the environment to agent i.

[0051] In step 1, a multi-agent reinforcement learning framework model based on the deep deterministic policy gradient algorithm is constructed. Specifically, a reinforcement learning framework model is built for each agent, which is the DDPG model, such as... Figure 1 The diagram shows a reinforcement learning framework model for a single agent, including an observation layer, a state space layer, an experience pool, an Actor network, and a Critic network. The observation layer transmits the observed information to the state space layer, which is connected to the Actor network. The Actor network and the Critic network are interconnected. The experience pool D is used to store the agent's current state, action, reward, next state, and agent potential field value (s,a,r,s′).

[0052] The observation layer is used to observe the agent's environmental information, including the agent's position information, motion information, and artificial potential field information.

[0053] like Figure 4 As shown, the Actor network includes an Online Actor network and a Target Actor network, while the Critic network includes an Online Critic network and a Target Critic network. The Online and Target networks have the same neural network structure, differing only in their update methods. As a deterministic policy gradient algorithm, the Actor network updates based on the input state s. t Afterwards, its output action value is a deterministic action a. t Instead of a probability distribution, update the real-world Actor network parameters θ simultaneously. μ The target Actor network is for the next state s t+1 Execute the next deterministic action a t+1 , using θ μ For the target Actor network parameters θ μ Perform a soft update. The real Critic network combines the current state value s. t For action value a t Calculate the corresponding Q value Q(s) t ,a t And update the real-world Critic network parameters θ Q Target Critic network calculations t+1 The Q-value and the target Critic network parameters θ are given below. Q ′ through θ Q Update.

[0054] As mentioned earlier, DDPG contains four networks, among which the Reality Critic network is updated by minimizing a loss function, which is:

[0055]

[0056] Where N is the sample batch size, i represents the i-th agent, and y i For the actual network output value, s i With a i These represent the input state values ​​and output action values ​​of the real network, θ. Q These are the parameters of the real Critic network.

[0057] Real-world Actor networks update by sampling the policy gradient, where the policy gradient is:

[0058]

[0059] During training, the target network parameters are updated periodically using a soft update method. The soft update method is as follows:

[0060] θ Q′ ←τθ Q +(1-τ)θ Q′

[0061] θ μ′ ←τθ μ +(1-τ)θ μ′

[0062] Where μ and Q are the parameters of the Actor network and Critic network, respectively. μ' and Q' are the parameters of the target network. τ is much less than 1.

[0063] To avoid slow convergence in the later stages of agent training, this embodiment combines deep reinforcement learning with traditional artificial potential field methods.

[0064] In some embodiments, in step 2, the original state space may include agent position information and motion information, which are information received from the environment. The agent's position information includes the agent's own position, the positions of other agents, the positions of obstacles, and the position of the target point in the current state. The motion information includes the agent's own motion speed and the motion speeds of other agents. As shown in equation (1) and Table 1:

[0065] s=(s curr_pos ,s othe_pos ,s land_pos ,s goal_pos ,s curr_vel ,s othe_vel (1)

[0066] Table 1 Original State Space

[0067]

[0068] In the process of achieving intelligent agent landmark coverage, the MADDPG algorithm suffers from slow training speed due to the agent's lack of knowledge about the surrounding environment. This embodiment proposes an improved multi-agent landmark coverage algorithm based on the MADDPG algorithm training model and combined with the artificial potential field method.

[0069] The artificial potential field method virtualizes the environment in which the intelligent agent resides through a force field, and generates potential field information of the intelligent agent in different environmental states in real time.

[0070] The method for determining artificial potential field information is as follows: set the target point to generate an attractive field on the agent, and the obstacle to generate a repulsive field on the agent. Calculate the corresponding attractive or repulsive force by the relative distance between the agent and the obstacle, and between the agent and the target point. The sum of the repulsive and attractive forces is the net force on the agent, and the net force on the agent is the artificial potential field information.

[0071] Specifically, the force situation of the agent is shown in Figure 5. The target point exerts a gravitational force F on the agent. att The obstacle exerts a repulsive force F on the agent. rep The net force acting on the intelligent agent is F:

[0072] The gravitational field function of the artificial potential field method is shown in equation (2):

[0073]

[0074] Among them, X agent = (x, y) represents the current position of the agent, X goal =(x goal ,y goal ) represents the coordinates of the target point that the agent wants to reach, ξ is the gravitational field scale factor, and ρ(X) is the coordinates of the target point. agent ,X goal )=||X agent -X goal || represents the distance between the agent's current state and the target.

[0075] The repulsive field function is shown in equation (3):

[0076]

[0077] Among them, X landmark =(x landmark ,y landmark ) represents the obstacle's position coordinates, η represents the repulsive force field scale factor, and ρ(X) represents the obstacle's position coordinates. agent ,X landmark )=||X agent -X landmark|| represents the distance between the agent's current state and the obstacle. ρ0 represents the radius of influence of the obstacle's repulsive force field, meaning that when the distance between the agent and the obstacle is greater than ρ0, the obstacle does not exert a repulsive force on the agent.

[0078] The negative gradient of the potential field is the force, as shown in the following formula:

[0079]

[0080]

[0081] The attraction and repulsion forces are superimposed to form an artificial potential field force, as shown in equation (6):

[0082] F(X agent ) = F att (X)+F rep (X) (6)

[0083] Among them, F att (X) represents gravity, F rep (X) represents the repulsive force.

[0084] In this embodiment, to address the slow convergence speed and non-optimal path issues in current multi-agent landmark coverage, artificial potential field forces are introduced into the state space. By adding artificial potential field forces to the original state space, a state space combining artificial potential fields is obtained. Based on this state space, different potential values ​​are formed at different locations in the environment model, and an optimal collision avoidance path from the initial point to the target point can be obtained according to these different potential values. The state space combining artificial potential fields is shown in equation (7):

[0085] s=(s curr_pos ,s othe_pos ,s land_pos ,s goal_pos ,s curr_vel ,s othe_vel ,F(X agent (7)

[0086] Wherein, F(X) agent ) represents the artificial potential field force of the agent at its current position.

[0087] Furthermore, it also includes setting the action space of the agent, using the agent's speed information as the action space.

[0088] The policy output during the interaction between the agent and the environment is the action space. When the input is a discrete action value, the policy output is a discrete action space. For a continuous action space, the policy output is a continuous action space. Here, the action space is a continuous value, as shown in Equation (8). The agent's velocity information is selected as the action space, as shown in Table 2, which is the action space of an agent containing two agents.

[0089] A = [a1, a2, a3, a4] (8)

[0090] Table 2 Action Space of the Agent

[0091]

[0092]

[0093] In step 3, to address the issue of sparse rewards obtained by the agent in the early stages of exploration, this embodiment designs a reward function by combining the agent's step count information, the relative distance between the agent and the target point, the relative distance between the agent and obstacles, and the distance between the agent's position and the guidance path.

[0094] In each state of the agent, the agent is rewarded or punished based on distance and step count information, which enables the agent to effectively avoid obstacles and improve the agent's learning efficiency.

[0095] In reinforcement learning, an agent observes states from the environment, inputs these states into a training network model, and then takes actions based on the current policy. Simultaneously, the environment provides the agent with different reward values ​​for each action, further adjusting the network parameters in an iterative process. The agent accumulates rewards from the start of training to maximize the objective, resulting in a cumulative reward. The design of the reward function is crucial for training the policy network. Reinforcement learning environments often suffer from reward sparsity, leading to slow training speeds.

[0096] The rewards in this embodiment include: distance guidance rewards, step count penalty rewards, obstacle collision penalty rewards, and target arrival rewards. By designing distance guidance rewards and step count penalty rewards, redundant steps during agent training can be avoided. Obstacle collision penalty rewards and target arrival rewards are also included.

[0097] Specifically, a distance-guided reward is set based on the relative distance between the agent and the target point. The closer the agent is to the target point, the greater the reward. The distance-guided reward is shown in Equation (10):

[0098]

[0099]

[0100] Among them, (xagent ,y agent (x) represents the current position coordinates of the agent. goal ,y goal d represents the coordinates of the target point of the intelligent agent. agent_goal This represents the distance between the agent and the target.

[0101] Furthermore, to enable the agent to reach the target point as quickly as possible with the fewest steps and avoid redundant steps, the agent is penalized with a set score for each step. The set score can be set as needed; in this embodiment, it is set to -5. The step penalty reward in this embodiment is:

[0102] R step =-5 (11)

[0103] To minimize collisions during the path, the collision penalty and reward are as follows:

[0104] R collide =-30 (12)

[0105] To guide the agent to the target point along the optimal path, an artificial potential field is used for reward guidance.

[0106]

[0107] R APF =cos(d agent_APF) (14)

[0108] Among them, (x APF ,y APF ) represents the path coordinates of the artificial potential field, d agent_APF This represents the distance the agent deviates from the path coordinates of the artificial potential field.

[0109] The reward for reaching the target is:

[0110] R arrive =200 (15)

[0111] In summary, the total reward received by the agent is:

[0112]

[0113] Furthermore, the training method for a multi-agent reinforcement learning framework model based on a deep deterministic policy gradient algorithm includes the following steps:

[0114] Step S1: Set the number of training rounds N;

[0115] Step S2: Initialize a random process for action training to obtain the initial state value s;

[0116] Step S3: For each agent, randomly select action a and add random noise;

[0117] Step S4: Calculate the potential field value of the agent in the current state and add it to the state space:

[0118] F(X agent ) = F att (X)+F rep (X)

[0119] Step S5: Perform an action in the current state to obtain feedback reward r and the next state s′;

[0120] Step S6: Pack the current state, action, reward, next state, and agent potential field value (s,a,r,s′) into the experience pool D;

[0121] Step S7: Update the next state value to the current state: s′→s;

[0122] Randomly select samples from the experience pool D, train the model, and update the model parameters until the training iterations are reached. The steps are as follows:

[0123] Step S8: Randomly draw minibatch samples (s) from the experience pool D. j ,a j ,r j ,s′ j ),definition:

[0124]

[0125] Where r is the agent's reward value, γ is the discount factor, and γ∈[0,1]; s′ j This provides state information for other intelligent agents. To estimate the action value function, μ is the Actor network parameter.

[0126] Step S9: Update the Critic real-world network parameters by minimizing the loss function value.

[0127]

[0128] The DDPG structure includes a Critic network and an Actor network. The Critic network, also known as the policy network, is divided into a Critic reality network and a Critic target network.

[0129] Step S10: Update the Actor real-world network parameters using deterministic gradient values:

[0130]

[0131] Step S11: Perform soft updates on the Critic target network parameters and the Actor target network parameters;

[0132] The Critic target network parameters are softly updated as follows:

[0133] θ Q ′←τθ Q +(1-τ)θ Q ′

[0134] The Actor target network parameters are soft-updated as follows:

[0135] θ μ ′←τθ μ +(1-τ)θ μ ′

[0136] In step 5, the task allocation information includes the coordinates of the target location to be covered, and the agent's state information includes the agent's position information, motion information, and artificial potential field information.

[0137] To illustrate the performance of the improved method in this embodiment, experiments were designed for both simple and complex environments. During the simulation, the number of agents M = 2. The simple environment is shown in Figure 7, where hollow circles represent agents and solid circles represent target points. The complex environment, as shown in Figure 7, adds solid circle obstacles between the target point and the agents. The complex environment is essentially the simple environment with added obstacles to test the algorithm's effectiveness under obstacle conditions.

[0138] Reward value can serve as an important metric for measuring the learning effect of an algorithm. The ultimate goal of reinforcement learning is to learn a strategy that maximizes the cumulative reward in a round through continuous training. Therefore, to verify the effectiveness of the proposed algorithm, this paper controls environmental variables in both simple and complex environments, and compares the cumulative reward value obtained by the agent in one round of training with and without the addition of an artificial potential field as the evaluation metric.

[0139] Figure 5(a) shows a simple environment built for the land cover task in a multi-particle environment, and Figure 5(b) shows the task completion status of the algorithm after training. The MADDPG algorithm and the APF-MADDPG algorithm were run for 30,000 rounds in the simple environment. The reward curves were compared after a moving average operation, as shown in Figure 6. The horizontal axis represents the number of training rounds, and the vertical axis represents the cumulative reward value obtained by the agent in one round. Figure 6(a) shows the reward value curve obtained by the MADDPG algorithm without an artificial potential field. During training, the cumulative reward was averaged every 5 rounds. It can be seen that the reward value of the agent converged and stabilized around 3850 on the horizontal axis, that is, around 3850*5=19250 rounds, but fluctuations occurred in the later stages. Figure 6(b) shows the reward value obtained by the APF-MADDPG algorithm after adding an artificial potential field. It can be seen that the reward value of the agent converged and stabilized around 700 rounds on the horizontal axis, that is, around 700*5=3500 rounds. The comparison of the two sets of experiments shows that when there are only agents and target points in the environment, the training speed of the algorithm is significantly improved after adding an artificial potential field.

[0140] While the algorithm's training speed has improved compared to the benchmark algorithm in a simple environment, the overly simplistic environment makes it impossible to assess the landmark coverage effect when interference factors such as agents are present. Therefore, this paper adds obstacles to the simple environment to test whether the improved algorithm remains effective.

[0141] Figure 7(a) shows the complex environment built for the land cover task in a multi-particle environment, and Figure 7(b) shows the task completion status of the algorithm after training. Figure 8 shows a comparison of the MADDPG algorithm and the APF-MADDPG algorithm running for 30,000 rounds in the complex environment. The curve in Figure 8(a) is the reward value curve obtained by the MADDPG algorithm without an artificial potential field. During training, the cumulative reward was averaged every 5 rounds. It can be seen that the reward value of the agent converges and stabilizes around 3200 on the horizontal axis, i.e., around 3200*5=16000 rounds. The curve in Figure 8(b) shows the reward value obtained by the APF-MADDPG algorithm after adding an artificial potential field. It can be seen that the reward value of the agent converges and stabilizes around 2250 on the horizontal axis, i.e., around 2250*5=11250 rounds. The comparison of the two sets of experiments shows that when obstacles exist in the environment, adding an artificial potential field can significantly improve the training speed of the algorithm.

[0142] In this embodiment, the MADDPG algorithm from multi-agent deep reinforcement learning is combined with the traditional artificial potential field method to achieve a multi-agent landmark coverage task. The algorithm is validated by designing simple and complex environments. The results show that the improved algorithm significantly improves training speed and can complete obstacle avoidance tasks in complex environments.

[0143] Example 2

[0144] Based on Example 1, this example provides a multi-agent landmark coverage system based on deep reinforcement learning, including:

[0145] Model building module: configured to build a multi-agent reinforcement learning framework model based on a deep deterministic policy gradient algorithm for multi-agent systems;

[0146] State space construction module: It is configured to add artificial potential field information to the original state space to increase the dimensionality of the state space, and combine the potential field of each state of the agent with the original state space to update the state of the agent itself.

[0147] Reward setting module: It is configured to combine steps and relative distance to improve the reward function, and to reward or punish the agent in each state based on distance and steps.

[0148] Model training module: configured to train the multi-agent reinforcement learning framework model of deep deterministic policy gradient algorithm to obtain updated model parameters based on the upgraded state space and the improved reward function;

[0149] Path generation module: It is configured to obtain task allocation information for landmark coverage and agent state information, input them into the trained multi-agent reinforcement learning framework model, and obtain the optimal motion planning path for the multi-agent.

[0150] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0151] Example 3

[0152] This embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it performs the steps described in the method of Embodiment 1.

[0153] Example 4

[0154] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps described in the method of Embodiment 1.

[0155] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0156] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A multi-agent landmark coverage method based on deep reinforcement learning, characterized in that, Comprising the following steps: For multiple agents, a multi-agent reinforcement learning framework model based on deep deterministic policy gradient algorithm is established; The artificial potential field information is added to the original state space to upgrade the state space, and the potential field of the agent at each state is combined with the original state space to update the state of the agent itself; The step number and the relative distance are combined to improve the reward function, and the agent is rewarded or punished according to the distance and step number information at each state; The improved reward function includes distance guidance reward, step number punishment reward, obstacle collision punishment reward and target arrival reward; According to the upgraded state space and the improved reward function, the multi-agent reinforcement learning framework model based on deep deterministic policy gradient algorithm is trained to obtain updated model parameters; Obtain the task allocation information for landmark coverage and the state information of the agent, and input them into the trained multi-agent reinforcement learning framework model to obtain the optimal motion planning path of the multi-agent; The determination method of the artificial potential field information is that the target point generates a gravitational field to the agent, and the obstacle generates a repulsive field to the agent, the relative distance between the agent and the obstacle and the relative distance between the agent and the target point are calculated to obtain the corresponding gravity or repulsion, and the sum of the repulsion and the gravity is the resultant force suffered by the agent, which is the artificial potential field information; The distance guidance reward is expressed as: ； ； wherein, is the current position coordinate of the agent, is the target point coordinate of the agent, is the distance between the agent and the target; The artificial potential field is used for reward guidance, and the formula is: ； ； wherein, is the artificial potential field path coordinate, is the distance of the agent from the artificial potential field path coordinate.

2. The deep reinforcement learning based multi-agent landmark coverage method of claim 1, wherein: The multi-agent reinforcement learning framework model based on deep deterministic policy gradient algorithm is to construct a reinforcement learning framework model for each agent, and the reinforcement learning framework model of a single agent includes an observation layer, a state space layer, an experience pool, an Actor network and a Critic network; The observation layer transmits the observed information to the state space layer, the state space layer is connected to the Actor network, the Actor network and the Critic network are interactively connected, and the experience pool is used to store the current state, action, reward, next state and agent potential field value of the agent.

3. The deep reinforcement learning based multi-agent landmark coverage method of claim 1, wherein: The original state space includes agent position information and motion information.

4. The deep reinforcement learning based multi-agent landmark coverage method of claim 1, wherein: The agent velocity information is used as the action space.

5. The deep reinforcement learning based multi-agent landmark coverage method of claim 1, wherein, The training method of the multi-agent reinforcement learning framework model based on deep deterministic policy gradient algorithm comprises the following steps: Set the number of training rounds, initialize a random process for action training, and obtain an initial state value; For each agent, randomly select an action and add random noise; Calculate the potential field value of the agent in the current state and add it to the state space: Execute the action in the current state to obtain the feedback reward and the next state; Pack the current state, action, reward, next state and agent potential field value into the experience pool; Update the next state value to the current state; Randomly extract a sample from the experience pool D, train it, and update the model parameters until the training times are reached.

6. A multi-agent landmark coverage system based on deep reinforcement learning, characterized in that, It comprises: A model construction module configured to establish a multi-agent reinforcement learning framework model based on deep deterministic policy gradient algorithm for multiple agents; The state space construction module is configured to add artificial potential field information to the original state space to upgrade the state space, and combine the potential field of each state of the agent with the original state space to update the state of the agent itself. The artificial potential field information is determined by setting a target point to generate a gravitational field for the agent, and obstacles to generate a repulsive field for the agent. The corresponding gravity or repulsion is calculated through the relative distance between the agent and the obstacles and the relative distance between the agent and the target point. The sum of the repulsion and the gravity is the resultant force acting on the agent, which is the artificial potential field information. The reward setting module is configured to combine the step number and the relative distance to improve the reward function, and reward or punish the agent according to the distance and step number information at each state of the agent. The improved reward function includes distance guidance reward, step number punishment reward, obstacle collision punishment reward and target arrival reward. The model training module is configured to train the multi-agent reinforcement learning framework model of the deep deterministic policy gradient algorithm according to the upgraded state space and the improved reward function, to obtain updated model parameters. The path generation module is configured to obtain landmark coverage task assignment information and state information of the agent, and input them into the trained multi-agent reinforcement learning framework model to obtain the optimal motion planning path of the multi-agent. The distance guidance reward is represented as: ； ； wherein, is the current position coordinate of the agent, is the target point coordinate of the agent, is the distance between the agent and the target; The reward guidance is performed by using the artificial potential field, and the formula is: ； ； wherein, is the artificial potential field path coordinate, is the distance of the agent from the artificial potential field path coordinate.

7. An electronic device, comprising: A computer program product including a memory and a processor and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of the method of any one of claims 1-5 are completed.

8. A computer-readable storage medium, characterized in that, A computer program product for storing computer instructions, when the computer instructions are executed by the processor, the steps of the method of any one of claims 1-5 are completed.

Images