Multi-agent cooperative control method and system based on deep reinforcement learning
By combining MAPPO and ORCA algorithms, the BiGRU module is used to optimize the action and state space of the multi-agent system, the complexity of agent decision-making and path planning in the multi-agent system is solved, and efficient agent collaborative control and path planning are achieved. .
Patent Information
- Application Number
- CN202411130661.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-26
AI Technical Summary
In multi-agent systems, agents need to coordinate and compete in a dynamic environment to handle high-dimensional observation and action spaces, and existing algorithms are difficult to effectively handle complex decisions and path planning between agents, resulting in high computational complexity , low efficiency.
Adopting a multi-agent collaborative control method based on deep reinforcement learning, combining multi-agent proximal strategy optimization (MAPPO) and optimal reciprocity obstacle avoidance (ORCA) algorithm, the intelligent body is optimized by adding a bidirectional gated cycle unit (BiGRU) module. The action and state space of the agent can be constructed to build the optimal mutual obstacle avoidance feasible domain for individual agents, and realize coordinated control and path planning between agents.
In a dynamic environment, the coordination and competitiveness between agents are improved, the computational complexity is reduced, efficient path planning is realized in various scenarios, and the agent's autonomous obstacle avoidance and coordinated control capabilities are enhanced.
Smart Images

Figure CN119026666A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of multi-agent control technology, and specifically relates to a multi-agent collaborative control method and system based on deep reinforcement learning. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] In a highly complex environment, a single agent is often unable to effectively cope with diverse tasks, and the collaborative work of multiple agents has gradually become a reliable solution. A multi-agent system (MAS) is a system composed of multiple agents with autonomy, sociality, responsiveness, and foresight. Due to its robustness, reliability, efficiency, and scalability, multi-agents have been widely used in computer networks, robots, manufacturing, traffic control, virtual reality, and other fields, and have become the main research object in distributed artificial intelligence research. In a multi-agent system, agents communicate with each other, cooperate or compete with each other, and solve complex problems in parallel, thereby effectively improving problem-solving capabilities.
[0004] Multi-agent systems must consider real-time changes in the environment, and the actions chosen by different agents are determined by the local environmental information they perceive at the current moment, and the actions will in turn affect the current environment. Since the decisions made rely on the previously observed environment, but the environment has changed due to the actions, the training process is no longer stable. This problem is the non-steady-state problem. The existence of the non-steady-state problem requires agents to consider the impact of other agents' behaviors on the environment and themselves when making decisions, which increases the complexity of the training process. Therefore, algorithms are required to be able to make better decisions under dynamic changes.
[0005] Game theory provides a framework for multi-agent reinforcement learning. Static games consider decisions made at the same time, repeated games deal with multiple decisions in the same state, and random games go a step further and consider multiple states and multiple decisions. This random game model is closer to real situations, especially in complex systems involving environmental changes and interactions between agents. In this context, multi-agent reinforcement learning not only needs to deal with high-dimensional observation and action spaces, but also needs to achieve collaboration and competition between agents in a dynamic environment.
[0006] The distributed cooperative control of multi-agents only relies on local information between agents, which greatly reduces communication costs and energy consumption, and has good flexibility and robustness. The distributed cooperative ability of multi-agents is the key to exerting the advantages of multi-agents and is also the embodiment of the intelligence of the entire multi-agent system.
[0007] It is very common for multiple agents in a multi-agent system to solve tasks in a collaborative manner. When multiple agents need to collaborate to solve a task, the strategy used by a single agent to solve the task cannot be directly applied to the multi-agent system. Therefore, multi-agent collaborative control is required to achieve this. Multi-agent collaborative control needs to consider the state and actions of other agents when making decisions to avoid collisions and conflicts and ensure the coordination of the overall planning; this requires the decision-making algorithm to not only consider the local information of the individual, but also to perceive and process the global state of the entire system. Summary of the invention
[0008] In order to solve the above problems, the present invention proposes a multi-agent collaborative control method and system based on deep reinforcement learning. Based on a multi-agent obstacle avoidance and path planning algorithm combining multi-agent proximal policy optimization (MAPPO) and optimal reciprocal collision avoidance (ORCA), the multi-agent collaborative control problem is modeled, the collaboration and competition between agents are realized in a dynamic environment, and path planning is realized in various scenarios.
[0009] According to some embodiments, a first solution of the present invention provides a multi-agent collaborative control method based on deep reinforcement learning, which adopts the following technical solutions:
[0010] A multi-agent collaborative control method based on deep reinforcement learning, comprising:
[0011] Obtain the state space and action space of several intelligent agents in the multi-agent system;
[0012] Considering the implicit reward function of the individual intelligent agent, the optimal mutual obstacle avoidance feasible domain of the individual intelligent agent is constructed;
[0013] In the optimal mutual obstacle avoidance feasible domain of the constructed intelligent individuals, a multi-agent proximal strategy optimization algorithm with a bidirectional gated recurrent unit is used to perform deep reinforcement learning of the multi-agents, optimize the action space and state space of the intelligent individuals, and obtain the dynamic state estimation value of the multi-agents.
[0014] According to the obtained dynamic state estimation value, the optimal path of the individual intelligent agent is planned to complete the collaborative control of multiple intelligent agents.
[0015] As a further technical limitation, in the process of multi-agent deep reinforcement learning using a multi-agent proximal policy optimization algorithm with a bidirectional gated recurrent unit, a global value function is added to promote collaboration between individual agents; in the behavioral neural network Actor, the individual agent uses a strategy to select actions based on current local observations; in the evaluation neural network Critic, the multi-agent actions are evaluated based on the joint observations and joint actions of all individual agents, and the evaluation results are fed back to the Actor network to guide the selection of actions of other individual agents; the individual action space and state space of the agents are optimized through continuous cycles of iterations to obtain the dynamic state estimation value of the multi-agent.
[0016] As a further technical limitation, the implicit reward function r collision for Among them, diff v The distance between the speed selected by the strategy and the expected speed calculated by the optimal reciprocal obstacle avoidance method, i.e., diff v =||v t -v des ||, is the expected speed of the agent, ORCA is the optimal mutual obstacle avoidance feasible domain of the agent, v t is the current speed of the agent; ξ is the expected minimum time for the agent to collide with an obstacle at the current speed; a, b, c, d, and e are all constant values, where a and c are the basic rewards for encouraging behavior when the current speed is close to the expected speed or the current collision risk is low, and b is the diff v The coefficient of the weight, that is, the desired speed of movement, d and e are both (ξ+f) -1 Weight coefficient; (ξ+f) -1 Indicates a motion that maintains a longer expected collision time.
[0017] As a further technical limitation, in the process of planning the optimal path of the intelligent individual, the speed of the dynamic obstacles is taken into consideration, and the relative position between the dynamic obstacles and the intelligent individual is converted into speed in combination with the time window. Based on the optimal reciprocal obstacle avoidance feasible domain of the intelligent individual, the optimal speed with the smallest adjustment amount with the speed of the intelligent individual is selected to realize autonomous obstacle avoidance of the intelligent individual in a dynamic environment and complete the planning of the optimal path for the intelligent individual.
[0018] As a further technical limitation, an implicit reward function is considered to calculate the speed of the individual agent and its optimal reciprocal obstacle avoidance feasible domain in each state. If the obtained speed is within the optimal reciprocal obstacle avoidance feasible domain, a positive reward is given to the individual agent, otherwise a negative reward is estimated to guide the agent to perform actions within the obtained optimal reciprocal obstacle avoidance feasible domain.
[0019] Furthermore, the ray method is used to determine whether the speed of the individual intelligent agent is within the optimal reciprocal obstacle avoidance feasible domain. Specifically, a horizontal straight line is drawn from the speed point to be determined, and the number of intersections between the horizontal straight line and all edges in the optimal reciprocal obstacle avoidance feasible domain is counted. If there are an odd number of intersections with each edge, the speed is within the optimal reciprocal obstacle avoidance feasible domain; if there are an even number of intersections with each edge, the speed is not within the optimal reciprocal obstacle avoidance feasible domain.
[0020] According to some embodiments, a second solution of the present invention provides a multi-agent collaborative control system based on deep reinforcement learning, which adopts the following technical solutions:
[0021] A multi-agent collaborative control system based on deep reinforcement learning, comprising:
[0022] An acquisition module, configured to acquire state spaces and action spaces of several intelligent agent individuals in a multi-agent;
[0023] A construction module, which is configured to consider the implicit reward function of the individual intelligent agents and construct the optimal mutual obstacle avoidance feasible domain of the individual intelligent agents;
[0024] An optimization module is configured to perform deep reinforcement learning of multi-agents by using a multi-agent proximal strategy optimization algorithm with a bidirectional gated recurrent unit in the constructed optimal mutual obstacle avoidance feasible domain of the individual agents, optimize the action space and state space of the individual agents, and obtain the dynamic state estimation value of the multi-agents;
[0025] The collaborative control module is configured to plan the optimal path of the individual intelligent agents according to the obtained dynamic state estimation value and complete the collaborative control of multiple intelligent agents.
[0026] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:
[0027] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in the first embodiment of the present invention.
[0028] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:
[0029] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in the first scheme of the present invention are implemented.
[0030] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:
[0031] A computer program product includes software code, wherein the program in the software code executes the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in the first embodiment of the present invention.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] The present invention performs collaborative control of multi-agents based on a multi-agent obstacle avoidance and path planning algorithm that combines multi-agent proximal strategy optimization with optimal reciprocal obstacle avoidance, introduces the optimal reciprocal obstacle avoidance feasible domain into the reward function, adds a bidirectional GRU module into the multi-agent proximal strategy optimization, optimizes the decision-making ability in a partially observable multi-agent environment; processes high-dimensional observation and action spaces, realizes collaboration and competition between agents in a dynamic environment, reduces computational complexity, improves planning efficiency, and can realize path planning in a variety of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.
[0035] Figure 1 Schematic diagram of mutual obstacle avoidance based on speed obstacle method in Embodiment 1 of the present invention; wherein, Figure 1 (a) is a schematic diagram of mutual obstacle avoidance in the initial state. Figure 1 (b) is a schematic diagram of mutual obstacle avoidance after a collision at the midpoint. Figure 1 (c) is a schematic diagram of mutual obstacle avoidance with optimal path;
[0036] Figure 2 Schematic diagram of mutual obstacle avoidance in Embodiment 1 of the present invention; wherein, Figure 2 (a) is a schematic diagram of mutual obstacle avoidance in the initial state. Figure 2 (b) is a schematic diagram of mutual obstacle avoidance based on the reciprocal speed obstacle method. Figure 2 (c) is a schematic diagram of mutual obstacle avoidance based on ORCA;
[0037] Figure 3 Schematic diagram of agent A and agent B in the first embodiment of the present invention; wherein, Figure 3 (a) is a schematic diagram of the radius of agent A and agent B; Figure 3 (b) is a schematic diagram of the relative speed between agent A and agent B. Figure 3 (c) is another schematic diagram of the relative speed between agent A and agent B;
[0038] Figure 4 It is a schematic diagram of a speed obstacle area and a speed linear selection area in the first embodiment of the present invention;
[0039] Figure 5 Schematic diagram of the optional speed area of the agent in the first embodiment of the present invention; wherein, Figure 5 (a) is a schematic diagram of the intersection area of multiple half-planes. Figure 5 (b) is a schematic diagram of the new speed selection area allowed for the agent;
[0040] Figure 6 A schematic diagram of a network structure of a path planning framework based on MAPPO in the first embodiment of the present invention;
[0041] Figure 7 It is a schematic diagram of the illustration of the ray method in the first embodiment of the present invention;
[0042] Figure 8 This is a flowchart of multi-agent reinforcement learning in Embodiment 1 of the present invention;
[0043] Fig. 9 Schematic diagram of a multi-agent path planning training simulation environment in Embodiment 1 of the present invention; wherein: Fig. 9 (a) is a schematic diagram of a simple environment simulation environment with four intelligent agents. Fig. 9 (b) is a schematic diagram of a complex environment simulation environment with 10 intelligent agents;
[0044] Fig.10 A schematic diagram of a simulation environment with obstacles added in the first embodiment of the present invention;
[0045] Fig.11 A schematic diagram of a multi-agent path planning random simulation environment in Embodiment 1 of the present invention;
[0046] Fig.12 Schematic diagram of reward curves of four agents in the env1 environment in Example 1 of the present invention;
[0047] Fig.13 Schematic diagram of reward curves of 10 agents in the env2 environment in Example 1 of the present invention;
[0048] Fig.14 A schematic diagram showing a comparison of the success rates of reaching a target point in a training environment for two algorithms in Embodiment 1 of the present invention;
[0049] Fig.15 Schematic diagram of the planning of the opposite motion paths of four intelligent agents in the first embodiment of the present invention; wherein: Fig.15 (a) is a schematic diagram of the path planning of the four agents moving in opposite directions at the 0th time step. Fig.15(b) is a schematic diagram of the path planning of the four agents moving in opposite directions at the 39th time step. Fig.15 (c) is a schematic diagram of the opposite motion path planning of the four agents at the 62nd time step;
[0050] Fig.16 Schematic diagram of the planning of the opposite motion paths of 10 agents in the first embodiment of the present invention; wherein: Fig.16 (a) is a schematic diagram of the path planning of the 10 agents moving in opposite directions at the 0th time step. Fig.16 (b) is a schematic diagram of the opposite motion path planning of 10 agents at the 13th time step. Fig.16 (c) is a schematic diagram of the opposite motion path planning of 10 agents at the 26th time step. Fig.16 (d) is a schematic diagram of the opposite motion path planning of 10 agents at the 39th time step. Fig.16 (e) is a schematic diagram of the path planning of the 10 agents moving in opposite directions at the 52nd time step. Fig.16 (f) is a schematic diagram of the opposite motion path planning of 10 agents at the 65th time step;
[0051] Fig.17 This is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle in the first embodiment of the present invention; wherein, Fig.17 (a) is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 13th time step. Fig.17 (b) is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 14th time step. Fig.17 (c) in the figure is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 28th time step. Fig.17 (d) is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 42nd time step. Fig.17 (e) in the figure is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 56th time step. Fig.17 (f) is a schematic diagram of the opposite motion path planning of an intelligent agent after adding a dynamic obstacle at the 69th time step;
[0052] Fig.18 Schematic diagram of the planning of the opposite motion paths of 20 agents in the first embodiment of the present invention; wherein: Fig.18 (a) is a schematic diagram of the path planning for the opposite motion of 20 agents at the 0th time step. Fig.18 (b) is a schematic diagram of the path planning for the opposite motion of 20 agents at the 24th time step. Fig.18 (c) is a schematic diagram of the path planning for the opposite motion of 20 agents at the 48th time step. Fig.18(d) is a schematic diagram of the path planning of the 20 agents moving in opposite directions at the 72nd time step. Fig.18 (e) is a schematic diagram of the path planning of the 20 agents moving in opposite directions at the 96th time step. Fig.18 (f) is a schematic diagram of the opposite motion path planning of 20 agents at the 119th time step;
[0053] Fig.19 Schematic diagram of path planning for 10 agents in a random environment in Embodiment 1 of the present invention; wherein: Fig.19 (a) Schematic diagram of path planning of 10 agents in a random environment at time step 0. Fig.19 (b) Schematic diagram of the path planning of 10 agents in a random environment at the 14th time step. Fig.19 (c) Schematic diagram of the random environment path planning of 10 agents at the 27th time step. Fig.19 (d) Schematic diagram of the path planning of 10 agents in a random environment at the 41st time step. Fig.19 (e) Schematic diagram of the path planning of 10 agents in a random environment at the 55th time step. Fig.19 (f) Schematic diagram of path planning of 10 agents in a random environment at the 68th time step. DETAILED DESCRIPTION
[0054] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0055] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0057] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.
[0058] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. For relevant scientific research or technical personnel in this field, the specific meanings of the above terms in the present invention can be determined according to specific circumstances, and they cannot be understood as limitations on the present invention.
[0059] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0060] Embodiment 1
[0061] Embodiment 1 of the present invention introduces a multi-agent collaborative control method based on deep reinforcement learning.
[0062] In a multi-agent system, each agent makes independent decisions based on local environmental information. This problem is called a decentralized partially observable Markov decision process (DEC-POMDP). A multi-agent system can be modeled as a Markov decision process in which each agent can only observe part of the environmental state. DEC-POMDP can be defined as a tuple<S,A,O,R,P,n,γ> , where n is the total number of agents, S is the global environment state, and constitutes the entire state space, A = (a 1 ,a 2 ,...,a n ) represents the joint action space of all agents, which is also the action space shared by each agent. O = (o 1 ,o 2 ,...,o n ) represents the joint observation of n agents, o n represents the local observation of agent n in the global state S, R(s,A) represents the reward function shared by the agents, P(s'|s,A) represents the probability of the agent moving from state S to S' under the joint action A, and γ is the reward discount factor. The joint strategy of the agents is π=(π 1 ,π 2 ,...,π n ), in most cases, each agent will use the same strategy π θ (a i |o i ), guiding the agent to make corresponding actions under local observation.
[0063] The optimal reciprocal obstacle avoidance algorithm has excellent performance in obstacle avoidance of intelligent agents. The algorithm is based on velocity obstacle (VO), considers the speed of dynamic obstacles, introduces a time window τ, converts relative position into speed, and uses several half-plane constraints in velocity space. Through linear programming, it finds the speed selection area (ORCA area) where no collision will occur within a certain time, selects the optimal speed with the smallest adjustment amount with the original speed, and realizes autonomous obstacle avoidance in a dynamic environment.
[0064] The main advantage of the speed barrier method is that it comprehensively considers the relative position, speed and radius between the agent and the obstacle, and has the characteristics of small calculation and simple implementation. The core idea of the speed barrier method is to only consider the agent's own actions and regard all objects in the environment that hinder its own movement as dynamic obstacles. The position, geometry and speed information of the current agent and obstacles are obtained through the perception system. According to the perception information, a speed barrier area where conflicts may occur between the agent and the obstacle is constructed. This area represents the set of all speeds that may cause collisions. By only selecting the speed outside the speed barrier area, the agent can move safely without collision.
[0065] However, the disadvantages of the speed barrier method are also very obvious. Figure 1 As shown in the figure, the agent is simplified into a solid circle with speed. Agents A and B are moving towards each other and will collide at the midpoint of their current positions. Therefore, they will each choose the obstacle avoidance method of the VO speed obstacle method, such as Figure 1 As shown in (b) in the figure, after this moment, because the speed barrier method assumes that other agents move forward at a stable constant speed, agents A and B perceive that there will be no collision in the future, and they will use the optimal path to obtain Figure 1 (c) in the above formula means returning to the initial state. Figure 1 (a) in the figure, which will cause jitter.
[0066] For a single agent, although the correct obstacle avoidance method was selected, it failed to avoid obstacles because it did not consider that other agents would also avoid obstacles. The Reciprocal Velocity Obstacle (RVO) algorithm is an improvement on the VO algorithm. When the agent chooses the direction and speed of the next frame, it will consider the position and ideal direction and speed of other agents. The agents that affect each other will avoid a distance in the direction of the speed away. The other agent uses the same strategy as itself, and both change half to ensure that there is no collision during the movement.
[0067] Although RVO solves the problem of jitter in the opposite motion between two agents in the VO algorithm and almost always performs well in one-on-one obstacle avoidance, all agents in RVO assume that the other party only considers itself. When the number of agents increases, the phenomenon of agent jitter still occurs. Figure 2 As shown in the figure, A and B are moving towards C together. When using the RVO algorithm, A and B will each move toward the outside. Since C cannot judge whether to match the speed with A or B, C will modify its own speed according to the unexpected speed of the other party at the next moment, resulting in jitter. ORCA can effectively solve this jitter problem.
[0068] like Figure 3 (a) shows two agents A and B, speed obstacles Represents: Starting from the current time, the set of relative velocities of A relative to B that will cause the collision between agents A and B within the time range T. Represent each agent with a circle:
[0069] D(p,r)={q| ||qr|| <r} (1)
[0070] Among them, p and r are the center and radius of the circle representing the agent respectively.
[0071] Translate the origin of the coordinate system to p A After that, the time parameter τ is introduced, and we have:
[0072]
[0073] And then:
[0074] for Figure 3 In the gray part of (b), when the relative speed of A relative to B is When , a collision will occur within τ time. Define the speed area that can ensure collision avoidance (Collision-avoiding):
[0075]
[0076] in, represents the Minkowski sum of the left and right sets, that is:
[0077]
[0078] In order to ensure that the agent does not collide during operation, a speed set that satisfies the conditions should be found. When the relative speed of A and B is , a collision will occur within time τ, such as Figure 3 In order to avoid collision, find the shortest speed moving distance u so that That is, by moving the shortest distance u, the relative speed v A -v B The vector moves to boundary area to avoid collision.
[0079] On the other hand, according to formula (3) and formula (4), it can be easily inferred that and and v A -v B With v B -v A Symmetric about the origin of the coordinate system. Therefore, v B -v A Move to The minimum moving speed vector required for the boundary is -u, v A -v B +u means that the speed of B will not be changed, but the speed of A will be changed by u to avoid a collision between A and B.
[0080] v B -v A +u means that the speed of A is not changed, but the speed of B is changed to -u to avoid collision between B and A. That is to say, at a certain moment, when v A If we only consider the relative position, it is equivalent to v A Moved 2u. In order to find a solution that guarantees v A -v B The minimum vector that escapes the collision velocity set, let v A With v B Move 0.5u and -0.5u respectively, that is:
[0081] v A -v B +u=(v A +0.5u)-(v B -0.5u) (6)
[0082] In order to make the minimum vector u contain more speed, u is used as the normal vector, v A +0.5u is taken as a point on the line, and a straight line is drawn. The lower half plane along the u direction is defined as the ORCA region, which is recorded as It can be expressed by formula (7), as Figure 4 When the new speed selected by A is When in the area, it can be guaranteed that there will be no collision with B. Similarly, we can find
[0083]
[0084] When agent A faces many neighboring agents, it calculates the ORCA area for each neighboring agent and selects the speed within each ORCA intersection area. For example, the dotted area 5 in the figure is the intersection area of multiple half planes, which is the new speed selection area allowed for the agent.
[0085] The limitations of the ORCA algorithm are mainly reflected in three aspects. First, although the time complexity of the ORCA algorithm can be optimized to O(n) through approximate methods, where n represents the number of agents, the decision time of the algorithm increases with the number of agents, making it difficult to efficiently handle decision-making problems for a large number of agents. Second, the ORCA algorithm cannot determine the ideal speed selection interval under certain circumstances and must rely on approximate solutions, which affects the optimality of collision avoidance decisions. Finally, the ORCA algorithm fails to fully consider the kinematic limitations of the agent, especially when the kinematics are nonlinear, which will result in complex nonlinear programming problems.
[0086] In view of these limitations, the present invention uses the ORCA algorithm as an auxiliary tool in the early stage of reinforcement learning to help the intelligent agent quickly obtain positive feedback. In the later stage of learning, the intelligent agent mainly explores the environment and pays attention to time-related constraints to make up for the shortcomings of the ORCA algorithm and achieve a better navigation strategy. Through this gradual transition strategy, it aims to improve the intelligent agent's autonomous obstacle avoidance ability in complex dynamic environments.
[0087] In multi-agent systems, in order to meet the requirements of Markov decision processes, strategy and value estimation must consider not only the state of a single agent, but also the state of other agents, the so-called joint state. However, the complexity of the joint state can sometimes be very high, making it difficult to calculate and process.
[0088] To solve this problem, an improved multi-agent proximal policy optimization algorithm is proposed, which combines the individual state of the agent with the state of other agents. In the value estimation process, MAPPO uses the joint state of all agents to provide a global perspective in order to more accurately estimate future rewards. In the strategy estimation process, in order to simplify calculations and improve decision-making efficiency, only the individual state of the agent is used for decision-making, and each agent is trying to learn how to maximize its total reward in the interaction with other agents. It not only retains the global perception ability in the multi-agent environment, but also avoids excessive complexity in processing joint states, so that agents can learn and perform tasks more effectively in the multi-agent system.
[0089] The Multi-Agent PPO (MAPPO) algorithm is a reinforcement learning algorithm for multi-agent systems. This algorithm is an extension of the PPO algorithm and is also based on the CTED framework. It belongs to the AC class of algorithms. The PPO algorithm optimizes the strategy by using an objective function that limits the deviation between the new strategy and the old strategy, which helps to avoid extreme strategy updates during the learning process. MAPPO adds a global value function to the PPO algorithm to promote collaboration among agents. In the Actor network, each agent uses a strategy based on its current local observations. Select action a i ;In the Critic network, input the joint observation of all agents O = (o 1 ,o 2 ,...,o n ) and the combined action A=(a 1 ,a 2 ,...,a n ), evaluates the value of the action, and provides feedback to the Actor network to guide subsequent action selection. MAPPO uses a policy gradient method to gradually update the agent's policy to avoid large deviations from the previous policy.
[0090] The objective function of the actor in the MAPPO algorithm is:
[0091]
[0092] Among them, ε is the clipping parameter.
[0093] In order to better cope with multi-agent reinforcement learning tasks, a reinforcement learning framework based on MAPPO is designed. A bidirectional gated recurrent unit (BiGRU) for temporal data processing and a series of fully connected layers are integrated in the architecture to optimize the ability to make decisions in partially observable multi-agent environments. Multi-agent systems usually need to process sequential data in order to make wise decisions based on the observations of the agents and the state of the environment. This framework combines sequence processing capabilities with policy and value networks to output estimates of action state values, providing correct action guidance for the agents.
[0094] The network structure of the path planning framework based on MAPPO is as follows Figure 6 As shown, including:
[0095] (1) Observation input: O self Represents the current agent's own observation space, Represents observations of other agents or obstacles in the environment.
[0096] (2) Bidirectional Gated Recurrent Unit (Bi-GRU): Processes the forward and backward input time series separately. This solution can help the network find the potential relationship between input vectors more accurately with limited information, thereby better presenting the input sequence. In each step, a series of observation vectors are sorted and input into the Bi-GRU model to obtain two hidden states, the forward h for and backward h back The two state outputs are combined to form a fixed-size 256-dimensional feature representation vector h m .
[0097] (3) Output h m and the ontological observation space O self Connect them to form a comprehensive fixed-length observation vector O.
[0098] (4) Layer Normalization (LN): The input observation vector is normalized to have a zero mean and unit variance, which helps stabilize the training, speed up the training process, and ensure that all parts have a common scale. The processed observation vector is used as the input of the two neural networks.
[0099] (5) Fully connected layer (FC): A series of dense layers that further process the normalized observations to abstract higher-level features. Each fully connected layer here contains 256 rectified linear units (ReLU) for nonlinear transformation.
[0100] (6) Output: Output an estimate of the state value. Here, two neural networks output V mean and V ψ (o), where the first neural network is the action actor’s policy network π θ , consists of two hidden fully connected layers (FC), each with 256 rectified linear units; the last layer outputs a two-dimensional vector, the vector V mean represents the expected return of the policy given the current observation and is activated by the hyperbolic tangent activation function (Tanh). The action output is sampled from a Gaussian distribution, defining a probability distribution in the action space. This distribution is represented by its action mean V mean and logarithmic standard deviation V log std Composition, where V log std are updated independently. The second neural network is the value network of the policy critic, which also consists of two hidden fully connected layers (FC), each with 256 rectified linear units. The last layer is activated by the identity function, and its output is a scalar value V ψ (o), used to evaluate each state-action pair.
[0101] The agent cluster realizes obstacle avoidance and path planning in a shared working area. It relies on the position and velocity set in the XY plane for planning. Therefore, the path planning problem can be transformed into an optimization problem of finding a series of optimal speeds to minimize the walking time under the collision avoidance constraint. Each agent works independently without communicating with its companions, but can detect the existence of its companions within a specific distance range. The information mastered by each agent includes its own radius, instantaneous speed, direction of travel, and target speed. The target speed is the maximum speed that the agent can reach when moving directly from the current position to the target when assuming that there are no obstacles. The information of surrounding agents detected by the current agent includes their radius, relative position and speed. By inputting these data vectors into the neural network, a collision avoidance strategy that can adapt to the environment is trained. Considering the impact of the unselectable speed area and the potential collision time on the current speed, a reward function that can evaluate the state and behavior is designed. The path planning strategy is trained in a simulation environment and optimized using the MAPPO algorithm. Most reinforcement learning methods directly output the control vector as a strategy. The model output of the proposed method is the rate of change of speed, thereby achieving a smoother and more efficient agent control strategy.
[0102] Therefore, the research problem is modeled. n disk-shaped agents are set in a shared workspace. The agents perceive the positions of themselves and the target points in the map at every moment. The radius of agent i is set to r, and its maximum speed is limited to v. max , the state at time t for speed for These agents all have a sensing range, and can sense m neighboring agents within a sensing range of n meters and detect the distance to these agents. The size of v max , the direction is the straight line path direction from the current position to the target point, Expressed as The observation space of each agent is divided into two parts: the body perception state and the external perception state of the surrounding environment, namely The proprioceptive observation space includes the agent’s current speed, direction, expected speed, and virtual collision avoidance radius, i.e. The external observation contains the position p of the jth neighboring agent or obstacle at the current speed j , speed v j and the radius R j , so the external observation state space is with o self and The joint environment observation is input, and θ is the policy neural network π θParameters, the output action is the speed increment of the agent at each time step, that is, a = Δv t =[Δv xt ,Δv yt ]. If the speed at the next time step is not only collision-free but also has the smallest difference from the agent's expected speed, the calculated speed increment is optimal. That is, for each agent, the following constrained optimization problem needs to be solved at each time step Δt:
[0103]
[0104] Among them, ||·|| is the Euclidean norm operation of the vector, μ is a hyperparameter for adjusting the speed increment range, and d j is the agent with position p jt , the collision radius is R jc The distance between the jth neighbors of . All robots share the same navigation strategy π θ , and find the optimal speed independently.
[0105] The goal of the agent is to reach the end point. When the distance between the agent and the target point is ρ t Less than the threshold d goal When , it is considered to have reached the target point, and a fixed large reward is given to guide the agent to complete the task. In addition, since the above reward is a sparse reward, that is, it can only be generated when the agent reaches the end point, this kind of reward is difficult to obtain in the early stage of training, which is not conducive to the continuity of learning. Therefore, an implicit reward r is added collision , so that the agent can be rewarded or punished instantly at every moment, which can also guide the agent to reach the target more accurately. This reward integrates the Optimal Reciprocal Obstacle Avoidance (ORCA) algorithm. Since ORCA observations do not directly contain position and speed information, the speed selected by the ORCA algorithm in each state and its feasible domain are calculated, and the agent speed generated by the current strategy is compared with the feasible domain. If the speed is within the feasible domain, a positive reward is given, otherwise a negative reward is given to guide the agent to perform actions within the speed feasible domain.
[0106] The ray method is used to determine whether the agent speed is in the ORCA feasible domain. The principle of the ray method can be briefly described as follows: starting from the target point speed, draw a ray perpendicular to the y-axis, and count the number of intersections between this ray and all edges in the polygonal feasible domain. If there are an odd number of intersections with each edge, it means that the agent speed is inside the ORCA feasible domain; if there are an even number of intersections, it means that the agent speed is outside the feasible domain. Figure 7 As shown in the figure, the figure is a concave polygon with 13 sides. The red point is the target point that needs to be judged whether it is inside the polygon. The ray has an odd number of intersections with each side of the polygon, so the red point is inside the polygon.
[0107] In addition, the expected collision time is a better representation of the collision risk than the distance to the obstacle, because the expected collision time takes into account both the distance to the obstacle and the relative speed between the agent and the obstacle. Based on the expected collision time, the reward function is designed as shown in formula (10):
[0108]
[0109] The implicit reward function r of each state collision The design is as follows:
[0110]
[0111] For implicit rewards, diff in formula (11) v The distance between the speed selected by the strategy and the expected speed calculated by the ORCA algorithm, i.e., diff v =||v t -v des ||, ξ is the expected minimum time for the agent to collide with an obstacle at the current speed. a, b, c, d, and e are all constant values that can adjust the policy performance. a and c are the basic rewards that encourage behavior when the current speed is close to the expected speed or the current collision risk is low. These two values are often positive. b is the diff v The coefficient of the weight indicates the desired speed of movement. d, e is (ξ+f) -1 The coefficient of the weight, (ξ+f) -1 represents the motion that maintains a longer expected collision time, (ξ+f) -1 The value range is [0,f -1 ]. In summary, for crowded scenes, larger d and e values are used to ensure collision avoidance. The main idea is to give the agent a positive reward when the speed is in the feasible region of ORCA or the expected collision time is greater than 5 seconds. When the speed is not in the feasible region of ORCA and the expected collision time is in the range of 0.1 seconds to 5 seconds, the agent should pay attention to the potential collision judged by the expected collision time and give a smaller negative reward. When the expected collision time is less than or equal to 0.1 seconds, the task can be considered to have failed, and a larger negative reward is given to the agent to guide the agent to avoid collisions in subsequent exploration.
[0112] The process of multi-agent reinforcement learning as shown in N is:
[0113] (1) Environment interaction: Each agent (Agent 1 to Agent N) receives initial state information from the environment.
[0114] (2) Local observation processing: Each agent processes its local observation data, namely its own radius, instantaneous speed, direction of travel, and target speed, and needs to consider historical information to maintain the consistency of the state.
[0115] (3) Strategy execution: Each agent uses its strategy (defined by the behavioral neural network Actor) to select actions based on local observations and executes them in the environment, resulting in changes in state and the generation of rewards.
[0116] (4) Global evaluation: Evaluates the actions performed by the neural network critic and calculates an estimate of the value function based on the global state and the agent’s actions.
[0117] (5) Strategy update: Update the Actor’s strategy based on the value function estimate provided by the Critic and the actual reward obtained.
[0118] (6) Experience replay: The interaction experience (state, action, reward, etc.) is stored in the experience replay buffer for subsequent learning and strategy updating.
[0119] (7) Iterative update: Follow the above steps repeatedly until the termination condition is met, that is, the maximum number of training steps is reached or the strategy converges stably.
[0120] In order to verify the feasibility of the collaborative control method in this embodiment, this embodiment sets up multiple simulation environments, uses the deep reinforcement learning algorithm to perform path planning tests in the simulation environment, and analyzes the simulation experiment results from three indicators: success rate, driving time and average speed, and discusses the performance of the algorithm under different conditions and its stability.
[0121] The MAPPO-ORCA path planning strategy is trained using a simulation scenario and implemented using Pytorch (Python 3.8). The simulation scenario is developed based on OpenAI Gym, a popular toolkit for developing reinforcement learning algorithms, and Matplotlib is selected as the visualization interface. In the simulation environment, multiple disk-shaped agents are used to train the collision avoidance strategy in a 10*10 blank scene. The perception range of each agent is d j The distance is fixed at 4 meters. Within the sensing range, up to 5 neighboring agents can be sensed at the same time according to the distance. All agents are distributed equidistantly along the circumference, and the opposite side facing the center of the circle is the destination of each agent. This layout promotes frequent interactions between agents, which helps the system to train strategies that meet the needs more quickly. The simulation environment is as follows Fig. 9 As shown, in this environment, Fig. 9 The four agents shown in a) and Fig. 9 The 10 agents shown in b) are trained.
[0122] In the feasibility verification of the strategy, static or dynamic obstacles can be added to the simulation environment to block the movement of the agent, such as Fig.10 As shown, add 4 dynamic obstacles.
[0123] like Fig.11 As shown in the figure, random starting points and target points are set in the simulation environment. The starting point of the agent is the blue point, and the target point is the corresponding yellow point. Each agent has only a unique target point to verify the generalization of the algorithm.
[0124] During the training process, all agents execute the same strategy. After each training cycle, parameters are collected and strategies are updated according to the MAPPO algorithm. Once an agent collides or exceeds the set maximum number of steps, the simulation environment will reset.
[0125] The main problem of deep reinforcement learning tasks is the trade-off between the agent's exploration and development of the environment. It is difficult to converge directly in a high-dimensional complex environment, and a lot of exploration is required to find an effective strategy. In order to speed up the convergence of training, a phased training method is adopted. Training in a simple environment can ensure that the agent's strategy converges quickly on simple tasks. The agent can build a basic understanding of the environment and avoid blind exploration in subsequent complex environments. In a complex environment, the optimal strategy obtained in a simple environment is used as a pre-training model for the current stage, and it is expanded and adjusted on its existing strategy framework instead of learning from scratch. This can reduce the exploration cost and avoid the agent's disorderly exploration, thereby improving learning efficiency.
[0126] In the initial stage of the experiment, four agents were set up for basic training in a relatively simple environment to train a path planning strategy that meets the current environment. After 250 training cycles, the current optimal strategy was used as the initial strategy. Fig. 9 In the environment, 10 agents are set up and training is continued for 2000 cycles. Through such progressive training, the current optimal strategy can be successfully obtained, ensuring that the agent can effectively plan the path and reach the destination without collision. The hyperparameter selection during the training process is shown in Table 1.
[0127] Table 1 Hyperparameters used in strategy training
[0128]
[0129] During the training process, the MAPPO algorithm will not terminate the current round of training due to reaching the target point or collision with some agents. It will only reset the environment and continue to explore the environment. In the setting of the reward function, each step has an implicit reward function rcollision , there will be a large sparse reward when reaching the target point. In order to better reflect the impact of the reward function on the training process, the accumulated rewards for a single agent in each round for a complete exploration process are counted, that is, the accumulated reward value corresponding to the accumulated number of steps taken by the agent from the starting point to the target point or collision. Considering that the agent can complete multiple exploration processes in one round, the average of these accumulated reward values is taken as the accumulated reward of the current agent in the current round. Since multiple agents use the same planning strategy during the training process, the accumulated rewards of these agents are averaged to obtain the average reward of the round to be analyzed.
[0130] During the training process, the reward function curve obtained by the training strategy of 4 agents training for 250 rounds at the same time is as follows Fig.12 As shown in the figure, the x-axis represents the number of training rounds, the y-axis represents the average reward obtained in each round, the blue curve is the average reward obtained after adding the bidirectional gated recurrent unit (BiGRU) module to the neural network architecture, and the red curve is the average reward obtained after using the MAPPO-LSTM-ORCA algorithm (adding a long short-term memory network to the neural network architecture). In order to see the changing trend of the reward curve more clearly, the original data of the reward curve is sliding averaged. Fig.12 It can be seen that in the early stage of training, the average rewards of both algorithms are low, and the average reward of MAPPO-BiGRU-ORCA is lower. As the number of rounds increases, the reward curves of the two multi-agent algorithms show a fluctuating upward trend, among which MAPPO-BiGRU-ORCA rises faster. After about 120 rounds, the average round reward of the algorithm tends to be stable, and the curve is relatively smooth; for MAPPO-LSTM-ORCA, the average reward value and convergence speed are slightly lower than the algorithm with the BiGRU module added. The experimental results show that after adding the BiGRU module, the improved algorithm has better performance, which verifies that the module can provide the algorithm with better time series data processing capabilities, thereby discovering more potential relationships between surrounding agents and obtaining better performance in the environment.
[0131] Three indicators are used in the experiment to evaluate the quality of the strategy, including success rate, travel time and average speed. The success rate refers to the ratio of the agent to successfully reach the destination without colliding or getting stuck somewhere during the journey, which describes the collision avoidance ability of the strategy. Travel time refers to the amount of time expressed in the number of iterations for all agents in the simulation to reach the target location, reflecting the efficiency of the strategy. The average speed is the overall average moving speed of the multi-agent team navigation process, which measures the performance of the strategy in the selection of effective speed and reflects the adaptability and coordination of the team in the face of complex environments. Training was continued for 2000 rounds in a denser environment containing 10 agents. The resulting reward function curve is shown below. Fig.13 shown.
[0132] from Fig.13 It can be seen that the average round reward of MAPPO-BiGRU-ORCA is almost always higher than that of MAPPO-LSTM-ORCA, and as the number of training rounds increases, the average reward value of the former has been on an upward trend and tends to stabilize when approaching 1000 rounds. There is no sharp drop in strategy rewards. The curve is smoother, showing higher stability, while the latter is more volatile, reflecting that the multi-agent information using LSTM modules is not sufficient to solve the problem of multi-agent collaborative control in crowded workspaces.
[0133] During the training process, the policy model generated in the current round of the training process is saved every 100 rounds and tested in the training environment. The exploration strategy is not used during the test. The action instructions generated by the policy network control the agents to navigate to their respective target points. During the operation, all agents do not collide, and the task is considered completed when they reach the target point within the limited number of steps. Each model is tested 100 times in the environment, and the model task success rate, the number of steps required to complete the task, and the average speed of the agents of MAPPO-BiGRU-ORCA and MAPPO-LSTM-ORCA in different training rounds are counted, as shown in the figure below: Fig.14 As shown in the figure, the horizontal axis is the number of training rounds, and the vertical axis is the task success rate. The initial success rate of MAPPO-BiGRU-ORCA is lower, but the success rate increases faster and has a more stable performance, without a significant drop in strategy. However, the success rate of MAPPO-ORCA increases slowly, and the algorithm stability is not high enough, resulting in a significant drop in success rate.
[0134] The two models with the best verification effect and the highest corresponding success rate in the training model were tested in a test simulation environment with 10 agents. Each model was tested 100 times and their success rate, average step length required to reach the target and average speed were counted. The results are shown in Table 2.
[0135] Table 2 Algorithm index comparison
[0136]
[0137] As can be seen from Table 2, the multi-agent obstacle avoidance and path planning algorithm using the BiGRU module performs better in complex environments and can reach the corresponding target point with fewer steps and faster speed, thus completing the task.
[0138] The strategies generated by MAPPO-BiGRU-ORCA algorithm training are tested in different scenarios. Fig.15 This is a test of the obstacle avoidance navigation of four agents in opposite directions. The blue line is the walking trajectory of the agent. At this time, the training strategy obtained by training four agents at the same time for 250 rounds is used. Fig.15 It can be seen that the agents successfully avoided each other and reached their respective target points.
[0139] Fig.16 This is a test of 10 agents' opposing motion obstacle avoidance navigation. In this case, the optimal training strategy is obtained by using 10 agents to train for 2000 rounds simultaneously. Fig.16 As shown in a), at the 0th time step, 10 agents are distributed equidistantly along the circumference of the square scene, and the target point of each agent is the opposite side facing the center of the circle. Fig.16 As shown in b), each agent moves toward the center of the environment. Fig.16 c) and Fig.16 As shown in d), the agents are concentrated in the central area, and the external perception environment of each agent becomes more complex. The agents adopt various behavior modes to avoid the obstacles of other agents and move towards their respective target points. Fig.16 As shown in e), all agents have left the dense area. Fig.16 As shown in f), at the 65th time step, all 10 agents successfully generated feasible trajectories and reached the target point.
[0140] Since the trained navigation strategy focuses on the agent avoiding collisions and reaching the target point, in order to verify the performance of the strategy in an environment with dynamic obstacles, four circular dynamic obstacles are added to the simulation environment. The radius of the obstacle is set to 0.1m, the maximum speed is 0.3m / s, and the obstacles can move randomly in the environment. Fig.17 This is the path planning result after adding 4 dynamic obstacles. The experimental results show that although the scene becomes more crowded, each agent can still avoid other agents or dynamic obstacles and successfully reach the target point.
[0141] Since 10 agents were used for path planning at the same time during training, in order to verify the scalability of the team size, 20 agents were added to the same environment and were required to navigate to their respective target points without collision. Fig.18The path planning results verified in the 10m*10m scenario. From the experimental results, it can be seen that multiple agents perform their own obstacle avoidance in the middle position, and pass through the most crowded central area in turn, and finally reach their respective target points. However, the number of steps required for 20 agents is much greater than that for 10 agents. The main reason is that each agent independently adopts a strategy and does not communicate with each other, resulting in overfitting of the real-time planned path. Some agents choose to slow down or even stop, allowing other agents to pass through the blocked area first.
[0142] In order to verify the generalization ability of the path planning strategy trained from a specific scenario, the initial position and target position of each agent are randomly set, the minimum distance from the starting point to the target point is 1m, and the starting point and target point are randomly reset in each round during the test. Fig.19 It is the path planning result of 10 agents in a random scene of 10m*10m for strategy verification. It can be seen from the experimental results that all agents move along their planned routes. When there is a path conflict among agents, the path planning strategy is used to avoid obstacles or one of the agents slows down to avoid them, ensuring that there is no collision during the walking process and reaching the target point.
[0143] The strategy was used to conduct 100 tests in the current environment, with a success rate of 73.00%, an average number of steps to reach the target point of 68.49, and an average speed of 0.85. The reason why the success rate in random scenarios is not as good as that in specific scenarios is that the strategy obtained by reinforcement learning is overfitted to the training environment, and the optimal strategy found in the training environment cannot show excellent performance in random environments.
[0144] In view of the high computational complexity and long computational time in traditional methods, this embodiment proposes a multi-agent path planning method combining MAPPO and ORCA algorithms. The state space and action space of the experimental environment are designed according to the requirements of the experimental environment. The concept of ORCA feasible domain is introduced in the reward function to improve the obstacle avoidance performance of the multi-agent in the environment. In addition, a bidirectional GRU module is added to the algorithm to optimize the decision-making ability in a partially observable multi-agent environment. Subsequently, a network training process is designed and a simulation experiment is conducted. The algorithm with the LSTM module added is compared. The experimental results show that the algorithm with BiGRU added has a faster convergence speed and better reward stability, which verifies that the BiGRU module can provide the algorithm with better time series data processing capabilities, thereby discovering more potential relationships between surrounding agents and obtaining better performance in the environment. Finally, the MAPPO-BiGRU-ORCA algorithm is used to verify the effectiveness of the algorithm in multi-agent path planning. The experimental results show that the algorithm can realize path planning in a variety of scenarios. In addition, the performance of the algorithm is analyzed in detail, and the performance and stability of the algorithm under different conditions are discussed.
[0145] This embodiment performs collaborative control of multi-agents based on a multi-agent obstacle avoidance and path planning algorithm that combines multi-agent proximal strategy optimization with optimal reciprocal obstacle avoidance. The optimal reciprocal obstacle avoidance feasible domain is introduced into the reward function, and a bidirectional GRU module is added to the multi-agent proximal strategy optimization to optimize the decision-making ability in a partially observable multi-agent environment. It processes high-dimensional observation and action spaces, realizes collaboration and competition between agents in a dynamic environment, and can realize path planning in a variety of scenarios while reducing computational complexity and improving planning efficiency.
[0146] Embodiment 2
[0147] Embodiment 2 of the present invention introduces a multi-agent collaborative control system based on deep reinforcement learning.
[0148] A multi-agent collaborative control system based on deep reinforcement learning, comprising:
[0149] An acquisition module, configured to acquire state spaces and action spaces of several intelligent agent individuals in a multi-agent;
[0150] A construction module, which is configured to consider the implicit reward function of the individual intelligent agents and construct the optimal mutual obstacle avoidance feasible domain of the individual intelligent agents;
[0151] An optimization module is configured to perform deep reinforcement learning of multi-agents by using a multi-agent proximal strategy optimization algorithm with a bidirectional gated recurrent unit in the constructed optimal mutual obstacle avoidance feasible domain of the individual agents, optimize the action space and state space of the individual agents, and obtain the dynamic state estimation value of the multi-agents;
[0152] The collaborative control module is configured to plan the optimal path of the individual intelligent agents according to the obtained dynamic state estimation value and complete the collaborative control of multiple intelligent agents.
[0153] The detailed steps are the same as the multi-agent collaborative control method based on deep reinforcement learning provided in Example 1, and will not be repeated here.
[0154] Embodiment 3
[0155] Embodiment 3 of the present invention provides a computer-readable storage medium.
[0156] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in Embodiment 1 of the present invention.
[0157] The detailed steps are the same as the multi-agent collaborative control method based on deep reinforcement learning provided in Example 1, and will not be repeated here.
[0158] Embodiment 4
[0159] A fourth embodiment of the present invention provides an electronic device.
[0160] An electronic device comprises a memory, a processor and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in Example 1 of the present invention are implemented.
[0161] The detailed steps are the same as the multi-agent collaborative control method based on deep reinforcement learning provided in Example 1, and will not be repeated here.
[0162] Embodiment 5
[0163] Embodiment 5 of the present invention provides a computer program product.
[0164] A computer program product includes software code, wherein the program in the software code executes the steps in the multi-agent collaborative control method based on deep reinforcement learning as described in Example 1 of the present invention.
[0165] The detailed steps are the same as the multi-agent collaborative control method based on deep reinforcement learning provided in Example 1, and will not be repeated here.
[0166] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.
Claims
1. A multi-agent collaborative control method based on deep reinforcement learning, characterized in that: include: Obtain the state space and action space of several intelligent agents in the multi-agent system; Considering the implicit reward function of the individual intelligent agent, the optimal mutual obstacle avoidance feasible domain of the individual intelligent agent is constructed; In the optimal mutual obstacle avoidance feasible domain of the constructed intelligent individuals, a multi-agent proximal strategy optimization algorithm with a bidirectional gated recurrent unit is used to perform deep reinforcement learning of the multi-agents, optimize the action space and state space of the intelligent individuals, and obtain the dynamic state estimation value of the multi-agents. According to the obtained dynamic state estimation value, the optimal path of the individual intelligent agent is planned to complete the collaborative control of multiple intelligent agents.
2. A multi-agent collaborative control method based on deep reinforcement learning as described in claim 1, characterized in that: In the process of multi-agent deep reinforcement learning using a multi-agent proximal policy optimization algorithm with a bidirectional gated recurrent unit, a global value function is added to promote collaboration between individual agents; in the behavioral neural network, the individual agents use strategies to select actions based on current local observations; in the evaluation neural network, the multi-agent actions are evaluated based on the joint observations and joint actions of all individual agents, and the evaluation results are fed back to the Actor network to guide the selection of actions of other individual agents; the individual action space and state space of the agents are optimized through continuous cycles of iterations to obtain the dynamic state estimation value of the multi-agent.
3. A multi-agent collaborative control method based on deep reinforcement learning as described in claim 1, characterized in that: The implicit reward function r collision for Among them, diff v The distance between the speed selected by the strategy and the expected speed calculated by the optimal reciprocal obstacle avoidance method, i.e., diff v =||v t -v des ||, is the expected speed of the agent, ORCA is the optimal mutual obstacle avoidance feasible domain of the agent, v t is the current speed of the agent; ξ is the expected minimum time for the agent to collide with an obstacle at the current speed; a, b, c, d, and e are all constant values, where a and c are the basic rewards for encouraging behavior when the current speed is close to the expected speed or the current collision risk is low, and b is the diff v The coefficient of the weight, that is, the desired speed of movement, d and e are both (ξ+f) -1 Weight coefficient; (ξ+f) -1 Indicates a motion that maintains a longer expected collision time.
4. A multi-agent collaborative control method based on deep reinforcement learning as described in claim 1, characterized in that: In the process of planning the optimal path of the intelligent individual, the speed of the dynamic obstacles is taken into consideration, and the relative position between the dynamic obstacles and the intelligent individual is converted into speed in combination with the time window. Based on the optimal reciprocal obstacle avoidance feasible domain of the intelligent individual, the optimal speed with the smallest adjustment amount with the speed of the intelligent individual is selected to realize the autonomous obstacle avoidance of the intelligent individual in a dynamic environment and complete the planning of the optimal path for the intelligent individual.
5. A multi-agent collaborative control method based on deep reinforcement learning as described in claim 1, characterized in that: Considering the implicit reward function, the speed of the individual agent and its optimal reciprocal obstacle avoidance feasible domain are calculated in each state. If the obtained speed is within the optimal reciprocal obstacle avoidance feasible domain, a positive reward is given to the individual agent, otherwise a negative reward is estimated to guide the agent to perform actions within the obtained optimal reciprocal obstacle avoidance feasible domain.
6. A multi-agent collaborative control method based on deep reinforcement learning as described in claim 5, characterized in that: The ray method is used to determine whether the speed of the individual intelligent agent is within the optimal reciprocal obstacle avoidance feasible domain. Specifically, a horizontal straight line is drawn from the speed point to be determined, and the number of intersections of the horizontal straight line with all edges in the optimal reciprocal obstacle avoidance feasible domain is counted. If there are an odd number of intersections with each edge, the speed is within the optimal reciprocal obstacle avoidance feasible domain. If there are an even number of intersections with each edge, the speed is not within the optimal reciprocal obstacle avoidance feasible domain.
7. A multi-agent collaborative control system based on deep reinforcement learning, characterized in that: include: An acquisition module, configured to acquire state spaces and action spaces of several intelligent agent individuals in a multi-agent; A construction module, which is configured to consider the implicit reward function of the individual intelligent agents and construct the optimal mutual obstacle avoidance feasible domain of the individual intelligent agents; An optimization module is configured to perform deep reinforcement learning of multi-agents by using a multi-agent proximal strategy optimization algorithm with a bidirectional gated recurrent unit in the constructed optimal mutual obstacle avoidance feasible domain of the individual agents, optimize the action space and state space of the individual agents, and obtain the dynamic state estimation value of the multi-agents; The collaborative control module is configured to plan the optimal path of the individual intelligent agents according to the obtained dynamic state estimation value and complete the collaborative control of multiple intelligent agents.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multi-agent collaborative control method based on deep reinforcement learning as described in any one of claims 1 to 6 are implemented.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the multi-agent collaborative control method based on deep reinforcement learning as described in any one of claims 1-6 are implemented.
10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the multi-agent collaborative control method based on deep reinforcement learning as described in any one of claims 1-6.
Citation Information
Cited By
Control method and control system of multi-agent water plant process system
CN119356270A
Multi-robot cooperative measurement path planning method based on reinforcement learning
CN120651248A
Power trusted wireless local area network access point layout design method and system
CN120881595A
Multi-agent sparse reward exploration method and system based on meta learning
CN122065872A
Method and device for constructing digital power system based on multifunctional intelligent agents
WO2026112864A1
