Multi-agent formation obstacle avoidance method and device, and storage medium

Abstract

The application relates to the technical field of automatic obstacle avoidance, and discloses a multi-agent formation obstacle avoidance method, a device and a storage medium, wherein the method comprises the following steps: S100: an action value network model and a training environment model are constructed, the action value network model is trained by using the training environment model, an action value network model of multi-agent formation obstacle avoidance is obtained, and the coordinates of a dynamic obstacle and the coordinates of an end point of an agent queue are randomly set in each training process; S200: each agent respectively observes observation data required by the action value network model, and inputs the observation data into the action value network model; the action value of all actions of each agent in a respective working space is evaluated by the action value network model; and the optimal action of each agent is obtained; S300: each agent respectively executes the respective optimal action until all agents successfully arrive at the end point. The method can improve the adaptability of the agents to different obstacle avoidance situations.

Description

Technical Field

[0001] This invention relates to the field of automatic obstacle avoidance technology, specifically to a multi-agent formation obstacle avoidance method, device, and storage medium. Background Technology

[0002] In recent years, with the rapid advancement of artificial intelligence technology, intelligent agent systems such as self-driving cars and drones have been gradually promoted in the civilian sector and deployed in increasingly complex operating environments. However, given the functional limitations of a single intelligent agent, in actual deployment, multiple intelligent agents often need to cooperate to achieve effective task completion. In this process, formation control and obstacle avoidance have become crucial topics in the research field of multi-agent systems.

[0003] Traditional obstacle avoidance strategy models require adjusting specific parameters during training based on the actual situation. They are poorly adaptable to unknown and complex environments and cannot provide effective solutions when the agent faces obstacles from both the front and back. Furthermore, while existing relative velocity-based obstacle avoidance algorithms can demonstrate excellent performance in complex and dynamic environments, they require all agents to follow the same obstacle avoidance strategy, which is difficult to meet in real-world obstacle avoidance applications involving multiple agents. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a multi-agent formation obstacle avoidance method that can improve the adaptability of multi-agents to complex environments and increase the obstacle avoidance success rate.

[0005] To solve the above problems, the technical solution adopted by the present invention is as follows: a multi-agent formation obstacle avoidance method, comprising the following steps:

[0006] S100: Construct an action value network model and a training environment model. Use the training environment model to train the action value network model to obtain a multi-agent formation obstacle avoidance action value network model. The training environment model includes an agent queue consisting of multiple agents and multiple dynamic obstacles. During each round of training, the coordinates of the dynamic obstacles and the endpoint coordinates of the agent queue are randomly set.

[0007] S200: Each agent observes the observation data required by the action value network model and inputs it into the action value network model. The action value network model evaluates the action value of all actions of each agent in its own workspace and obtains the optimal action with the highest action value for each agent.

[0008] S300: Each agent executes its own optimal action until all agents successfully reach the destination.

[0009] Compared to existing technologies, the advantages of this invention are as follows: By establishing a training environment that includes multiple agents and dynamic obstacles, and randomly setting the coordinates of the dynamic obstacles and the endpoint coordinates of the agent queue in the training environment, specific parameters need to be adjusted for different scenarios, thereby improving the agent's adaptability to unknown and complex environments. The trained action value network model outputs the optimal action for each agent, allowing each agent to execute its own optimal action without having to completely follow the same obstacle avoidance strategy, which is easily satisfied in practical application environments.

[0010] The multi-agent formation obstacle avoidance method described above, wherein step S100 includes:

[0011] S110: Initialize the parameters of the current action value network and the target action value network, and initialize an experience pool;

[0012] S120: Set up a training environment model, which includes an agent queue consisting of multiple agents and multiple dynamic obstacles.

[0013] S130: Initialize the training environment model, reset multiple agents to their respective starting points, and randomly set the coordinates of dynamic obstacles and the endpoint coordinates of the agent queue;

[0014] S140: Evaluate all actions of each agent in the action space through the current action value network, and obtain the action with the highest action value for each agent;

[0015] S150: Make each agent move according to the action with the highest action value determined in step S140, and obtain feedback from the training environment model.

[0016] S160: If the training environment model reports that the agent has not collided and the agent queue has not reached the end point, then the interaction data between the agent queue and the training environment model is stored in the experience pool in a preset format; if the training environment model reports that the agent has collided or the agent queue has reached the end point, then return to step S130.

[0017] S170: Take a batch of samples from the experience pool, evaluate the approximate actual value of the samples through the target action value network, calculate the loss function based on the approximate actual value and the action value evaluated by the current action value network, and optimize the parameters of the current action value network by gradient descent based on the loss function.

[0018] S180: Every preset number of rounds, synchronize the parameters of the current action value network to the target action value network;

[0019] S190: If the number of training rounds reaches the preset maximum number of rounds, then save the target action value network as an action value network model.

[0020] In the multi-agent formation obstacle avoidance method described above, in step S100, the reward function of the action value network model during training includes an obstacle avoidance reward function, a formation reward function, and a navigation reward function. The obstacle avoidance reward function encourages agents to avoid obstacles and avoid collisions between agents. The formation reward function encourages agents to maintain a given formation shape during actions. The navigation reward function encourages agents to move towards the destination. The reward value for each action of an agent is a weighted sum of the values ​​of the obstacle avoidance reward function, the formation reward function, and the navigation reward function.

[0021] The above-mentioned multi-agent formation obstacle avoidance method is characterized in that the obstacle avoidance reward function r a Defined as:

[0022]

[0023] In the formula, Represents a set of intelligent agents. Let b, c, and g represent the set of dynamic obstacles. a All are constant coefficients, δ ik δ represents the distance between an agent and other agents or dynamic obstacles. min This represents the minimum collision radius of the agent.

[0024] The multi-agent formation obstacle avoidance method described above, wherein the formation reward function r f Defined as:

[0025]

[0026] In the formula, ||.|| F Let tr denote the Frobenius norm, and tr{.} denote the trace of the matrix. The normalized Laplace matrix of the current formation of the agents. It is the normalized Laplace matrix of the ideal formation.

[0027] The above-mentioned multi-agent formation obstacle avoidance method, wherein the navigation reward function r n Defined as:

[0028]

[0029] In the formula, Represents a set of intelligent agents. and These are the distances from the agent to the destination at the current time step and the previous time step, respectively, D. g The radius of the endpoint, g g The bonus coefficient for reaching the finish line.

[0030] The multi-agent formation obstacle avoidance method described above includes an action value network model comprising a single LSTM network and two MLP networks. The observation values ​​of dynamic obstacles observed by the agents are sequentially input into the LSTM neural network in order of distance from the agents from farthest to closest to extract the feature values ​​of the dynamic obstacles. The agents input their own observation values ​​into the first layer of the MLP network to extract their own feature values. The feature values ​​of the dynamic obstacles and the feature values ​​of the agents are concatenated and input into the second layer of the MLP network to obtain the action value of the action.

[0031] A storage medium storing a computer program, which, when executed by a processor, implements the aforementioned multi-agent formation obstacle avoidance method.

[0032] A multi-agent formation obstacle avoidance control device includes: an acquisition module for acquiring observation data and endpoint information of the agent itself, dynamic obstacles, other agents, and other agents; an action value evaluation module for evaluating the action value of all actions in the action space of each agent based on a dynamic value network model and the observation data acquired by the acquisition module, and outputting the action with the highest action value for each agent; a training environment module for providing a training environment model for training the dynamic value network model, updating the position coordinates of each agent based on the input actions of each agent, and determining whether an agent collides with other agents or dynamic obstacles based on the position coordinates of each agent; a reward calculation module for calculating the reward of an action based on the state of the agent before and after the action during the training of the dynamic value network model; a model training module for training the dynamic value network model using the training environment module, controlling the initialization of the training environment module, and randomly setting the coordinates of dynamic obstacles and the endpoint of the agent queue in the training environment module; and a control module for controlling the agents to take actions according to the action with the highest action value output by the action evaluation module.

[0033] The aforementioned multi-agent formation obstacle avoidance control device includes a reward calculation module comprising: an obstacle avoidance reward calculation module for calculating the obstacle avoidance reward value of an action to encourage agents to avoid obstacles and prevent collisions between agents; a formation reward calculation module for calculating the formation reward value of an action to encourage agents to maintain a given formation shape during the action; and a navigation reward calculation module for calculating the navigation reward value of an action to encourage agents to move towards the destination. The reward calculation module calculates the final reward value of an action by calculating the weighted sum of the obstacle avoidance reward value, formation reward value, and navigation reward value.

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0035] Figure 1 This is a flowchart of a multi-agent formation obstacle avoidance method according to an embodiment of the present invention;

[0036] Figure 2 This is a flowchart illustrating the training process of the action value network model according to an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the action value network model according to an embodiment of the present invention. Detailed Implementation

[0038] The embodiments of the present invention are described in detail below, with reference to... Figure 1 The present invention provides a multi-agent formation obstacle avoidance method, comprising the following steps:

[0039] S100: Construct an action value network model and a training environment model. Use the training environment model to train the action value network model to obtain a multi-agent formation obstacle avoidance action value network model. The training environment model includes an agent queue consisting of multiple agents and multiple dynamic obstacles. During each round of training, the coordinates of the dynamic obstacles and the endpoint coordinates of the agent queue are randomly set.

[0040] S200: Each agent observes the observation data required by the action value network model and inputs it into the action value network model. The action value network model evaluates the action value of all actions of each agent in its own workspace and obtains the optimal action with the highest action value for each agent.

[0041] S300: Each agent executes its own optimal action until all agents successfully reach the destination.

[0042] The obstacle avoidance method of this invention establishes a training environment including multiple agents and dynamic obstacles. Using this training environment, an action value network model is trained. By randomly setting the coordinates of the dynamic obstacles and the endpoint coordinates of the agent queue, the action value network model can perform reinforcement learning for different scenarios. This avoids adjusting specific parameters for different scenarios and improves the agents' adaptability to unknown and complex environments. The trained action value network model outputs the optimal action for each agent. Each agent executes its own optimal action output by the action value network model, performing the formation and obstacle avoidance task in a distributed manner until all agents successfully reach the endpoint. Multiple agents do not need to completely follow the same obstacle avoidance strategy, which is easier to satisfy in real-world application environments, further improving the obstacle avoidance method's adaptability to different scenarios.

[0043] It is understandable that the action value network model is a convolutional neural network, and the training environment is a module that updates the position of each agent based on the input actions of each agent and determines whether collisions occur between agents and between agents and dynamic obstacles. The specific training process of the action value network model includes the following steps:

[0044] S110: Initialize the current action value network weights θ and the target action value network weights θ using orthogonal initialization. - The weights are initialized, and a memory space is allocated for the training program to store the training data, thus completing the initialization of the experience pool. In this embodiment, a set of training data in the experience pool D can be represented as (s t ,a t ,r t ,s t+1 ), where s t For the current time step state, a t For the action to be performed at the current time step, r t The reward s is the amount of money earned after performing an action at the current time step. t+1 Execution in s t Execute a t The transition state, that is, the state at the next time step;

[0045] S120: Set up a training environment model, which includes an agent queue consisting of multiple agents and multiple dynamic obstacles.

[0046] S130: Initialize the training environment model, reset multiple agents to their respective starting points, and randomly set the coordinates of dynamic obstacles and the endpoint coordinates of the agent queue;

[0047] S140: Evaluate all actions of each agent in the action space through the current action value network, and obtain the action with the highest action value for each agent;

[0048] S150: Make each agent move according to the action with the highest action value determined in step S140, and obtain feedback from the training environment model.

[0049] S160: If the training environment model reports that the agent has not collided and the agent queue has not reached the end point, then the interaction data between the agent queue and the training environment model is stored in the experience pool in a preset format; if the training environment model reports that the agent has collided or the agent queue has reached the end point, then return to step S130.

[0050] S170: Take a batch of samples from the experience pool, evaluate the approximate actual value of the samples through the target action value network, calculate the loss function based on the approximate actual value and the action value evaluated by the current action value network, and optimize the parameters of the current action value network by gradient descent based on the loss function.

[0051] S180: Every preset number of rounds, synchronize the parameters of the current action value network to the target action value network;

[0052] S190: If the number of training rounds reaches the preset maximum number of rounds, then save the target action value network as an action value network model.

[0053] Reference Figure 3 In this embodiment, to further improve the obstacle avoidance success rate of the agent in complex environments, the action value network model includes two layers of MLP neural networks and one layer of LSTM neural networks. The agent's own observations are directly input into the first layer of the MLP neural network for feature extraction. The observations of n obstacles observed by each agent are sorted according to their distance from the agent from farthest to closest and are sequentially input into the LSTM neural network to obtain the feature values ​​of dynamic obstacles. All perceptible obstacles are uniformly encoded. Since the feature values ​​obtained through the memory unit of the LSTM neural network are generally not on the same order of magnitude as the feature values ​​obtained directly through the MLP neural network, another layer of MLP neural network is needed after the LSTM neural network to unify their order of magnitude. Therefore, after obtaining the feature values ​​of the agent itself and the feature values ​​of the dynamic obstacles, the two are concatenated and input into the second layer of the MLP neural network to obtain the final action value.

[0054] In this embodiment, to improve training efficiency, the Temporal Difference (TD) algorithm is used to update the current value network weights θ. In step S170, the approximate actual value y of the target value network at the current time step is approximated. i for:

[0055]

[0056] Where γ is the discount factor, ranging from (0,1). Let s represent an action 'a' such that s t+1 The target action value function is maximized when the time is right.

[0057] In this embodiment, the loss function is defined based on the mean-square error (MES), and the loss function L(θ) is:

[0058]

[0059] Its meaning is the mathematical expectation of the square of the difference between the approximate actual value at the current time step and the value obtained using the current value network, where U(D) indicates that the selected samples follow a uniform distribution. Based on the loss function, the weights θ of the current value network can be updated using gradient descent:

[0060]

[0061] Where α is the learning rate. This represents the gradient of the current value network.

[0062] In this embodiment, to ensure that the optimal action output by the model allows each agent to avoid obstacles while maintaining the formation of the agent queue as much as possible, the target reward functions for the agents during training include an obstacle avoidance reward function, a formation reward function, and a navigation reward function. The obstacle avoidance reward function encourages agents to avoid obstacles and prevent collisions between agents, and can be a function positively correlated with the sum of the distances between the agent and other agents and dynamic obstacles. The formation reward function encourages agents to maintain a given formation shape during actions, and can be a function positively correlated with the similarity between the formation after the action and the preset formation. The navigation reward function encourages agents to move towards the destination, and can be a function negatively correlated with the distance between the agent and the destination. The reward value for each action of the agent is a weighted sum of the values ​​of the obstacle avoidance reward function, the formation reward function, and the navigation reward function. It is understood that the specific calculation methods of the obstacle avoidance reward function, the formation reward function, and the navigation reward function can be set according to actual needs.

[0063] In this embodiment, the obstacle avoidance reward function r a As shown in the following formula:

[0064]

[0065] in, Represents a set of intelligent agents. Denotes the set of observable obstacles; b, c, g a These are all constant coefficients, and the rewards can be adjusted according to the actual situation; δ represents distance, δ min The minimum collision radius is generally defined as the sum of the smallest circumcircle radius of the agent and the largest radius among all dynamically observable obstacles. By employing this obstacle avoidance reward function, when the distance δ between the agent and other obstacles or the agent... ij Less than the minimum collision radius δ min If the agent is deemed to have collided with another agent, it will receive a very large penalty, i.e., a negative reward of -g. a And directly end the current round of sampling and reset the training environment; if δ ijLess than twice δ min If the agent is deemed to be in danger, it will be subject to a minor penalty. However, this punishment increases exponentially, if δ ij Too close to δ min Similarly, they will be severely penalized. This obstacle avoidance reward setting can, on the one hand, prevent the intelligent agent from getting too close to other intelligent agents or obstacles, thus avoiding collisions caused by positioning deviations when the distance is too close, and on the other hand, allow the intelligent agent to enter a dangerous distance in special circumstances, rather than getting stuck in a local optimum.

[0066] In this embodiment, assuming that multiple agents can communicate with each other pairwise, the multi-agent queue can be defined as an undirected graph. The ideal formation diagram is defined as follows: Let D be the graph. The degree matrix, A is the graph. Given an adjacency matrix, the weight w of each edge. ij Defined as the distance between the center points of two adjacent agents, an undirected graph... The Laplace matrix L is defined as follows:

[0067] L = DA (5)

[0068] Normalizing the Laplace matrix, we get:

[0069]

[0070] Formation reward function r f As shown in the following formula:

[0071]

[0072] Among them, ∥.∥ F Let denote the Frobenius norm, and tr{.} denote the trace of the matrix; It is the normalized Laplace matrix of the ideal formation. This formation reward function quantifies the deviation between the current formation shape and the ideal formation shape. By employing symmetric normalization, it allows multi-agent formations to be scaled and rotated based on the ideal formation, in order to better perform obstacle avoidance tasks.

[0073] In this embodiment, the navigation reward function r n As shown in the following formula:

[0074]

[0075] in, and These are the distances from the agent to the destination at the current time step and the previous time step, respectively, D. gThe radius representing the endpoint is generally the same as the radius of the agent; g g This is the gain coefficient for the reward upon reaching the destination. This navigation reward function determines the reward based on the distance the agent traveled from the destination. If the distance from the destination in the previous time step is less than the distance in the current time step, it means the agent is far from the destination and needs to be penalized; conversely, if the distance is greater, it indicates the agent is close to the destination and needs to be rewarded. When the agent reaches the destination, it receives a large destination reward g. g And stop exercising and reset the training environment.

[0076] Finally, the reward r for each action is calculated using the following formula:

[0077] r = α1r a +α2r f +α3r n (9)

[0078] Among them, α1, α2 and α3 are reward discount factors, which can be adjusted according to the actual situation.

[0079] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described multi-agent formation obstacle avoidance method.

[0080] In some possible implementations, various aspects of the multi-agent formation obstacle avoidance method provided by the present invention can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the multi-agent formation obstacle avoidance method according to the various exemplary embodiments of the present application described above.

[0081] Based on the same inventive concept, embodiments of the present invention also provide a control device for implementing the above-described multi-agent formation obstacle avoidance method, including a processor and a memory, wherein the memory is electrically connected to the processor, and the processor implements the above-described multi-agent formation obstacle avoidance method by executing a computer program stored in the memory.

[0082] In one possible design, the processor may include one or more processing units. The processor and memory may be implemented on the same chip or on separate chips. The processor may be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the multi-agent formation obstacle avoidance method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0083] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. The memory in the embodiments of this application can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0084] By designing and programming the processor, the code corresponding to the multi-agent formation obstacle avoidance method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the multi-agent formation obstacle avoidance method of the embodiments of the present invention during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0085] Based on the same inventive concept, embodiments of the present invention also provide another multi-agent formation obstacle avoidance control device, including an acquisition module, an action value evaluation module, a training environment module, a reward calculation module, a model training module, and a control module. The acquisition module is used to acquire observation data of the agent itself, dynamic obstacles, and other agents. In this embodiment, the information required by each agent includes the coordinates (d) of the dynamic obstacle. xn ,d yn ,) Angle θ n and speed v n and the coordinates of the endpoint (g) x ,g y ) and angle θ g All the above parameters are obtained individually in the coordinate system of each agent. The action value evaluation module evaluates the action value of all actions in the action space of each agent based on the dynamic value network model and the observation data obtained by the acquisition module, and outputs the action with the highest action value for each agent. The training environment module provides a training environment model for the training of the dynamic value network model. It updates the position coordinates of each agent based on the input actions of each agent and determines whether an agent collides with other agents or dynamic obstacles based on its position coordinates. The reward calculation module calculates the reward for each action based on the state of the agent before and after the action during the training of the dynamic value network model. The model training module trains the dynamic value network model using the training environment module, controls the initialization of the training environment module, and randomly assigns coordinates to dynamic obstacles and the endpoint of the agent queue in the training environment module. The control module controls the agents to take actions based on the action with the highest action value output by the action evaluation module.

[0086] In this embodiment, the reward calculation module includes an obstacle avoidance reward calculation module, a formation reward calculation module, and a navigation reward calculation module. The obstacle avoidance reward calculation module calculates the obstacle avoidance reward value for an action to encourage agents to avoid obstacles and collisions between agents; the formation reward calculation module calculates the formation reward value for an action to encourage agents to maintain a given formation shape during the action; and the navigation reward calculation module calculates the navigation reward value for an action to encourage agents to move towards the destination. In this embodiment, the obstacle avoidance reward calculation module calculates the obstacle avoidance reward value for an action using equation (4), the formation reward calculation module calculates the formation reward value for an action using equation (7), the navigation reward calculation module calculates the formation reward value for an action using equation (8), and the reward calculation module calculates the final reward value for an action using equation (9).

[0087] In this embodiment, the training environment module includes a position update module and a collision detection module. The position update module calculates the position of each agent after its action using a state transition function. The collision detection module calculates the distances between agents and between agents and dynamic obstacles based on their positions, and determines whether a collision has occurred by comparing these distances with the minimum collision distance.

[0088] In this embodiment, the control device further includes a communication module for communication with the agents, so that agents in the agent queue can obtain the location and observation data of other agents.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," "exceeding," etc. are understood to exclude the stated number, while "above," "below," "within," etc. are understood to include the stated number. If "first" or "second" is mentioned, it is only for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0093] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0094] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A multi-agent formation obstacle avoidance method, characterized in that, The method comprises the following steps: S100: constructing an action value network model and a training environment model, training the action value network model by using the training environment model, and obtaining the action value network model for multi-agent formation obstacle avoidance, wherein the training environment model comprises an agent queue formed by a plurality of agent formations and a plurality of dynamic obstacles, and the coordinates of the dynamic obstacles and the end point coordinates of the agent queue are randomly set in each training process; S200: each agent respectively observes the observation data required by the action value network model, and inputs the observation data into the action value network model, evaluates the action value of all actions of each agent in the respective workspace by the action value network model, and obtains the optimal action with the highest action value of each agent; S300: each agent respectively performs the optimal action, until all agents successfully reach the end point; In the step S100, the reward function of the action value network model in the training process comprises an obstacle avoidance reward function, a formation reward function and a navigation reward function; The obstacle avoidance reward function r a is defined as: wherein, denotes a set of agents, denotes a set of dynamic obstacles, b, c, g a are constant coefficients, δ ij denotes the distance between an agent and other agents or dynamic obstacles, δ min denotes the minimum collision radius of an agent; The platoon reward function r f is defined as: where ||.|| denotes the Frobenius norm, tr{.} denotes the trace of a matrix, F where ||.|| denotes the Frobenius norm, tr{.} denotes the trace of a matrix, is the normalized Laplacian matrix of the ideal formation, is the normalized Laplacian matrix of the ideal formation, The navigation reward function r n is defined as: wherein, denotes a set of agents, and D and D are the distances of the agents at the current time step and the previous time step to the goal, respectively, g denotes the radius of the goal, g g is a gain factor for the reward of reaching the goal.

2. The multi-agent formation obstacle avoidance method of claim 1, wherein, The step S100 comprises: S110: initializing the parameters of the current action value network and the target action value network, and initializing an experience pool; S120: setting a training environment model, wherein the training environment model comprises an agent queue formed by a plurality of agent formations and a plurality of dynamic obstacles; S130: initializing the training environment model, resetting the plurality of agents to the respective start points, and randomly setting the coordinates of the dynamic obstacles and the end point coordinates of the agent queue; S140: evaluating all actions of each agent in the action space by the current action value network, and obtaining the action with the highest action value of each agent; S150: making each agent move according to the action with the highest action value determined in the step S140, and obtaining the feedback of the training environment model; S160: if the training environment model feedbacks that the agents do not collide and the agent queue does not reach the end point, storing the interaction data of the agent queue and the training environment model in the experience pool in a preset format; if the training environment model feedbacks that the agents collide or the agent queue reaches the end point, returning to the step S130; S170: taking a batch of samples from the experience pool, evaluating the approximate actual value of the samples by the target action value network, calculating a loss function according to the approximate actual value and the action value evaluated by the current action value network, and optimizing the parameters of the current action value network by the gradient descent method according to the loss function; S180: synchronizing the parameters of the current action value network to the target action value network every preset number of rounds; S190: if the training rounds reach the preset maximum number of rounds, saving the target action value network as the action value network model.

3. The multi-agent formation obstacle avoidance method according to any one of claims 1 or 2, characterized in that, In the step S100, the obstacle avoidance reward function encourages the agents to avoid obstacles and avoid mutual collision between the agents, the formation reward function encourages the agents to maintain a given formation shape during the actions, and the navigation reward function encourages the agents to move towards the direction of the terminal point. The reward value of each action of the agents is a weighted sum of the values of the obstacle avoidance reward function, the formation reward function, and the navigation reward function.

4. The multi-agent formation obstacle avoidance method of claim 1, wherein, The action value network model comprises one LSTM network and two MLP networks. The observation values of the dynamic obstacles observed by the agents are sequentially input into the LSTM neural network in the order of the distance from the agents, so as to extract feature values of the dynamic obstacles. The observation values of the agents are input into the first MLP network, so as to extract feature values of the agents. The feature values of the dynamic obstacles and the feature values of the agents are spliced and input into the second MLP network, so as to obtain the action value of the action.

5. A storage medium storing a computer program, characterized by The computer program is called and executed by the processor to implement the multi-agent formation obstacle avoidance method according to any one of claims 1 to 4.

6. A multi-agent formation obstacle avoidance control device for implementing the multi-agent formation obstacle avoidance method according to any one of claims 1 to 4, characterized in that, Comprise: An acquisition module configured to acquire observation data of the agents, dynamic obstacles, other agents, and terminal point information; An action value evaluation module configured to evaluate action values of all actions in an action space of each agent according to a dynamic value network model and the observation data acquired by the acquisition module, and output an action with the highest action value of each agent; A training environment module configured to provide a training environment model for training of the dynamic value network model, update position coordinates of the agents according to input actions of the agents, and determine whether the agents collide with other agents or the dynamic obstacles according to the position coordinates of the agents; A reward calculation module configured to calculate a reward of an action according to states before and after the action of the agent during the training of the dynamic value network model; A model training module configured to train the dynamic value network model by using the training environment module, control the training environment module to be initialized, and randomly set coordinates of the dynamic obstacles and a terminal point of the agent queue in the training environment module; A control module configured to control the agents to perform actions respectively according to the action with the highest action value of each agent output by the action evaluation module.

7. The multi-agent formation obstacle avoidance control device according to claim 6, wherein, The reward calculation module comprises: An obstacle avoidance reward calculation module configured to calculate an obstacle avoidance reward value of an action to encourage the agents to avoid obstacles and avoid mutual collision between the agents; A formation reward calculation module configured to calculate a formation reward value of an action to encourage the agents to maintain a given formation shape during the action; A navigation reward calculation module configured to calculate a navigation reward value of an action to encourage the agents to move towards the direction of the terminal point; The reward calculation module calculates a final reward value of the action by calculating a weighted sum of the obstacle avoidance reward value, the formation reward value, and the navigation reward value of the action.

Images